github/vac.dev
1
1
Fork 0
mirror of https://github.com/vacp2p/vac.dev.git synced 2026-01-09 22:58:06 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
deploy-develop
vac.dev/rlog/atom.xml
e7bb01433f Update documentation
2025-09-02 12:08:33 +00:00

5938 lines
1.5 MiB
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<id>https://vac.dev/rlog</id>
<title>Vac Research Blog</title>
<updated>2025-09-02T14:00:00.000Z</updated>
<generator>https://github.com/jpmonette/feed</generator>
<link rel="alternate" href="https://vac.dev/rlog"/>
<subtitle>Vac Research Blog</subtitle>
<icon>https://vac.dev/theme/image/favicon.ico</icon>
<entry>
<title type="html"><![CDATA[Decentralized Message Layer Security (De-MLS) with Waku]]></title>
<id>https://vac.dev/rlog/de-mls-with-waku</id>
<link href="https://vac.dev/rlog/de-mls-with-waku"/>
<updated>2025-09-02T14:00:00.000Z</updated>
<summary type="html"><![CDATA[This post introduces de-MLS, a decentralized variant of Message Layer Security (MLS)]]></summary>
<content type="html"><![CDATA[<p>This post introduces de-MLS, a decentralized variant of Message Layer Security (MLS)
that reimagines group messaging by replacing centralized delivery services with peer-to-peer protocols
while retaining strong guarantees such as forward secrecy (FS) and post-compromise security (PCS).</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/de-mls-with-waku#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Secure Group Messaging (SGM) is resource-intensive when aiming for robust security features
like forward secrecy (FS) and post-compromise security (PCS).</p>
<p>One straightforward approach to SGM is a pairwise group chat,
where each pair of group members establishes a unique encryption key using Diffie-Hellman.
While this method ensures security, it falls short in terms of practicality:</p>
<ul>
<li><strong>High storage requirements</strong>: Each participant must store encryption keys for every other participant.</li>
<li><strong>Inefficient encryption</strong>: Each message must be encrypted separately for every participant,
leading to significant computational overhead.</li>
<li><strong>Inefficient message storage and delivery</strong>: Each separately encrypted message must then be sent over the wire,
whatever this wire might be. Or stored in database.</li>
<li><strong>Cumbersome group management</strong>: Adding or removing users and refreshing keys becomes
increasingly inefficient as the group grows.</li>
</ul>
<p>One scalable for Secure Group Messaging (SGM) is Message Layer Security (MLS), as standardized in <a href="https://datatracker.ietf.org/doc/rfc9420/" target="_blank" rel="noopener noreferrer">RFC 9420</a>.
Leveraging TreeKEM, MLS organizes group members in a cryptographic tree structure,
where each participant is responsible for maintaining specific parts of the tree.</p>
<p>While MLS offers scalability and strong security guarantees,
its reliance on server-based delivery services poses limitations for fully decentralized environments.</p>
<p>In this post, we present the implementation details of the first version of Decentralized MLS (de-MLS)
which is an SGM protocol. De-MLS can serve groups that cannot rely on central servers,
such as journalists and activists seeking secure communication.
It is also well suited for DAOs, where Ethereum-based authentication can restrict access to members
holding a minimum ETH balance, and for NGOs or research consortia that prefer not to host their own servers while still
requiring end-to-end encrypted group messaging. Decentralized MLS (de-MLS) satisfies the following features:</p>
<ul>
<li>Decentralized</li>
<li>Scalable</li>
<li>End-to-end encrypted (E2EE)</li>
<li>FS and PCS provided</li>
<li>Ethereum authenticated</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="https://vac.dev/rlog/de-mls-with-waku#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mls">MLS<a href="https://vac.dev/rlog/de-mls-with-waku#mls" class="hash-link" aria-label="Direct link to MLS" title="Direct link to MLS">​</a></h3>
<p>The Message Layer Security (MLS) protocol offers scalable and secure group messaging protocol
by organizing participants into a cryptographic tree structure,
enabling efficient operations like adding or removing members with logarithmic time complexity
relative to the group size. MLS provides strong security guarantees, including FS and PCS.</p>
<p>MLS assumes that two services are provided:</p>
<ul>
<li>An Authentication Service (AS): It enables group members to
authenticate the credentials presented by other group members.</li>
<li>A Delivery Service (DS) that routes MLS messages among the
participants in the protocol in the correct order and manage the <code>keyPackage</code> of the users
where the <code>keyPackage</code> is the objects that provide some public information about a user.</li>
</ul>
<p>Despite its scalability, MLS has a notable limitation:
it is inherently designed for server-based federated architectures for delivery service (DS),
even when the servers themselves don't need to be trusted.
To achieve a decentralized protocol, the functionality of DS must be reimagined
to eliminate reliance on a central server while preserving the protocol's security properties.
Thus, we proposed decentralized MLS (de-MLS),
leveraging Waku nodes as peer-to-peer communication protocols to eliminate reliance on centralized servers.</p>
<p>Lastly, MLS operates on an epoch-based model,
where group state changes (e.g., adding/removing users or key refreshes) occur between epochs
that are always required to be conducted by a single entity.
For example, if a user is removed in epoch <code>E</code>,
the rest of the group members generate a new key in epoch <code>E + 1</code> by passing the new entropy.
The removed user cannot decrypt messages sent after epoch <code>E + 1</code>.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku">Waku<a href="https://vac.dev/rlog/de-mls-with-waku#waku" class="hash-link" aria-label="Direct link to Waku" title="Direct link to Waku">​</a></h3>
<p><a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a> is a decentralized messaging protocol designed for secure and efficient communication in peer-to-peer networks.
It operates as a broadcast-based routing layer where content topics can be used to tag and filter messages.
Users join channels by subscribing to specific content topics,
which determine the scope and type of messages exchanged.
This enables flexible and efficient communication patterns in a decentralized environment.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="de-mls">de-MLS<a href="https://vac.dev/rlog/de-mls-with-waku#de-mls" class="hash-link" aria-label="Direct link to de-MLS" title="Direct link to de-MLS">​</a></h2>
<p>Decentralized MLS (de-MLS) is a peer-to-peer secure group messaging protocol
that can work with any delivery service (DS) meeting a minimal set of requirements.
In this post, we highlight its integration with <a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a> as the messaging protocol,
while emphasizing that de-MLS itself remains agnostic to the underlying DS.
Further technical details can be found in the <a href="https://rfc.vac.dev/vac/raw/eth-mls-offchain" target="_blank" rel="noopener noreferrer">de-MLS RFC</a>.</p>
<p>Decentralization is achieved not only at the delivery service (DS) level
but also within the authentication service (AS).
Multiple special nodes named Steward in the group serve as authorized identities to authenticate users
before they join or are removed from the group transparently.</p>
<p>de-MLS provides two different user management configurations, both utilizing the Waku protocol for DS:</p>
<ol>
<li><strong>Single Steward</strong>:<!-- -->
<ul>
<li>A single authorized identity (Steward) manages the group,
including removing or adding users with agreement among users by a voting-based consensus.</li>
</ul>
</li>
<li><strong>Multi-Steward</strong>:<!-- -->
<ul>
<li>Multiple Stewards have equal authority to add or remove users.</li>
<li>A consensus mechanism ensures consistency by resolving concurrent changes
within the same epoch and preventing possible conflicts.
In each epoch, all modifications are managed exclusively by a single Steward.</li>
</ul>
</li>
</ol>
<p>Note: We chose the term Steward to reflect the role of transparently coordinating and organizing passengers at stations, much like Stewards do in transit systems.</p>
<p>In multi-Steward settings, de-MLS requires a consensus among Stewards
that have equal rights in the group since changes in an epoch in MLS are required
to be conducted by a single identity, that is the Steward.</p>
<p>For the consensus integration, ongoing research explores two promising approaches:</p>
<ol>
<li><strong>On-chain consensus mechanisms</strong>:
Outsourcing consensus to a smart contract solution for transparent and immutable agreement.</li>
<li><strong>Off-chain consensus mechanisms</strong>:
Utilizing off-chain consensus protocols to design efficient, decentralized protocols.</li>
</ol>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="waku-integration">Waku Integration<a href="https://vac.dev/rlog/de-mls-with-waku#waku-integration" class="hash-link" aria-label="Direct link to Waku Integration" title="Direct link to Waku Integration">​</a></h3>
<p>Waku integration is a crucial step in the construction of de-MLS,
aiming to replace traditional client-server communication with decentralized messaging.
The specifics of Waku integration will be detailed in a separate RFC;
for now, our main priority is the de-MLS RFC.</p>
<p>The main challenge in this transition is transforming the centralized Delivery Service (DS)
into a decentralized equivalent, which performs two essential functions:</p>
<ol>
<li>Message Delivery and Ordering:
The DS is responsible not only for delivering messages to the correct recipients,
but also for preserving the correct order of these messages, which is critical for the consistency of group state.</li>
<li>Key Package Management:
The DS manages key packages, which are essential for adding members securely to a group.</li>
</ol>
<p>To maintain a truly decentralized architecture,
key packages cannot be stored in a centralized location.
Initially, we considered using a smart contract (SC) as a decentralized substitute for server-side key package storage.
However, this approach proved impractical.
Blockchains are immutable by design—once data is written, it cannot be fully removed.
This contradicts a core requirement of MLS: each key package must be used exactly once and then deleted,
to prevent replay or reuse attacks.
Instead, our solution is to require users to actively provide their key packages upon request,
allowing validation at the moment of use without persistent storage.
While this approach may lose some benefits of asynchronicity,
we plan to address this in the future by introducing store nodes that can temporarily hold key packages.
This ensures both compliance with MLS's security model and alignment with decentralized system principles.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="flow">Flow<a href="https://vac.dev/rlog/de-mls-with-waku#flow" class="hash-link" aria-label="Direct link to Flow" title="Direct link to Flow">​</a></h3>
<p>The flow section explains the processes that
when a user wants to join a group in both Steward and users side also their interactions.
The flow of de-MLS is as follows:</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Figure 1" src="https://vac.dev/assets/images/flow-e56eebf7e59df3cb5a6acc738dbeb72e.png" width="921" height="627" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-steward-joins-the-welcome-topic">1. Steward joins the welcome topic<a href="https://vac.dev/rlog/de-mls-with-waku#1-steward-joins-the-welcome-topic" class="hash-link" aria-label="Direct link to 1. Steward joins the welcome topic" title="Direct link to 1. Steward joins the welcome topic">​</a></h3>
<p>The welcome topic is a topic created and monitored by the Steward for a specific secure messaging group,
allowing any Waku node to subscribe permissionlessly.
Being in the welcome topic does not imply group membership,
it acts as a waiting room where users can send their key material,
which the Steward listens for and processes before granting access to the secure group.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-group-initialization">2. Group initialization<a href="https://vac.dev/rlog/de-mls-with-waku#2-group-initialization" class="hash-link" aria-label="Direct link to 2. Group initialization" title="Direct link to 2. Group initialization">​</a></h3>
<p>Steward initalizes a group with parameters such as cipher suite and group ID.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-emitting-group-anouncement-ga-by-steward">3. Emitting Group Anouncement (GA) by Steward<a href="https://vac.dev/rlog/de-mls-with-waku#3-emitting-group-anouncement-ga-by-steward" class="hash-link" aria-label="Direct link to 3. Emitting Group Anouncement (GA) by Steward" title="Direct link to 3. Emitting Group Anouncement (GA) by Steward">​</a></h3>
<p>Steward creates group announcement (GA) periodically to the welcome channel
that the users can find the who the Steward is.
This will be important for the next step.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-user-joins-the-welcome-topic">4. User joins the welcome topic<a href="https://vac.dev/rlog/de-mls-with-waku#4-user-joins-the-welcome-topic" class="hash-link" aria-label="Direct link to 4. User joins the welcome topic" title="Direct link to 4. User joins the welcome topic">​</a></h3>
<p>As first, the users who wants to be part of the decentralized MLS should subscribe the welcome channel.
Then user can find the group name and also corresponding GA message from Steward.
This GA message helps the user to create a valid <code>keyPackages</code> which define in section 10
in <a href="https://datatracker.ietf.org/doc/rfc9420/" target="_blank" rel="noopener noreferrer">RFC9420</a> for the group.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-user-creates-its-key-package">5. User creates its key package<a href="https://vac.dev/rlog/de-mls-with-waku#5-user-creates-its-key-package" class="hash-link" aria-label="Direct link to 5. User creates its key package" title="Direct link to 5. User creates its key package">​</a></h3>
<p>User creates the <code>keyPackage</code> and encrypt by public key of the Steward then send it to the Steward.
Since the message is encrypted, stay secure though the welcome (permissionless) topic.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="6-steward-receives-the-users-key-package">6. Steward receives the User's key package<a href="https://vac.dev/rlog/de-mls-with-waku#6-steward-receives-the-users-key-package" class="hash-link" aria-label="Direct link to 6. Steward receives the User's key package" title="Direct link to 6. Steward receives the User's key package">​</a></h3>
<p>Steward receives the user's <code>keyPackage</code> and decrypt it.
After decrypted, Steward also verifies the validity of the <code>keyPackage</code> by signature verification.
If the <code>keyPackage</code> is not valid, the Steward just drops the message,
otherwise it moves to the next step which is proposal creation.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="7-creation-of-voting-proposals">7. Creation of Voting proposals<a href="https://vac.dev/rlog/de-mls-with-waku#7-creation-of-voting-proposals" class="hash-link" aria-label="Direct link to 7. Creation of Voting proposals" title="Direct link to 7. Creation of Voting proposals">​</a></h3>
<p>Voting proposals are special MLS application messages that may come from any participant, including the Steward.
In this context, any member can create a proposal corresponding to the user’s <code>keyPackage</code>.
In regular MLS, proposals are automatically converted into commit messages,
which can change the structure of the tree. However, in de-MLS, since the process is decentralized,
proposals must be voted on before being converted into a commitment.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="8-voting-for-proposal">8. Voting for proposal<a href="https://vac.dev/rlog/de-mls-with-waku#8-voting-for-proposal" class="hash-link" aria-label="Direct link to 8. Voting for proposal" title="Direct link to 8. Voting for proposal">​</a></h3>
<p>Voting applies decentralization by protecting small groups can control.
Therefore, proposals must be voted on before committing.
The consensus mechanism should be a lightweight consensus that cannot be a bottleneck for treeKEM scalability.
Basically, the consensus returns the binary result for a given proposal.
If voting result is NO, the proposal is dropped; otherwise, the Steward transforms it into an MLS proposal.
MLS proposal message is a distinct type of MLS application message,
where the Steward attaches the voting result instead of directly releasing a commit message.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="9-creating-commit-message">9. Creating commit message<a href="https://vac.dev/rlog/de-mls-with-waku#9-creating-commit-message" class="hash-link" aria-label="Direct link to 9. Creating commit message" title="Direct link to 9. Creating commit message">​</a></h3>
<p>Commit messages are the messages that start new epochs.
They include key and tree material that existing members can use to generate the new state of the tree.</p>
<p>After Steward gets the YES from consensus, Steward creates commit messages
that injects new entropy for the existing group members.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="10-sending-messages">10. Sending messages<a href="https://vac.dev/rlog/de-mls-with-waku#10-sending-messages" class="hash-link" aria-label="Direct link to 10. Sending messages" title="Direct link to 10. Sending messages">​</a></h3>
<p>After Steward creates and then sends two messages:</p>
<ol>
<li>Commit message informs existing group member to update their key
to align with the new member’s key for the upcoming epoch.</li>
<li>The welcome message informs the newly joined user to generate a group key
that matches the key existing members will use in the upcoming epoch.</li>
</ol>
<p>Although existing users had different group keys in the previous epoch and the new user had none,
the Steward message ensures that both existing and new users converge on the same group key in the next epoch.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="11-applying-welcome-message">11. Applying welcome message<a href="https://vac.dev/rlog/de-mls-with-waku#11-applying-welcome-message" class="hash-link" aria-label="Direct link to 11. Applying welcome message" title="Direct link to 11. Applying welcome message">​</a></h3>
<p>User can generate the next epoch group key by using the welcome message as well as
existing users extract the same <code>groupKey</code> by using commit messages.</p>
<p>The commit message helps existing members generate the next group key <code>Gk+1</code>,
while the welcome message helps the newly joining user generate the same <code>Gk+1</code>.
This provides two important security properties:</p>
<ol>
<li>
<p>Forward Secrecy (FS):
The new user cannot read previous messages since they were encrypted with the old key <code>Gk</code></p>
</li>
<li>
<p>Post-Compromise Security (PCS):
If a user is removed from the group,
they cannot read future messages since those messages will be encrypted with the new key <code>Gk+1</code></p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="benchmark">Benchmark<a href="https://vac.dev/rlog/de-mls-with-waku#benchmark" class="hash-link" aria-label="Direct link to Benchmark" title="Direct link to Benchmark">​</a></h2>
<p>This section presents the performance evaluation of de-MLS.
One of the key advantages of the MLS protocol is its efficiency,
as it eliminates the need for pairwise message exchanges between all participants.
Instead, the decentralized DS enables the addition of new participants by sending only two messages to the group:
a commit message and a welcome message.
However, despite this advantage, the protocol does have certain bottlenecks, which are as follows:</p>
<ul>
<li>Firstly, the Steward must receive the key packages from each member wishing to join the group.
This process requires sequential message exchanges and involves computationally intensive tasks such as encryption,
decryption, and digital signature verification.
Even when multiple users are added to the group simultaneously, the process is essentially sequential.
The tree structure is updated one user at a time,
followed by sending the final commit message to the existing group members
and a single welcome message to the new members.</li>
<li>Secondly adding a member to a group requires rebuilding the tree and computing new keys.</li>
</ul>
<p>The following measurements were made as follows:</p>
<ol>
<li>The time required for the entire sequence of receiving a user key package is presented here.
This includes generating the Steward key, creating messages with signatures and encryption,
and processing these messages.</li>
</ol>
<p><code>Share Key Package - 1.8395 ms</code></p>
<p>Note that these measurements do not account for the time taken to forward messages.</p>
<ol>
<li>The time required for creating the commit and welcome message
from a ready-made package bunches is shown in this table.</li>
</ol>
<table><thead><tr><th>Group Size (by users)</th><th>Time</th></tr></thead><tbody><tr><td>10</td><td>1.8662 ms</td></tr><tr><td>100</td><td>14.124 ms</td></tr><tr><td>500</td><td>121.85 ms</td></tr><tr><td>1000</td><td>412.39 ms</td></tr><tr><td>5000</td><td>~ 15-20 s</td></tr><tr><td>10000</td><td>~ 1-1.5 min</td></tr></tbody></table>
<p>The tests were conducted on the following configuration:
Apple M3 Pro @ 4.05GHz and 12-Core CPU/18-Core GPU.</p>
<p>Here, the network latency and the time taken by users to apply the received commits are also excluded.
These aspects are planned to be measured and evaluated in future work.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="potential-drawbacks-and-countermeasures">Potential drawbacks and countermeasures<a href="https://vac.dev/rlog/de-mls-with-waku#potential-drawbacks-and-countermeasures" class="hash-link" aria-label="Direct link to Potential drawbacks and countermeasures" title="Direct link to Potential drawbacks and countermeasures">​</a></h2>
<p>Since de-MLS replace the servers by P2P, we could lose some good features of servers based MLS.
In this section we present the potential drawbacks and possible countermeasures of de-MLS.</p>
<ul>
<li>Offline users: <code>keyPackage</code>s are provided by the users directly without any storing,
this is required each user must be online for joining to a group.<!-- -->
<ul>
<li>We can consider to use <a href="https://docs.waku.org/guides/js-waku/store-retrieve-messages/" target="_blank" rel="noopener noreferrer">Waku sync nodes</a>
that are nodes has storing ability for a temporary storing of <code>keyPackage</code>s.</li>
</ul>
</li>
<li>DoS attack to Steward: Steward is known in welcome message from periodic group announcement message
so Steward can be targeted for DoS attack.<!-- -->
<ul>
<li>As always we consider to use Rate-Limiting Nullifier (RLN) with Waku to protect network from spam.</li>
</ul>
</li>
<li>Message loss or delay : Because of P2P and consensus settings, message can be lost or delayed.,<!-- -->
<ul>
<li>We can integrate reliability mechanisms to Waku such as
<a href="https://github.com/waku-org/nim-sds" target="_blank" rel="noopener noreferrer">scalable data sync (SDS)</a></li>
<li>Consensus mechanism requires to provide liveness property against offline nodes, for example,
it may provides default YES or NO options for a silent users who do not vote.</li>
</ul>
</li>
<li>Enchanced authentication<!-- -->
<ul>
<li>Ethereum authentication could be inefficient.
We can configure the authentication mechanism for example asking minimum balance or etc.</li>
</ul>
</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/de-mls-with-waku#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>To summarize, the approach to solving decentralized DS tasks with Waku
can be outlined as shown in the comparison table:</p>
<table><thead><tr><th>Feature</th><th>MLS</th><th>de-MLS</th></tr></thead><tbody><tr><td>Message Distribution</td><td>Messages are sent from the server to clients</td><td>Messages are sent by publishing/subscribing to pub-sub topics</td></tr><tr><td>Commit Message Handling</td><td>Relies on a server</td><td>Relies on a consensus and transparent Steward</td></tr><tr><td>Key Package Management</td><td>Key packages are stored and distributed by the server</td><td>Key packages are provided by the users themselves</td></tr></tbody></table>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future Work<a href="https://vac.dev/rlog/de-mls-with-waku#future-work" class="hash-link" aria-label="Direct link to Future Work" title="Direct link to Future Work">​</a></h2>
<p>In the next iterations, the implementations are planned as following:</p>
<ul>
<li>Dual-Consensus Multi-Steward Support: One consensus mechanism selects an Steward from all users,
while a second governs group decisions among the elected Stewards</li>
<li>Consensus mechanism for handling concurrent changes within the same epoch</li>
<li>Key rotation support</li>
<li>Benchmarking for the multi-Steward configuration including the network time</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/de-mls-with-waku#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li>[1] RFC 9420: The Messaging Layer Security (MLS) Protocol. Retrieved from <a href="https://datatracker.ietf.org/doc/rfc9420/" target="_blank" rel="noopener noreferrer">https://datatracker.ietf.org/doc/rfc9420/</a></li>
<li>[2] OpenMLS. Retrived from <a href="https://github.com/openmls/openmls" target="_blank" rel="noopener noreferrer">https://github.com/openmls/openmls</a></li>
<li>[3] Waku. Retrived from <a href="https://waku.org/" target="_blank" rel="noopener noreferrer">https://waku.org/</a></li>
<li>[4] de-MLS. Retrived from <a href="https://github.com/vacp2p/de-mls/" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/de-mls/</a></li>
</ul>]]></content>
<author>
<name>Ekaterina</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Scaling libp2p GossipSub for Large Messages: An Evaluation of Performance Improvement Proposals]]></title>
<id>https://vac.dev/rlog/gsub-perf-imp-comparison</id>
<link href="https://vac.dev/rlog/gsub-perf-imp-comparison"/>
<updated>2025-08-12T12:00:00.000Z</updated>
<summary type="html"><![CDATA[The original GossipSub design emphasizes robustness, with less focus on message sizes.]]></summary>
<content type="html"><![CDATA[<p>The original GossipSub design emphasizes robustness, with less focus on message sizes.
However, emerging use cases—such as Ethereum's EIP-4844 and
data availability sampling (DAS) require rapid propagation of high data volumes,
often in the form of large messages.
Many ongoing research efforts attempt to find solutions
for efficiently forwarding large messages through the GossipSub network.
This post provides a concise overview and performance evaluation of some research initiatives
aimed at improving GossipSub to meet the performance needs of modern P2P networks.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview">Overview<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#overview" class="hash-link" aria-label="Direct link to Overview" title="Direct link to Overview">​</a></h2>
<p>For each subscribed topic, GossipSub nodes maintain a full-message mesh of peers with a target degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>,
bounded by lower and upper thresholds <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{low}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{high}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">hi</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>​, respectively.
In parallel, a metadata-only (gossip) mesh includes additional <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> peers for metadata exchange.
All messages flow through the full-message mesh, and metadata flows through the gossip mesh.</p>
<p>The metadata contains IHAVE messages, announcing IDs of seen messages.
Upon receiving an IHAVE announcement about an unseen message ID,
a receiver responds with an IWANT request to retrieve the announced message.
IHAVE announcements serve multiple purposes:</p>
<ol>
<li>Offer additional resilience in situations involving non-conforming peers or network partitions.</li>
<li>Speed up message propagation by allowing far-off peers to fetch overdue messages.</li>
</ol>
<p>Since GossipSub v1.1 [<a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md" target="_blank" rel="noopener noreferrer">1</a>],
replying to IWANT requests is optional, to safeguard honest peers from adversaries.
This change encourages peers to make redundant IWANT requests for the same message.
While this arrangement works well for small messages, it can be inefficient for bigger ones.
This inefficiency arises from an increase in duplicates
and a higher number of IWANT requests due to longer transmission times for large messages.
As a result, we observe a significant rise in bandwidth utilization and message dissemination times across the network.</p>
<p>IDONTWANT messages in GossipSub v1.2 [<a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md" target="_blank" rel="noopener noreferrer">2</a>] help reduce some duplicates.
However, further efforts are necessary to mitigate this problem
[<a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921" target="_blank" rel="noopener noreferrer">3</a>,
<a href="https://ethresear.ch/t/impact-of-idontwant-in-the-number-of-duplicates/22652" target="_blank" rel="noopener noreferrer">4</a>].
That is why many recent works focus on improving GossipSub's performance when handling large messages
[<a href="https://www.arxiv.org/abs/2505.17337" target="_blank" rel="noopener noreferrer">5</a>,
<a href="https://arxiv.org/abs/2504.10365" target="_blank" rel="noopener noreferrer">6</a>,
<a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw" target="_blank" rel="noopener noreferrer">7</a>,
<a href="https://ethresear.ch/t/fulldas-towards-massive-scalability-with-32mb-blocks-and-beyond/19529" target="_blank" rel="noopener noreferrer">8</a>,
<a href="https://github.com/libp2p/specs/pull/681" target="_blank" rel="noopener noreferrer">9</a>,
<a href="https://ethresear.ch/t/pppt-fighting-the-gossipsub-overhead-with-push-pull-phase-transition/22118/1" target="_blank" rel="noopener noreferrer">10</a>,
<a href="https://github.com/libp2p/specs/pull/654" target="_blank" rel="noopener noreferrer">11</a>,
<a href="https://github.com/libp2p/specs/pull/653" target="_blank" rel="noopener noreferrer">12</a>].</p>
<p>In this post, we evaluate the push-pull phase transition (PPPT) mechanism
[<a href="https://ethresear.ch/t/pppt-fighting-the-gossipsub-overhead-with-push-pull-phase-transition/22118/1" target="_blank" rel="noopener noreferrer">10</a>],
the GossipSub v1.4 proposal [<a href="https://github.com/libp2p/specs/pull/654" target="_blank" rel="noopener noreferrer">11</a>,
<a href="https://www.arxiv.org/abs/2505.17337" target="_blank" rel="noopener noreferrer">5</a>],
and the GossipSub v2.0 proposal [<a href="https://github.com/libp2p/specs/pull/653" target="_blank" rel="noopener noreferrer">12</a>]
for performance against Gossipsub v1.2.
For this purpose, we implement minimal proof-of-concept (PoC) versions of these protocols in nim-libp2p
[<a href="https://github.com/vacp2p/nim-libp2p" target="_blank" rel="noopener noreferrer">13</a>]
and use the shadow simulator to provide performance evaluation results.</p>
<p>We begin by discussing the issues of message dissemination times and duplicate messages in GossipSub.
Next, we provide a brief overview of the proposals under review,
followed by the experiments and findings from our performance evaluations.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="message-transfer-time-and-duplicates">Message Transfer Time and Duplicates<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#message-transfer-time-and-duplicates" class="hash-link" aria-label="Direct link to Message Transfer Time and Duplicates" title="Direct link to Message Transfer Time and Duplicates">​</a></h2>
<p>Assuming uniform link characteristics, message dissemination to full-message mesh concludes in
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub><mo>≈</mo><mo stretchy="false">(</mo><mi>D</mi><mo>×</mo><msub><mi>τ</mi><mrow><mi>t</mi><mi>x</mi></mrow></msub><mo stretchy="false">)</mo><mo>+</mo><msub><mi>τ</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D \approx (D \times \tau_{tx}) + \tau_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6331em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> time, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mrow><mi>t</mi><mi>x</mi></mrow></msub><mo>=</mo><mfrac><mi>S</mi><mi>R</mi></mfrac></mrow><annotation encoding="application/x-tex">\tau_{tx} = \frac{S}{R}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em">S</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>,
with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi></mrow><annotation encoding="application/x-tex">S</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi></mrow><annotation encoding="application/x-tex">R</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">\tau_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> being the message size, data rate, and link latency, respectively.</p>
<p>This simplifies network-wide dissemination time to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>N</mi></msub><mo>≈</mo><msub><mi>τ</mi><mi>D</mi></msub><mo>×</mo><mi>h</mi></mrow><annotation encoding="application/x-tex">\tau_{N} \approx \tau_D \times h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6331em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">N</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span>,
with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span> indicating the number of hops along the longest path.
This implies that a tenfold increase in message size results in an eightyfold rise in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo>×</mo><msub><mi>τ</mi><mrow><mi>t</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D \times \tau_{tx}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> for a mesh with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">D = 8</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">8</span></span></span></span>,
which accumulates across <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span> hops.
This leads to two fundamental problems:</p>
<ol>
<li>
<p>A longer contention interval (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>) increases the chance that peers receive the same message from multiple mesh members during that interval,<br>
<!-- -->leading to redundant transmissions and more duplicates.
Reducing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> inadvertently increases <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span> — potentially slowing propagation overall.</p>
</li>
<li>
<p>Peers are unaware of ongoing message receptions and may generate redundant IWANT requests
for the messages they are already receiving.
Most of these requests are entertained by early message receivers,
which increases message dissemination time by increasing the workload at peers situated along the optimal path.</p>
</li>
</ol>
<p>Talking about duplicates, a network comprising <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span> peers,
each with a degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>, has a total of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi>N</mi><mo>×</mo><mi>D</mi></mrow><mn>2</mn></mfrac></mrow><annotation encoding="application/x-tex">\frac {N \times D}{2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">N</span><span class="mbin mtight">×</span><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span> edges (links),
as every link connects two peers.
Assuming that a message traverses every link exactly once,
we get at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi>N</mi><mo>×</mo><mi>D</mi></mrow><mn>2</mn></mfrac></mrow><annotation encoding="application/x-tex">\frac {N \times D}{2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">N</span><span class="mbin mtight">×</span><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span> transmissions.</p>
<p>Only <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> transmissions are necessary for delivering a message to all peers.
As a result, we get <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi>N</mi><mo>×</mo><mi>D</mi></mrow><mn>2</mn></mfrac><mo>−</mo><mo stretchy="false">(</mo><mi>N</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\frac {N \times D}{2} -(N-1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">N</span><span class="mbin mtight">×</span><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">)</span></span></span></span> duplicates in the network.
We can simplify average duplicates received by a single peer to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>d</mi><mo>ˉ</mo></mover><mrow><mi>m</mi><mi>i</mi><mi>n</mi></mrow></msub><mo>≈</mo><mfrac><mi>D</mi><mn>2</mn></mfrac><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\bar{d}_{min} \approx \frac{D}{2}-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9812em;vertical-align:-0.15em"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8312em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal">d</span></span><span style="top:-3.2634em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.0833em"><span class="mord">ˉ</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">min</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>.
Here, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>d</mi><mo>ˉ</mo></mover><mrow><mi>m</mi><mi>i</mi><mi>n</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\bar{d}_{min}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9812em;vertical-align:-0.15em"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8312em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal">d</span></span><span style="top:-3.2634em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.0833em"><span class="mord">ˉ</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">min</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> represents the lower bound on average duplicates
because we assume that the send and receive operations are mutually exclusive.
This assumption requires that message transmission times (and link latencies)
are so small that no two peers simultaneously transmit the same message to each other.</p>
<p>However, a large message can noticeably increase the contention interval,
which increases the likelihood that many peers will simultaneously transmit the same message to each other.</p>
<p>Authors in [<a href="https://streamr-public.s3.amazonaws.com/streamr-network-scalability-whitepaper-2020-08-20.pdf" target="_blank" rel="noopener noreferrer">14</a>]
explore the upper bound (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>d</mi><mo>ˉ</mo></mover><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\bar{d}_{max}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9812em;vertical-align:-0.15em"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8312em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal">d</span></span><span style="top:-3.2634em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.0833em"><span class="mord">ˉ</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ma</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>) on duplicates.
They argue that a node can forward a received message to a maximum of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">D-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> peers
while the original publisher sends it to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> peers.
As a result, we get <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo stretchy="false">(</mo><mi>D</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N(D-1)+1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> transmissions in the network.
Only <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> transmissions are necessary to deliver a message to all peers.
Remaining <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo stretchy="false">(</mo><mi>D</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo><mo>+</mo><mn>1</mn><mo>−</mo><mo stretchy="false">(</mo><mi>N</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">N(D-1)+1-(N-1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">)</span></span></span></span> transmissions are duplicates,
which simplifies the upper bound on average duplicates received by each peer to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>d</mi><mo>ˉ</mo></mover><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>≈</mo><mi>D</mi><mo>−</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">\bar{d}_{max} \approx D-2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9812em;vertical-align:-0.15em"></span><span class="mord"><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8312em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal">d</span></span><span style="top:-3.2634em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.0833em"><span class="mord">ˉ</span></span></span></span></span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ma</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">2</span></span></span></span>.
This rise indicates that larger messages can lead to more duplicates due to longer contention intervals.
It is essential to highlight that the impact of IWANT/IDONTWANT messages is not considered in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>d</mi><mo>ˉ</mo></mover></mrow><annotation encoding="application/x-tex">\bar{d}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8312em"></span><span class="mord accent"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8312em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span class="mord mathnormal">d</span></span><span style="top:-3.2634em"><span class="pstrut" style="height:3em"></span><span class="accent-body" style="left:-0.0833em"><span class="mord">ˉ</span></span></span></span></span></span></span></span></span></span> computations above.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="protocols-considered">Protocols Considered<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#protocols-considered" class="hash-link" aria-label="Direct link to Protocols Considered" title="Direct link to Protocols Considered">​</a></h2>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="push-pull-phase-transition-pppt"><a href="https://ethresear.ch/t/pppt-fighting-the-gossipsub-overhead-with-push-pull-phase-transition/22118/1" target="_blank" rel="noopener noreferrer">Push-Pull Phase Transition (PPPT)</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#push-pull-phase-transition-pppt" class="hash-link" aria-label="Direct link to push-pull-phase-transition-pppt" title="Direct link to push-pull-phase-transition-pppt">​</a></h3>
<p>In PPPT, authors argue that most redundant transmissions occur during the later stages of message propagation.
As a message traverses through the network,
the peers forwarding the message should gradually reduce the number of mesh members that directly receive
the message (push) and send immediate IHAVE announcements to the remaining peers in the mesh.
The remaining mesh members can fetch any missing messages using IWANT requests (Pull).
The authors also suggest two strategies to estimate the message propagation stage:</p>
<ol>
<li>
<p>Include a hop count in the message header to identify the number of hops traversed by that message.
When a peer forwards a received message,
it performs a pull operation for a subset of mesh members that equals the specified hop count
and a push operation for the remaining mesh members.</p>
</li>
<li>
<p>Infer the message propagation stage by looking into
the number of received IHAVE announcements and duplicates for that message.
Use this information to choose a balance between pull-based and push-based message forwarding.</p>
</li>
</ol>
<p>The authors suggest that instead of simultaneously pushing a message to the selected peers,
sequentially initiating transmission to each peer after a short delay
enhances the likelihood of timely receiving a higher number of IDONTWANT requests.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="key-considerations-and-poc-implementation">Key Considerations and <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_pppt" target="_blank" rel="noopener noreferrer">PoC Implementation</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#key-considerations-and-poc-implementation" class="hash-link" aria-label="Direct link to key-considerations-and-poc-implementation" title="Direct link to key-considerations-and-poc-implementation">​</a></h4>
<p>The use of hop count is a more effective and straightforward method for identifying the message propagation stage
than relying on the duplicate count.
However, this approach may compromise the publisher's anonymity and reveal information about the publisher and its mesh members.
Additional due diligence may be needed to address these issues.</p>
<p>In the PoC implementation [<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_pppt" target="_blank" rel="noopener noreferrer">15</a>],
we use hop count to determine the message propagation stage.
When forwarding a message, every peer selects a subset of mesh members
equal to the advertised hop count for pull operation and
forwards the message to the remaining mesh members.
If the advertised hop count exceeds the number of mesh members chosen for message forwarding,
the sender relays the message to all selected peers.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gossipsub-v14-proposal"><a href="https://github.com/libp2p/specs/pull/654" target="_blank" rel="noopener noreferrer">GossipSub v1.4 Proposal</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#gossipsub-v14-proposal" class="hash-link" aria-label="Direct link to gossipsub-v14-proposal" title="Direct link to gossipsub-v14-proposal">​</a></h3>
<p>GossipSub v1.4 proposal considers longer large-message transfer times (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>) as contention intervals and
argues that most duplicates occur during these intervals for two fundamental reasons:</p>
<ol>
<li>
<p>Peers are unaware of ongoing message receptions and
may generate redundant IWANT requests for messages they are already receiving.</p>
</li>
<li>
<p>Peers can send IDONTWANT announcements only after receiving the entire message.
However, a large contention interval increases the likelihood that
many redundant transmissions will start before IDONTWANT messages are issued.</p>
</li>
</ol>
<p>GossipSub v1.4 proposal eliminates contention interval with help from two new control messages:
PREAMBLE and IMRECEIVING.
A PREAMBLE precedes every large message transmission.
Upon receiving a preamble, a peer learns about the messages it is receiving and performs two actions:</p>
<ol>
<li>
<p>Notify mesh members about ongoing message reception using an IMRECEIVING announcement.
On receiving an IMRECEIVING announcement from a peer,
mesh members defer sending the announced message to that peer.</p>
</li>
<li>
<p>Defer IWANT requests for messages that are currently being received.
Peers also limit the outstanding IWANT requests for any message to one.</p>
</li>
</ol>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="key-considerations-and-poc-implementation-1">Key Considerations and <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v1_4" target="_blank" rel="noopener noreferrer">PoC Implementation</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#key-considerations-and-poc-implementation-1" class="hash-link" aria-label="Direct link to key-considerations-and-poc-implementation-1" title="Direct link to key-considerations-and-poc-implementation-1">​</a></h4>
<p>The use of PREAMBLE/IMRECEIVING addresses the limitation of IDONTWANT messages.
For instance, consider a peer <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> begins receiving a message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> at time <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>t</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">t_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">t</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.
It can transmit IDONTWANT only after receiving <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span>, i.e., at time <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>t</mi><mn>1</mn></msub><mo>+</mo><msub><mi>τ</mi><mi>D</mi></msub></mrow><annotation encoding="application/x-tex">t_1+\tau_D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">t</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.
Therefore, it can not cancel any duplicate receptions of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> that start before <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><mn>1</mn><mo>+</mo><msub><mi>τ</mi><mi>D</mi></msub><mo>+</mo><msub><mi>τ</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">t1 + \tau_D + \tau_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">t</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.
In contrast, IMRECEIVING announcements for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> start at <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>t</mi><mn>1</mn></msub><mo>+</mo><mi mathvariant="normal">Δ</mi></mrow><annotation encoding="application/x-tex">t_1 + \Delta</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">t</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Δ</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Δ</mi></mrow><annotation encoding="application/x-tex">\Delta</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Δ</span></span></span></span> denotes PREAMBLE processing time and satisfies <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Δ</mi><mo>≪</mo><msub><mi>τ</mi><mi>D</mi></msub></mrow><annotation encoding="application/x-tex">\Delta \ll \tau_D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord">Δ</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≪</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.
As a result, peer <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> can eliminate all duplicate receptions of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> that start after <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>t</mi><mn>1</mn></msub><mo>+</mo><mi mathvariant="normal">Δ</mi><mo>+</mo><msub><mi>τ</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">t_1 + \Delta +\tau_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">t</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord">Δ</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>,
which noticeably reduces duplicates.</p>
<p>The use of PREAMBLE also allows deferring IWANT requests for messages we are already receiving,
which can also improve message dissemination time by
reducing the workload on peers along the optimal message forwarding path.</p>
<p>It is worth mentioning that a malicious peer can exploit this approach by sending a PREAMBLE and
never completing (or deliberately delaying) the promised message transfer.
The optional safety strategy in GossipSub v1.4 proposal suggests using a peer score threshold
for PREAMBLE processing and a behavior penalty for broken promises.
A timeout strategy helps recover such messages.</p>
<p>It is essential to mention that sending and processing of PREAMBLE and IMRECEIVING messages is optional.
This flexibility allows for the use of custom safety strategies in various implementations.
For example, the ongoing production-grade implementation of GossipSub v1.4 in nim-libp2p
allows peers to ignore PREAMBLEs unless they come from mesh members with higher data rates
(bandwidth estimation becomes trivial with PREAMBLEs) and good peer scores.
This implementation also lets peers choose between a push or pull strategy for handling broken promises.</p>
<p>For the performance evaluations in this post, we utilize the PoC implementation of GossipSub v1.4
[<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v1_4" target="_blank" rel="noopener noreferrer">16</a>].
A complete, production-grade version is currently undergoing testing and validation
[<a href="https://github.com/vacp2p/nim-libp2p/pull/1448" target="_blank" rel="noopener noreferrer">17</a>].</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gossipsub-v20-proposal"><a href="https://github.com/libp2p/specs/pull/653" target="_blank" rel="noopener noreferrer">GossipSub v2.0 Proposal</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#gossipsub-v20-proposal" class="hash-link" aria-label="Direct link to gossipsub-v20-proposal" title="Direct link to gossipsub-v20-proposal">​</a></h3>
<p>GossipSub v2.0 introduces a hybrid method for message dissemination
that combines both push and pull strategies through two new control messages: IANNOUNCE and INEED.
These messages are analogous to IHAVE and IWANT messages, respectively.
However, IANNOUNCE messages are issued to the mesh members
immediately after validating a received message without waiting for the heartbeat interval.
Similarly, INEED requests are made exclusively to mesh members,
and a peer generates only one INEED request for a received message.</p>
<p>The balance between push and pull approaches is determined by the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{announce}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> parameter.
On receiving a message, a peer forwards it to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo>−</mo><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D - D_{announce}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> mesh members and
sends IANNOUNCE messages to the remaining mesh peers.
On receiving an IANNOUNCE for an unseen message,
a peer can request it using an INEED message.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="key-considerations-and-poc-implementation-2">Key Considerations and <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v2_0" target="_blank" rel="noopener noreferrer">PoC Implementation</a><a href="https://vac.dev/rlog/gsub-perf-imp-comparison#key-considerations-and-poc-implementation-2" class="hash-link" aria-label="Direct link to key-considerations-and-poc-implementation-2" title="Direct link to key-considerations-and-poc-implementation-2">​</a></h4>
<p>The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{announce}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> parameter governs the balance between push and pull operations.
Setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub><mo>=</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">D_{announce} = D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> results in a pull-only operation,
which can eliminate duplicates at the cost of increased message dissemination time.
In contrast, setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{announce}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to zero reverts to standard GossipSub v1.2 operation.
The authors suggest setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub><mo>=</mo><mi>D</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">D_{announce} = D-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> to moderately decrease dissemination time
while incurring only a small number of duplicate transmissions.</p>
<p>It is important to note that malicious peers can exploit this approach
by delaying or entirely omitting responses to INEED requests.
Similarly, sending INEED requests to suboptimal or overwhelmed peers can further increase message dissemination time.
The authors propose using a timeout strategy and negative peer scoring to address these issues.
If a message transfer does not complete within the specified interval,
the receiver decreases the sender's peer score
and issues a new INEED request to an alternative mesh member.</p>
<p>For the performance evaluations in this post,
we utilize the PoC implementation of GossipSub v2.0
[<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v2_0" target="_blank" rel="noopener noreferrer">18</a>] from nim-libp2p.
We set <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub><mo>=</mo><mi>D</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">D_{announce} = D-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> and allow any peer to send a single IWANT request for a message,
only if it has not previously sent an INEED request for the same message.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="experiments">Experiments<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#experiments" class="hash-link" aria-label="Direct link to Experiments" title="Direct link to Experiments">​</a></h2>
<p>We conducted a series of experiments under various configurations to evaluate the performance of
the GossipSub v1.4 proposal, the PPPT approach, and the GossipSub v2.0 proposal
against the baseline GossipSub v1.2 protocol.
To support these evaluations, we extended the nim-libp2p implementation to include minimal PoC implementations of the considered protocols
[<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v1_4" target="_blank" rel="noopener noreferrer">16</a>,
<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_pppt" target="_blank" rel="noopener noreferrer">15</a>,
<a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v2_0" target="_blank" rel="noopener noreferrer">18</a>]<br>
<!-- -->and used the Shadow simulator [<a href="https://github.com/vacp2p/dst-gossipsub-test-node/pull/6" target="_blank" rel="noopener noreferrer">19</a>]
to carry out performance evaluations.</p>
<p>For GossipSub v1.4 and PPPT, we also report results from delayed forwarding,
where peers introduce a short delay before relaying a message to every mesh member.
This delay helps reduce the number of redundant transmissions
by increasing the likelihood of timely receiving a higher number of IDONTWANT notifications.</p>
<p>We evaluate performance using network-wide message dissemination time (latency),
network-wide bandwidth utilization (bandwidth),
and the average number of duplicates received by a peer for every transmitted message.
We also report the average number of IWANT requests transmitted by a peer for a single message.</p>
<p>For each experiment, we transmit multiple messages in the network.
We average the network-wide dissemination time for these messages to report latency.
Bandwidth refers to the total volume of traffic in the network,
encompassing control messages and data transmissions (including duplicates and IWANT replies).
A peer usually receives multiple copies of any transmitted message.
Excluding the first received message, all copies are duplicates.
We compute average duplicates received by a peer as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi>N</mi><mi>M</mi></mrow></mfrac><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>M</mi></msubsup><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><msub><mi>d</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>j</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\frac{1}{N M} \sum_{j=1}^{M} \sum_{i=1}^{N} d_{i,j}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.417em;vertical-align:-0.4358em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8451em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">NM</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.9812em"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.2029em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">M</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.4358em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.9812em"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.2029em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">N</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2997em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> denote the number of peers and the number of transmitted messages, respectively,
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>j</mi></mrow></msub></mrow><annotation encoding="application/x-tex">d_{i,j}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9805em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> represents the number of duplicates received by peer <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span> for message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi></mrow><annotation encoding="application/x-tex">j</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.05724em">j</span></span></span></span>.
A similar mechanism computes average IWANT requests.</p>
<p>Three simulation scenarios are considered:</p>
<ul>
<li>
<p><strong>Scenario 1:</strong> The number of publishers and message size are kept constant while the network size gradually increases.</p>
</li>
<li>
<p><strong>Scenario 2:</strong> The number of publishers and the network size remain constant while the message size gradually increases.</p>
</li>
<li>
<p><strong>Scenario 3:</strong> The number of nodes and message size remain constant while the number of publishers gradually increases.</p>
</li>
</ul>
<p>In all experiments, we transmit multiple messages such that
every publisher sends exactly one message to the network.
After a publisher transmits a message,
each subsequent publisher waits for a specified interval (inter-message delay) before sending the next message.</p>
<p>Rotating publishers ensures that every message traverses a different path,
which helps achieve fair performance evaluation.
On the other hand, changing inter-message delays allows for creating varying traffic patterns.
A shorter inter-message delay implies more messages can be in transit simultaneously,
which helps evaluate performance against large message counts.
A longer delay ensures every message is fully disseminated before introducing a new message.
Similarly, increasing message size stresses the network.
As a result, we evaluate performance across a broader range of use cases.</p>
<p>The simulation details are presented in the tables below.
The experiments are conducted using the shadow simulator.
We uniformly set peer bandwidths and link latencies between 40-200 Mbps and
40-130 milliseconds in five variations.</p>
<p><strong>Table 1:</strong> Simulation Scenarios.</p>
<table><thead><tr><th style="text-align:center"><strong>Experiment</strong></th><th style="text-align:center"><strong>No. of Nodes</strong></th><th style="text-align:center"><strong>No. of Publishers</strong></th><th style="text-align:center"><strong>Message Size (KB)</strong></th><th style="text-align:center"><strong>Inter-Message Delay (ms)</strong></th></tr></thead><tbody><tr><td style="text-align:center">Scenario 1</td><td style="text-align:center">3000, 6000, 9000, 12000</td><td style="text-align:center">7</td><td style="text-align:center">150</td><td style="text-align:center">10000</td></tr><tr><td style="text-align:center">Scenario 2</td><td style="text-align:center">1500</td><td style="text-align:center">10</td><td style="text-align:center">200, 600, 1000, 1400, 1800</td><td style="text-align:center">10000</td></tr><tr><td style="text-align:center">Scenario 3</td><td style="text-align:center">1500</td><td style="text-align:center">25, 50, 75, 100, 125</td><td style="text-align:center">50</td><td style="text-align:center">50</td></tr></tbody></table>
<p><strong>Table 2:</strong> Simulation Parameters.</p>
<table><thead><tr><th style="text-align:center"><strong>Parameter</strong></th><th style="text-align:center"><strong>Value</strong></th><th style="text-align:center"><strong>Parameter</strong></th><th style="text-align:center"><strong>Value</strong></th></tr></thead><tbody><tr><td style="text-align:center"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></td><td style="text-align:center">8</td><td style="text-align:center"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{low}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></td><td style="text-align:center">6</td></tr><tr><td style="text-align:center"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>l</mi><mi>a</mi><mi>z</mi><mi>y</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{lazy}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">zy</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></td><td style="text-align:center">6</td><td style="text-align:center"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{high}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">hi</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></td><td style="text-align:center">12</td></tr><tr><td style="text-align:center">Gossip factor</td><td style="text-align:center">0.05</td><td style="text-align:center">Muxer</td><td style="text-align:center">yamux</td></tr><tr><td style="text-align:center">Heartbeat interval</td><td style="text-align:center">1000 ms</td><td style="text-align:center">Floodpublish</td><td style="text-align:center">false</td></tr><tr><td style="text-align:center">Peer Bandwidth</td><td style="text-align:center">40-200 Mbps</td><td style="text-align:center">Link Latency</td><td style="text-align:center">40-130ms</td></tr><tr><td style="text-align:center"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>n</mi><mi>n</mi><mi>o</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{announce}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">ann</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight">u</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">ce</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> GossipSub v2.0</td><td style="text-align:center">7</td><td style="text-align:center">Forward delay in PPPT/v1.4 with delay</td><td style="text-align:center">35 ms</td></tr></tbody></table>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="results">Results<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#results" class="hash-link" aria-label="Direct link to Results" title="Direct link to Results">​</a></h3>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="scenario1-increasing-network-size">Scenario1: Increasing Network Size<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#scenario1-increasing-network-size" class="hash-link" aria-label="Direct link to Scenario1: Increasing Network Size" title="Direct link to Scenario1: Increasing Network Size">​</a></h4>
<table><thead><tr><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Bandwidth" src="https://vac.dev/assets/images/S1_BW-b2ab12abebae802d40990001fbd11a67.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Latency" src="https://vac.dev/assets/images/S1_Lat-e1f0dca0457b1a0e60038589c73a7544.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Duplicates" src="https://vac.dev/assets/images/S1_Dup-6d051db936b98e3285e641bd998ad9f3.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="IWANTs" src="https://vac.dev/assets/images/S1_IWANT-8a6ae9d5cda1f492bb511d16a3036166.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead><tbody><tr><td style="text-align:center">Bandwidth</td><td style="text-align:center">Latency</td><td style="text-align:center">Average Duplicates</td><td style="text-align:center">Average IWANT Requests</td></tr></tbody></table>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="scenario2-increasing-message-size">Scenario2: Increasing Message Size<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#scenario2-increasing-message-size" class="hash-link" aria-label="Direct link to Scenario2: Increasing Message Size" title="Direct link to Scenario2: Increasing Message Size">​</a></h4>
<table><thead><tr><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Bandwidth" src="https://vac.dev/assets/images/S2_BW-ea495186217e35c35b96f6318d319bfe.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Latency" src="https://vac.dev/assets/images/S2_Lat-192eb6296e214f4c6359e67d24b63d36.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Duplicates" src="https://vac.dev/assets/images/S2_Dup-2efbb1f012d4e198b825adbeaa1d59fd.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="IWANTs" src="https://vac.dev/assets/images/S2_IWANT-fa39d5991998424426f512c35e3c9ca7.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead><tbody><tr><td style="text-align:center">Bandwidth</td><td style="text-align:center">Latency</td><td style="text-align:center">Average Duplicates</td><td style="text-align:center">Average IWANT Requests</td></tr></tbody></table>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="scenario3-increasing-number-of-publishers">Scenario3: Increasing Number of Publishers<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#scenario3-increasing-number-of-publishers" class="hash-link" aria-label="Direct link to Scenario3: Increasing Number of Publishers" title="Direct link to Scenario3: Increasing Number of Publishers">​</a></h4>
<table><thead><tr><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Bandwidth" src="https://vac.dev/assets/images/S3_BW-15887305919bb1540bb305c504f99faa.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Latency" src="https://vac.dev/assets/images/S3_Lat-6384468498a08f0c2ae8d73e65261c40.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Duplicates" src="https://vac.dev/assets/images/S3_Dup-988eff466b6e2e92de737c28f07d9fcc.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th style="text-align:center"><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="IWANTs" src="https://vac.dev/assets/images/S3_IWANT-e90d4ff755c4cfe0b9aad899d679dac9.png" width="1982" height="1058" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead><tbody><tr><td style="text-align:center">Bandwidth</td><td style="text-align:center">Latency</td><td style="text-align:center">Average Duplicates</td><td style="text-align:center">Average IWANT Requests</td></tr></tbody></table>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="findings">Findings<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#findings" class="hash-link" aria-label="Direct link to Findings" title="Direct link to Findings">​</a></h2>
<ul>
<li>
<p>The number of IWANT requests increases with the message size.
Limiting ongoing IWANT requests for each message to one can be beneficial.
Additionally, the use of message PREAMBLEs can help eliminate IWANT requests
for messages that are already being received.</p>
</li>
<li>
<p>Pull-based approaches can substantially reduce bandwidth utilization,
but may result in much longer message dissemination times.
However, these approaches can achieve simultaneous propagation of multiple messages
by implicitly rotating among outgoing messages.
As a result, increasing the number of messages yields similar dissemination times.</p>
</li>
<li>
<p>Transition from push to pull operation during the later stages of message propagation can reduce bandwidth consumption,
without compromising latency.
However, determining the propagation stage is challenging.
Methods like hop counts may compromise anonymity,
while using IHAVE announcements can be misleading. For instance, in the case of large messages,
peers may receive IHAVE announcements much earlier than the actual message spreads through the network.</p>
</li>
<li>
<p>Push-based approaches achieve the fastest message dissemination
but often produce a higher number of duplicate messages.
Employing mechanisms like PREAMBLE/IMRECEIVING messages
for guided elimination of duplicate messages can significantly reduce bandwidth consumption.
This reduction not only minimizes redundant transmissions
but also decreases the overall message dissemination time
by lessening the workload on peers located along optimal message forwarding paths.</p>
</li>
</ul>
<p>Please feel free to join the discussion and leave feedback regarding this post in the
<a href="https://forum.vac.dev/t/vac-research-blog-performance-evaluation-of-gossipsub-improvement-proposals/556" target="_blank" rel="noopener noreferrer">VAC forum</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/gsub-perf-imp-comparison#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<p>[1] <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md" target="_blank" rel="noopener noreferrer">GossipSub v1.1 Specifications</a></p>
<p>[2] <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md" target="_blank" rel="noopener noreferrer">GossipSub v1.2 Specifications</a></p>
<p>[3] <a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921" target="_blank" rel="noopener noreferrer">Number of Duplicate Messages in Ethereum’s Gossipsub Network</a></p>
<p>[4] <a href="https://ethresear.ch/t/impact-of-idontwant-in-the-number-of-duplicates/22652" target="_blank" rel="noopener noreferrer">Impact of IDONTWANT in the Number of Duplicates</a></p>
<p>[5] <a href="https://www.arxiv.org/abs/2505.17337" target="_blank" rel="noopener noreferrer">PREAMBLE and IMRECEIVING for Improved Large Message Handling</a></p>
<p>[6] <a href="https://arxiv.org/abs/2504.10365" target="_blank" rel="noopener noreferrer">Staggering and Fragmentation for Improved Large Message Handling</a></p>
<p>[7] <a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw" target="_blank" rel="noopener noreferrer">GossipSub for Big Messages</a></p>
<p>[8] <a href="https://ethresear.ch/t/fulldas-towards-massive-scalability-with-32mb-blocks-and-beyond/19529" target="_blank" rel="noopener noreferrer">FullDAS: Towards Massive Scalability with 32MB Blocks and Beyond</a></p>
<p>[9] <a href="https://github.com/libp2p/specs/pull/681" target="_blank" rel="noopener noreferrer">Choke Extension for GossipSub</a></p>
<p>[10] <a href="https://ethresear.ch/t/pppt-fighting-the-gossipsub-overhead-with-push-pull-phase-transition/22118/1" target="_blank" rel="noopener noreferrer">PPPT: Fighting the GossipSub Overhead with Push-Pull Phase Transition</a></p>
<p>[11] <a href="https://github.com/libp2p/specs/pull/654" target="_blank" rel="noopener noreferrer">GossipSub v1.4 Specifications Proposal</a></p>
<p>[12] <a href="https://github.com/libp2p/specs/pull/653" target="_blank" rel="noopener noreferrer">GossipSub v2.0 Specifications Proposal</a></p>
<p>[13] <a href="https://github.com/vacp2p/nim-libp2p" target="_blank" rel="noopener noreferrer">Libp2p Implementation in nim</a></p>
<p>[14] <a href="https://streamr-public.s3.amazonaws.com/streamr-network-scalability-whitepaper-2020-08-20.pdf" target="_blank" rel="noopener noreferrer">The Streamr Network: Performance and Scalability</a></p>
<p>[15] <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_pppt" target="_blank" rel="noopener noreferrer">PPPT: PoC Implementation in nim-libp2p</a></p>
<p>[16] <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v1_4" target="_blank" rel="noopener noreferrer">GossipSub v1.4: PoC Implementation in nim-libp2p</a></p>
<p>[17] <a href="https://github.com/vacp2p/nim-libp2p/pull/1448" target="_blank" rel="noopener noreferrer">GossipSub v1.4: Production-Grade Implementation in nim-libp2p</a></p>
<p>[18] <a href="https://github.com/vacp2p/nim-libp2p/tree/research_gs_v2_0" target="_blank" rel="noopener noreferrer">GossipSub v2.0: PoC Implementation in nim-libp2p</a></p>
<p>[19] <a href="https://github.com/vacp2p/dst-gossipsub-test-node/pull/6" target="_blank" rel="noopener noreferrer">nim-libp2p GossipSub Test Node</a></p>]]></content>
<author>
<name>Umar Farooq</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Zerokit optimizations: A performance journey]]></title>
<id>https://vac.dev/rlog/2025-zerokit-perf</id>
<link href="https://vac.dev/rlog/2025-zerokit-perf"/>
<updated>2025-07-25T18:30:00.000Z</updated>
<summary type="html"><![CDATA[Zerokit is a toolkit]]></summary>
<content type="html"><![CDATA[<p><a href="https://github.com/vacp2p/zerokit/" target="_blank" rel="noopener noreferrer"><u>Zerokit</u></a> is a toolkit
providing powerful zero-knowledge utilities, including a
means to answer the question "How do you prevent spam when
every message is anonymous?". Its use of the Merkle hash
tree, combined Poseidon hasher are keys to the answer we
seek here, and with other questions that ask the improbable.
These technologies, however, can take a heavy toll on
resources if not used correctly. What follows is a window
into the efforts made to squeeze out optimizations, and
culling of redundant resource use. A story of cripplingly
slow performance meets engineering talent, we arrive at a
place where Zerokit comes through, fast and efficient, ready
to face the world.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="background">Background<a href="https://vac.dev/rlog/2025-zerokit-perf#background" class="hash-link" aria-label="Direct link to Background" title="Direct link to Background">​</a></h2>
<p>Our friends over at <a href="https://free.technology/waku" target="_blank" rel="noopener noreferrer"><u>Waku</u></a> are
particularly enthusiastic about anonymous spam prevention
technologies. They have been using the Rate Limiting
Nullifier (<a href="https://crates.io/crates/rln" target="_blank" rel="noopener noreferrer"><u>RLN</u></a>) tooling that
Zerokit provides to enforce a message-rate policy among
users—a crucial feature unless we want a community bombarded
with "totally legit, not scams" messages on repeat. However,
as is often the case with new technology, some problematic
delays began to surface. Node recalculations, a common
operation, were taking tens of seconds at the scales being
tested and deployed—even exceeding 40 seconds at times.
These delays accumulate, leading to total delays on the
order of three hours under certain conditions.</p>
<p>Naturally, we couldn't just let this sit. While we've
touched on the issue of spam prevention, it's important to
recognize that this technology is foundational that
challenges conventional wisdom on how things must be done.
Does the idea of "smart contracts without gas" catch your
attention? Don't hold your breath just yet: the really
interesting applications of this tech will be dead in the
water, unless we can meet the challenge put to us.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-challenge">The Challenge<a href="https://vac.dev/rlog/2025-zerokit-perf#the-challenge" class="hash-link" aria-label="Direct link to The Challenge" title="Direct link to The Challenge">​</a></h2>
<p>The plan of attack that the team put together was twofold:
get rid of redundant operations and data taking up precious
resources, and make the remaining operations go
<a href="https://old.reddit.com/r/rust/comments/1avf1d8/blazingly_fast_memory_vulnerabilities_written_in/" target="_blank" rel="noopener noreferrer"><u><em>Blazingly Fast™</em>.</u></a></p>
<p>Introducing the star of the show for part 1: The main
point of having this tree is to generate proofs so that
peers can verify the claims being made. That doesn’t require
the whole Merkle tree, just a single path, from leaf to
root. The engineering work took us in a direction where
these paths were the primary context in which ZK proofs
operated, relegating the tree itself to an off-network
reference. This reduced the burden imposed on the network
significantly. Updating the data on the tree has similarly
reduced, with the exception being that the siblings of each
node were retained. This is called the <strong>stateless</strong>
approach.</p>
<p>Well, stateless in the context of proof generation and
verification. This is the critical context when it comes to
performance, and the stateless approach does a great job,
but these proofs have to come from <em>somewhere</em>. Each
participant still needs to maintain the Merkle tree in their
local environment. Without this tree, one cannot generate
proofs or verify the proofs provided to them. Fortunately,
one does not need to track the entire tree, but can be
limited to a subset of the tree needed. With millions of
participants on-chain, this can make the difference needed
to make Zerokit empowered technologies accessible to those
running raspberry Pis. Combine this with the fact that the
heavy lifting operations of proof gen/verification being
modular and separate, each participant can optimise to run
things according to the strengths and requirements of their
native hardware, easing the way to allow each participants
to run their tree implementation at the speed of
mach-ludicrous.</p>
<p>Fortunately, the core of our already existing implementation
was sane and well put together. Double-lucky for us, the
talents of newly minted VAC/ACZ team members Sylvain and Vinh were
readily available. Sylvain, with a solid background in the Rust
programming language, having already graduated from the most
challenging parts of its infamous learning curve. He quickly
got to work zeroing in on some subtle performance pathologies.
Something as simple as using a mutable iterator to change
values directly. Clever use of references to avoid copying
data, and other memory optimization techniques that can be
hidden to those that cannot “see the matrix” when working in
Rust lead to very promising bench-marking results.</p>
<p>Vinh, having recently graduated from his CS studies, was presented
with the challenge of parrelising computations. For those not
familiar with Rust, this might seem unreasonable, but thanks
to the <code>rayon</code> crate, and Rusts promise of "fearless concurrency"
afforded by its type and ownership system, this kind of refactor
becomes surprisingly easy, even for a talented individual at
the start of their career. Of particular note: These parallelisations
have been made available to the browser. Browser threads are
relatively now, and by diving into this bleeding-edge technology,
and making use of libraries that are still in early development
stages, <em>Blazingly Fast™</em> is now available within the browser.
With all that in the bag, all these performance gains are
gift-wrapped in the use of browser-native WASM runtimes.</p>
<p>Well done, everyone!</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-importance-of-benchmarks">The importance of benchmarks<a href="https://vac.dev/rlog/2025-zerokit-perf#the-importance-of-benchmarks" class="hash-link" aria-label="Direct link to The importance of benchmarks" title="Direct link to The importance of benchmarks">​</a></h2>
<p>No performance project is complete without high quality
benchmark data. Writing a quick benchmark for tracking
improvements through development is one thing, but having a
system of telemetry that allows you to credibly assert
claims of superior performance is what completes the
project. With such credible claims in hand, these efforts
can bring about significant impact on the field at large.
The key word being <strong>credible</strong>. Credibility cannot depend
on “trust me bro” (obviously). The truth of these claims
must come out of the directed efforts of a multitude of
thought-disciplines. The engineer must have a solid model to
understand the nature of the system. The statistician sets
the quality standards of the data. The Scientist must
diligently put relevant hypothesis to the test. The advocate
must see that the reports made reach out to where it makes
the most impact, the list goes on. Much of this is out of
scope for this article, and so I will treat you with
<a href="https://www.youtube.com/watch?v=qUN4Tln608Q&amp;list=PLtoQeavghzr3nlXyJEXaTLU9Ca0DXWMnt" target="_blank" rel="noopener noreferrer"><u>a link</u></a>.
Here’s your chance to see a hardcore OS engineer at the top
of their chosen field speak on the subject of their passion.</p>
<p>All this is to say we are not the only team implementing
Merkle tree tech, which also includes the Poseidon hash
function it needs. In order to be a premier research
destination, key aspect of why VAC exists, the fruits of our
labor is just the beginning. We must prove the merit of our
efforts through comparative benchmarks that satisfies the
skeptics and decision makers.</p>
<p>Comparative benchmarks are among the most high-stakes
element of performance critical projects. Get it right, and
quality research output can become industry standard
technology. Get it wrong, and be ready to lose the trust the
field has in you as your precious R&amp;D fades into obscurity.</p>
<p>For the time being, our comparative benchmarks have been
used internally to inform decision-makers. As benchmarks
become standardised, independently verified and executed,
this initial effort may be the first of many steps to a
community-wide ecosystem. A thunderdome of benchmarks,
leaving us with a single champion that cannot be denied, but
which technology will claim this mantle? May the bits be
<em>ever</em> in your favor...</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="benchmarking-with-rusts-criterion-crate">Benchmarking with Rust's criterion crate<a href="https://vac.dev/rlog/2025-zerokit-perf#benchmarking-with-rusts-criterion-crate" class="hash-link" aria-label="Direct link to Benchmarking with Rust's criterion crate" title="Direct link to Benchmarking with Rust's criterion crate">​</a></h2>
<p>Rust, much like Nim, offers unadulterated, fine-grained, and
direct control over performance, but with Rust, this control
is even more immediate. With its sophisticated ownership
model, powerful type system, and comprehensive tooling, Rust
has earned an unrivaled reputation for enabling "fearless
concurrency," ease of refactoring, and providing tooling
that effectively "pair programs with you" to help avoid
common pitfalls, includeing those of the performance
veriety.</p>
<p>The <a href="https://crates.io/crates/criterion" target="_blank" rel="noopener noreferrer"><u>criterion</u></a> crate is
considered the go-to library for micro-benchmarking within
the Rust ecosystem, and is generally regarded as an
invaluable tool for obtaining high-quality telemetry.
Through its ergonomic idioms and well-thought-out API,
writing high-quality benchmarks becomes straightforward once
you become familiar with its features. Criterion helps avoid
common traps such as inappropriate compiler optimizations,
improper performance sampling, and failing to prune
telemetry overhead. As is typical for the Rust ecosystem,
the documentation is thorough, leaving little to be desired,
and the examples are extensive, making the initial learning
process a pleasant experience.</p>
<p>Most importantly, it automatically generates tables and
graphs from this data, making the crucial task of analysis
straightforward and accessible. At this point, we are ready
to appreciate the results of our efforts.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="promising-results">Promising results<a href="https://vac.dev/rlog/2025-zerokit-perf#promising-results" class="hash-link" aria-label="Direct link to Promising results" title="Direct link to Promising results">​</a></h2>
<p>When it comes to Merkle trees, we have two elements to
consider: The tree itself, and the hashing function that is
plugged into it. In the benchmarks we put together for the
benefit of internal processes, we put our implementation up
against a corresponding FOSS implementation. Scenarios were
developed to isolate key performance telemetry, obtain a
statistically usable sampling, with the resulting data
rendered into a human readable form that can be read with a
reasonable degree of confidence: enjoy! The brief summary:
It appears that our in house implementation consistently
outperforms others, and we’ve decided to continue committing
to the R&amp;D of our in-house implementations. Congratulations
to the Zerokit team for this accomplishment.</p>
<p>Despite the promising results, these “micro-benchmarks” form
just some of the many pieces of the whole system performance
when it comes to product needs. How the system performs as a
whole is all that matters. This is a promising on it’s own,
but watching the performance benefits being realized in the
wild is the true goal.</p>
<p>Which brings us back to what started all this: Waku came to
us with concerns about performance issues within Zerokit
limiting the scope and scale in which it can be used. The
engineering talent brought to bear on this issue has
successfully achieved the performance goals needed, and the
results of these effort have demonstrated there is merit in
continuing our commitment to this project.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/2025-zerokit-perf#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>We’ve covered a story that starts with crippling performance
bottlenecks in Waku, and ends on this high-note: The
problematic performance scaling issues are no more, and in
the process of resolving this critical pain-point, we have
established internal benchmarks that allow us to confidently
state that what we are doing, we are doing well. These
accomplishments come down to a solid team effort. The open
communication coming in from Waku, the talented engineers
working together to bring their skills and contributions to
bear, the community developed tools and prior works that
allowed it all to happen, and those working quietly in the
background providing the leadership, resources, and
coordination needed to bring this all together. Two VAC/ACZ
engineers in particular call for specific mention:
<a href="https://github.com/seemenkina" target="_blank" rel="noopener noreferrer"><u>Ekaterina</u></a> for her role in
taking lead in the R&amp;D of the Zerokit ecosystem, and
<a href="https://github.com/sydhds" target="_blank" rel="noopener noreferrer"><u>Sylvain</u></a> for his efforts in
squeezing out some impressive optimizations.
<a href="https://github.com/vinhtc27" target="_blank" rel="noopener noreferrer"><u>Vinh</u></a> for unleashing the power of multiple threads, not
only for native, but for when running in the browser as well.</p>
<p>Perhaps you want to get involved! Maybe you have some ideas
about what the community needs for standard benchmarks.
Would you like to see another implementation added to the
thunderdome?
<a href="https://github.com/vacp2p/zerokit/issues/new" target="_blank" rel="noopener noreferrer"><u>Raise an issue</u></a>,
or join us on <a href="https://forum.vac.dev/" target="_blank" rel="noopener noreferrer"><u>our forum</u></a>. We look
forward to seeing your voice added.</p>
<p>This is just one story, coming out of one relatively small
project from VAC research. The two driving directives of the
team is to be a conduit of expertise within IFT, and to be a
premier research destination within the domains we work in.
You might be independent of IFT with an interest in what we
do, an IFT core contributor, or anything in between: our
services are at your disposal. Join us on discord to start
the conversation, email one of our team members, or maybe
you might hear a knock on your door, should something in
your field of work catch our interest.</p>]]></content>
<author>
<name>BenPH</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Nim in Logos - 1st Edition]]></title>
<id>https://vac.dev/rlog/nim-in-logos-01</id>
<link href="https://vac.dev/rlog/nim-in-logos-01"/>
<updated>2025-07-04T23:00:00.000Z</updated>
<summary type="html"><![CDATA[Welcome to the first edition of Nim in Logos — a newsletter covering major Nim features from Logos' perspective.]]></summary>
<content type="html"><![CDATA[<p>Welcome to the first edition of <strong>Nim in Logos</strong> — a newsletter covering major Nim features from Logos' perspective.</p>
<p>If you have comments or suggestions, feel free to reach out to the authors directly or start a thread in the <a href="https://discord.gg/logosnetwork" target="_blank" rel="noopener noreferrer">Logos Discord server</a>.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nim-22---better-stability-smarter-memory-and-smoother-development">Nim 2.2 – Better Stability, Smarter Memory, and Smoother Development<a href="https://vac.dev/rlog/nim-in-logos-01#nim-22---better-stability-smarter-memory-and-smoother-development" class="hash-link" aria-label="Direct link to Nim 2.2 – Better Stability, Smarter Memory, and Smoother Development" title="Direct link to Nim 2.2 – Better Stability, Smarter Memory, and Smoother Development">​</a></h2>
<p>The Nim 2.2 release series focuses on improving language stability, fixing long-standing bugs, and optimizing performance—particularly in the ORC memory management system. The latest patch in this series, version 2.2.4, continues to build on these goals.</p>
<p>Here are some of the key highlights from the 2.2 series:</p>
<ul>
<li><strong>More powerful generics and type expressions:</strong> Stabilization of generics, typedesc, and static types. These features now support arbitrary expressions that previously only worked in limited cases, making them more reliable.</li>
<li><strong>Improved tuple unpacking:</strong> Tuple unpacking now supports discarding values using underscores (_) and allows inline type annotations for unpacked elements.</li>
<li><strong>Memory leak fixes:</strong> Issues with memory leaks when using std/nre’s regular expressions or nested exceptions have been resolved.</li>
<li><strong>More efficient async code:</strong> Futures no longer always copy data, resulting in better performance in asynchronous workflows.</li>
<li><strong>String bug fixes:</strong> Several issues involving string and cstring usage have been corrected.</li>
</ul>
<p>In addition to core language improvements:</p>
<ul>
<li><strong>NimSuggest stability:</strong> The language server has received multiple fixes, improving the experience in IDEs and editors that rely on NimSuggest for autocompletion and error checking.</li>
<li><strong>Better code generation:</strong> Numerous issues related to invalid or broken C and C++ code generation and backend-specific bugs have been addressed, improving Nim’s reliability when targeting other languages.</li>
</ul>
<p>You can read the full release announcement and changelog <a href="https://nim-lang.org/blog/2024/10/02/nim-220-2010.html" target="_blank" rel="noopener noreferrer">here</a></p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="error-handling-in-nim-why-results-beat-exceptions">Error Handling in Nim: Why Results Beat Exceptions<a href="https://vac.dev/rlog/nim-in-logos-01#error-handling-in-nim-why-results-beat-exceptions" class="hash-link" aria-label="Direct link to Error Handling in Nim: Why Results Beat Exceptions" title="Direct link to Error Handling in Nim: Why Results Beat Exceptions">​</a></h2>
<p>Error handling is one of the most critical aspects of writing reliable software, yet it remains a contentious topic in many programming languages. In Nim, developers face a unique challenge: multiple error handling paradigms are supported, leading to confusion about which approach to choose. For robust, maintainable code, our answer at Logos is increasingly clear—favor Result types over exceptions.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-exception-problem">The Exception Problem<a href="https://vac.dev/rlog/nim-in-logos-01#the-exception-problem" class="hash-link" aria-label="Direct link to The Exception Problem" title="Direct link to The Exception Problem">​</a></h4>
<p>While exceptions might seem convenient for quick scripts and prototypes, they introduce significant challenges in complex, long-running applications:</p>
<ul>
<li><strong>Silent API Changes</strong>: One of the most dangerous aspects of exception-based error handling is that changes deep within dependencies can break your code without any compile-time warning. When a function suddenly starts throwing a new exception type, your code may fail at runtime under exceptional circumstances—often when you least expect it.</li>
<li><strong>Resource Management Issues</strong>: Exceptions create unpredictable control flow that can lead to resource leaks, security vulnerabilities, and unexpected crashes. When an exception unwinds the stack, resources may not be properly cleaned up.</li>
<li><strong>Refactoring Difficulties</strong>: The compiler provides little assistance when working with exception-based code. Adding a new exception type breaks the ABI but leaves the API unchanged, making it nearly impossible to track down all the places that need updating.</li>
</ul>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-result-advantage">The Result Advantage<a href="https://vac.dev/rlog/nim-in-logos-01#the-result-advantage" class="hash-link" aria-label="Direct link to The Result Advantage" title="Direct link to The Result Advantage">​</a></h4>
<p>The Result type offers a compelling alternative that makes error handling explicit, predictable, and compiler-verified:</p>
<div class="language-bash codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-bash codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain"># Enforce that no exceptions can't be raised in this module</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">{.push raises: [].}</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">import results</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">proc doSomething(): Result[void, string] =</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"># Implementation here</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">proc getRandomInt(): Result[int, string] =</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"># Implementation here</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">doSomething().isOkOr:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">echo "Failed doing something, error: ", &amp; error</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">randomInt = getRandomInt().valueOr:</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">echo "Failed getting random int, error: ", &amp; error</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>Notice that this usage of Result is much more concise and easier to follow than try-except blocks</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="best-practices-for-result-based-error-handling">Best Practices for Result-Based Error Handling<a href="https://vac.dev/rlog/nim-in-logos-01#best-practices-for-result-based-error-handling" class="hash-link" aria-label="Direct link to Best Practices for Result-Based Error Handling" title="Direct link to Best Practices for Result-Based Error Handling">​</a></h4>
<ul>
<li><strong>Make Errors Explicit</strong>: Use Result when multiple failure paths exist and calling code needs to differentiate between them. This makes error handling visible at the call site and forces developers to consciously handle failure cases.</li>
<li><strong>Handle Errors Locally</strong>: Address errors at each abstraction level rather than letting them bubble up through multiple layers. This prevents spurious abstraction leakage and keeps error handling logic close to where problems occur.</li>
<li><strong>Use Exception Tracking</strong>: Enable exception tracking with <code>{.push raises: [].}</code> at the module level. This helps identify any remaining exception-throwing code and ensures new code follows the Result pattern.</li>
</ul>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="when-to-break-the-rules">When to Break the Rules<a href="https://vac.dev/rlog/nim-in-logos-01#when-to-break-the-rules" class="hash-link" aria-label="Direct link to When to Break the Rules" title="Direct link to When to Break the Rules">​</a></h4>
<p>While Result should be your default choice, exceptions still have their place:</p>
<ul>
<li><strong>Assertions and Logic Errors</strong>: Use assertions for violated preconditions or situations where recovery isn't possible or expected.</li>
<li><strong>Legacy Integration</strong>: When interfacing with exception-heavy libraries, you may need to use exceptions at integration boundaries, but convert them to Result types as quickly as possible. To ensure safe exception handling, explicitly declare which exceptions a procedure may raise using the <code>{.raises: [SpecificException].}</code> pragma.</li>
</ul>
<p>Error handling in Nim continues to evolve, but the trend is clear: explicit error handling through Result types provides better safety, maintainability, and debugging experience than exceptions. By making errors part of your function signatures and forcing explicit handling at call sites, you create more robust software that fails gracefully and predictably.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="debugging-in-nim">Debugging in Nim<a href="https://vac.dev/rlog/nim-in-logos-01#debugging-in-nim" class="hash-link" aria-label="Direct link to Debugging in Nim" title="Direct link to Debugging in Nim">​</a></h2>
<p>Nowadays, analyzing the behavior of a Nim program is not as straightforward as debugging a C++ application, for example.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gdb">GDB<a href="https://vac.dev/rlog/nim-in-logos-01#gdb" class="hash-link" aria-label="Direct link to GDB" title="Direct link to GDB">​</a></h4>
<p>GDB can be used, and step-by-step debugging with GDB and VSCode is possible. However, the interaction is not very smooth. You can set breakpoints in VSCode and press F5 to run the program up to the breakpoint and continue debugging from there. That said, the state of variables is not fully demangled. For example:</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="img" src="https://vac.dev/assets/images/nim-gdb-fd0493d2f83bf5f19eb4476e41036b35.png" width="471" height="497" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>For that reason, GDB is not the preferred option in Logos</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="logs---chronicles">Logs - Chronicles<a href="https://vac.dev/rlog/nim-in-logos-01#logs---chronicles" class="hash-link" aria-label="Direct link to Logs - Chronicles" title="Direct link to Logs - Chronicles">​</a></h4>
<p>At Logos, we primarily debug Nim applications using log outputs. In particular, we make extensive use of the <a href="https://github.com/status-im/nim-chronicles" target="_blank" rel="noopener noreferrer">nim-chronicles</a> library.</p>
<p><code>nim-chronicles</code> is a robust logging library that automatically includes the following contextual information in each log entry:</p>
<ul>
<li>Calling thread ID</li>
<li>Current timestamp</li>
<li>Log level (e.g., TRACE, DEBUG, INFO, WARN, ERROR, FATAL)</li>
<li>Source file name</li>
<li>Line number of the log statement</li>
</ul>
<p>Additionally, <code>chronicles</code> supports attaching custom log messages along with relevant variable values, which proves especially useful for debugging. For instance, in the following example, the log message is <code>"Configuration. Shards"</code>, and it includes the value of an additional variable, <code>shard</code>.</p>
<div class="language-jsx codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-jsx codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token constant" style="color:rgb(189, 147, 249)">INF</span><span class="token plain"> </span><span class="token number">2025</span><span class="token operator">-</span><span class="token number">07</span><span class="token operator">-</span><span class="token number">01</span><span class="token plain"> </span><span class="token number">09</span><span class="token operator">:</span><span class="token number">56</span><span class="token operator">:</span><span class="token number">57.705</span><span class="token operator">+</span><span class="token number">02</span><span class="token operator">:</span><span class="token number">00</span><span class="token plain"> </span><span class="token maybe-class-name">Configuration</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token plain"> </span><span class="token property-access maybe-class-name">Shards</span><span class="token plain"> topics</span><span class="token operator">=</span><span class="token string" style="color:rgb(255, 121, 198)">"waku conf"</span><span class="token plain"> tid</span><span class="token operator">=</span><span class="token number">28817</span><span class="token plain"> file</span><span class="token operator">=</span><span class="token plain">waku_conf</span><span class="token punctuation" style="color:rgb(248, 248, 242)">.</span><span class="token property-access">nim</span><span class="token operator">:</span><span class="token number">147</span><span class="token plain"> shard</span><span class="token operator">=</span><span class="token number">64</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>There are also useful techniques for displaying more detailed information about specific variables:</p>
<ul>
<li><code>repr(p)</code> — Returns a string representation of the variable <code>p</code>, providing a more comprehensive view of its contents.</li>
<li><code>name(typeof(p))</code> — Extracts the type of the variable <code>p</code> as a string. This is particularly helpful when working with pointers or generics.</li>
</ul>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="logs---echo">Logs - echo<a href="https://vac.dev/rlog/nim-in-logos-01#logs---echo" class="hash-link" aria-label="Direct link to Logs - echo" title="Direct link to Logs - echo">​</a></h4>
<p>The <a href="https://nim-lang.org/docs/system.html#echo%2Cvarargs%5Btyped%2C%5D" target="_blank" rel="noopener noreferrer">echo</a> statement in Nim serves as a basic debugging tool, although it is less powerful and flexible compared to <code>nim-chronicles</code>.</p>
<p>Besides, <a href="https://www.notion.so/Nim-in-Logos-July-2025-2038f96fb65c8042bcbbe676ee8f2182?pvs=21" target="_blank" rel="noopener noreferrer">debugEcho</a> is an interesting alternative, which behaves similarly to <code>echo</code> but it allows working on routines marked with no side effects.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="heaptrack">Heaptrack<a href="https://vac.dev/rlog/nim-in-logos-01#heaptrack" class="hash-link" aria-label="Direct link to Heaptrack" title="Direct link to Heaptrack">​</a></h4>
<p>This technique enables precise insight into where memory is being consumed within a Nim application.</p>
<p>It is particularly useful for identifying potential memory leaks and is widely employed in <strong>nwaku</strong> (Nim Waku). For more details, refer to the documentation: <a href="https://github.com/waku-org/nwaku/blob/master/docs/tutorial/heaptrack.md" target="_blank" rel="noopener noreferrer">Heaptrack Tutorial</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="formatting-code-in-nim">Formatting code in Nim<a href="https://vac.dev/rlog/nim-in-logos-01#formatting-code-in-nim" class="hash-link" aria-label="Direct link to Formatting code in Nim" title="Direct link to Formatting code in Nim">​</a></h2>
<p>Maintaining a consistent code format is essential for readability and for facilitating clear diff comparisons during code reviews.</p>
<p>To support this, Logos strongly recommends using <a href="https://arnetheduck.github.io/nph/" target="_blank" rel="noopener noreferrer"><em>nph</em></a> across all Nim projects.</p>]]></content>
<author>
<name>Ivan</name>
</author>
<author>
<name>Gabriel</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[The MDSECheck method: choosing secure square MDS matrices for P-SP-networks]]></title>
<id>https://vac.dev/rlog/mdsecheck-method</id>
<link href="https://vac.dev/rlog/mdsecheck-method"/>
<updated>2025-02-28T23:00:00.000Z</updated>
<summary type="html"><![CDATA[This article introduces MDSECheck method — a novel approach]]></summary>
<content type="html"><![CDATA[<p>This article introduces MDSECheck method — a novel approach
to checking square MDS matrices for unconditional security
as the components of affine permutation layers of P-SP-networks.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/mdsecheck-method#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Maximum distance separable (MDS) matrices play a significant role
in algebraic coding theory and symmetric cryptography.
In particular, square MDS matrices are commonly used in
affine permutation layers of
partial substitution-permutation networks (P-SPNs).
These are widespread designs of
the modern symmetric ciphers and hash functions.
A classic example of the latter is Poseidon <a href="https://vac.dev/rlog/mdsecheck-method#references">[1]</a>,
a well-known hash function used in zk-SNARK proving systems.</p>
<p>Square MDS matrices differ in terms of security
that they are able to provide for P-SPNs.
The use of some such matrices in certain P-SPNs may result in existence
of infinitely long subspace trails of small period for the latter,
which make them vulnerable to differential cryptanalysis <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.</p>
<p>Two methods for security checking of square MDS matrices for P-SPNs
have been proposed in <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.
The first one, which is referred to as the three tests method
in the rest of the article, is aimed at security checking for
a specified structure of the substitution layer of a P-SPN.
The second method, which is referred here as the sufficient test method,
has been designed to determine whether a square MDS matrix satisfies
a sufficient condition of being secure regardless of the structure of
a P-SPN substitution layer, i.e. to check whether the matrix belongs to
the class of square MDS matrices, which are referred to
as unconditionally secure in the current article.</p>
<p>This article aims to introduce MDSECheck method —
a novel approach to checking square MDS matrices for unconditional security,
which has already been implemented in the Rust programming language as
the library crate <a href="https://vac.dev/rlog/mdsecheck-method#references">[3]</a>.
The next sections explain the notions mentioned above,
describe the MDSECheck method as well as its mathematical foundations,
provide a brief overview of the MDSECheck library crate
and outline possible future research directions.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mds-matrix-how-to-define-and-construct">MDS matrix: how to define and construct<a href="https://vac.dev/rlog/mdsecheck-method#mds-matrix-how-to-define-and-construct" class="hash-link" aria-label="Direct link to MDS matrix: how to define and construct" title="Direct link to MDS matrix: how to define and construct">​</a></h2>
<p>An <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> over a finite field is called MDS,
if and only if for distinct <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional column vectors <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>v</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">v_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>v</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">v_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>
the column vectors <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>v</mi><mn>1</mn></msub><mtext> </mtext><mi mathvariant="normal">∣</mi><mtext> </mtext><mi>M</mi><msub><mi>v</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">v_1 \: | \: M v_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">∣</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>v</mi><mn>2</mn></msub><mtext> </mtext><mi mathvariant="normal">∣</mi><mtext> </mtext><mi>M</mi><msub><mi>v</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">v_2 \: | \: M v_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">∣</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">|</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">∣</span></span></span></span> stands for vertical concatenation,
do not coincide in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> or more components.
The set of all possible column vectors <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi><mtext> </mtext><mi mathvariant="normal">∣</mi><mtext> </mtext><mi>M</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">v \: | \: M v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">∣</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span> for
some fixed matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is a systematic MDS code, i.e.
a linear code, which contains input symbols on their original positions
and achieves the Singleton bound.
The latter property results in good error-correction capability.</p>
<p>There are several equivalent definitions of MDS matrices,
but the next one is especially useful for constructing them
directly by means of algebraic methods.
A matrix over a finite field is called MDS,
if and only if all its square submatrices are nonsingular.
The matrix entries and the matrix itself are also considered submatrices.</p>
<p>One of the most efficient and straightforward methods to directly construct
an MDS matrix is generating a Cauchy matrix <a href="https://vac.dev/rlog/mdsecheck-method#references">[4]</a>.
Such an <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix is defined using
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">m</span></span></span></span>-dimensional vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span>,
for which all entries in the concatenation of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> are distinct.
The entries of the Cauchy matrix are described by the formula
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>M</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>j</mi></mrow></msub><mo>=</mo><mn>1</mn><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><mo stretchy="false">(</mo><msub><mi>x</mi><mi>i</mi></msub><mo>−</mo><msub><mi>y</mi><mi>j</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">M_{i, j} = 1 \: / \: (x_i - y_j)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.109em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">/</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0361em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.
It is obvious that any submatrix of a Cauchy matrix is also a Cauchy matrix.
The Cauchy determinant formula <a href="https://vac.dev/rlog/mdsecheck-method#references">[5]</a> implies that
every square Cauchy matrix is nonsingular.
Thus, Cauchy matrices satisfy the second definition of MDS matrices.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="partial-substitution-permutation-networks">Partial substitution-permutation networks<a href="https://vac.dev/rlog/mdsecheck-method#partial-substitution-permutation-networks" class="hash-link" aria-label="Direct link to Partial substitution-permutation networks" title="Direct link to Partial substitution-permutation networks">​</a></h2>
<p>Describing SPNs in algebraic terms is convenient,
so this approach has been chosen for this article.
SPNs are designs of the symmetric cryptoprimitives,
which operate on an internal state, which is represented
as an <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional vector over some finite field,
and update this state iteratively by means of
the round transformations described below.</p>
<p>Each round begins with an optional update of the internal state by
adding to its components some input data or extraction of
some of these components as the output data.
This optional step depends on the specific cryptoprimitive
and the current round number.
The next step is called the nonlinear substitution layer
and lies in replacing the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span>-th component of the internal state
with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">S_i(c)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">c</span><span class="mclose">)</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>1..</mn><mi>n</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">i \in [1..n]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">1..</span><span class="mord mathnormal">n</span><span class="mclose">]</span></span></span></span>, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> is the component value
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">S_i(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span> is a nonlinear invertible function over the finite field.
The function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>S</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">S_i(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span> is specific to the cryptoprimitive and called an S-Box.
The final step, which is known as the affine permutation layer,
replaces the internal state with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi><mi>X</mi><mo>+</mo><mi>c</mi></mrow><annotation encoding="application/x-tex">M X + c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">MX</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> is the current internal state,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is a nonsingular square matrix and
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> is the vector of the round constants.
The value of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> is specific to the cryptoprimitive
and the current round number,
while <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> typically depends only on the cryptoprimitive.
The data flow diagram for an SPN is given below.</p>
<div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">.................................. </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> │ │ │ │ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌────────────────────────────────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ Optional addition / extraction │ &lt;─────&gt; Input / output</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬────────┬────────┬────────┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│S₁(x)│ │S₂(x)│ │ ... │ │Sₙ(x)│ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌────────────────────────────────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ Affine permutation │ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬────────┬────────┬────────┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌────────────────────────────────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ Optional addition / extraction │ &lt;─────&gt; Input / output</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬────────┬────────┬────────┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│S₁(x)│ │S₂(x)│ │ ... │ │Sₙ(x)│ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">┌────────────────────────────────┐ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">│ Affine permutation │ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">└──┬────────┬────────┬────────┬──┘ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"> ▼ ▼ ▼ ▼ </span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">.................................. </span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>Partial SPNs are modifications of SPNs,
where for certain rounds some S-Boxes are replaced with
the identity functions to reduce computational efforts <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.
For example, the nonlinear substitution layers of the partial rounds of
Poseidon update only the first internal state component <a href="https://vac.dev/rlog/mdsecheck-method#references">[1]</a>.
In the case of P-SPNs, security considerations commonly demand to choose <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
as a square MDS matrix, because these matrices provide
perfect diffusion property for the affine permutation layer <a href="https://vac.dev/rlog/mdsecheck-method#references">[6]</a>.
Possessing this property means ensuring that
any two <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional internal states,
which differ in exactly <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span> components,
are mapped by the affine permutation layer to
two new internal states that differ in at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>−</mo><mi>t</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">n - t + 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6667em;vertical-align:-0.0833em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6984em;vertical-align:-0.0833em"></span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> components.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="square-mds-matrix-security-check-in-the-context-of-p-spns">Square MDS matrix security check in the context of P-SPNs<a href="https://vac.dev/rlog/mdsecheck-method#square-mds-matrix-security-check-in-the-context-of-p-spns" class="hash-link" aria-label="Direct link to Square MDS matrix security check in the context of P-SPNs" title="Direct link to Square MDS matrix security check in the context of P-SPNs">​</a></h2>
<p>Certain square MDS matrices should not be used in certain P-SPNs
to avoid making them vulnerable to differential cryptanalysis,
since it may exploit the existence of infinitely long subspace trails
of small period for vulnerable P-SPNs. <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.
Such matrices are called insecure with respect to particular P-SPNs.</p>
<p>An infinitely long subspace trail of period <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> exists for a P-SPN,
if and only if there is a proper subspace
of differences of internal state vectors,
such that if for a pair of initial internal states
the difference belongs to this subspace,
then the difference for the new internal states,
which are obtained from the initial ones
by means of the same <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span>-round transformation,
also belongs to this subspace <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.</p>
<p>Two methods for checking square MDS matrices for suitability for P-SPNs
in terms of existence of infinitely long subspace trails
have been proposed in <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a>.
The three tests method is aimed at checking
whether using a specified matrix for a P-SPN
with a specified structure of the substitution layer
leads to existence of infinitely long subspace trails of period <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span>
for this P-SPN for all <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span> no larger than a given <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span>.
The sufficient test method has been designed to determine
whether a square MDS matrix satisfies a sufficient condition
of non-existence of infinitely long subspace trails of period <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span>
for P-SPNs using this matrix for all <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span> no larger than a specified <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span>.</p>
<p>The sufficient test method is a direct consequence of
Theorem 8 in <a href="https://vac.dev/rlog/mdsecheck-method#references">[2]</a> and consists in checking that
the minimal polynomial of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span>-th power of the tested matrix
has maximum degree and is irreducible for all <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>1..</mn><mi>l</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">p \in [1..l]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7335em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">1..</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mclose">]</span></span></span></span>.
The aforesaid sufficient non-existence condition is satisfied by the matrix,
if and only if all the checks yield positive results.</p>
<p>It is convenient to define
the unconditional P-SPN security level of the square MDS matrix as follows:
this level is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> for the matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>,
if and only if the minimal polynomials of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">M^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">M^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>
have maximum degree and are irreducible,
but for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mrow><mi>l</mi><mtext> </mtext><mo>+</mo><mtext> </mtext><mn>1</mn></mrow></msup></mrow><annotation encoding="application/x-tex">M^{l \: + \: 1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">+</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span> the minimal polynomial does not have this property.
Using this definition, the purpose of the sufficient test method
can be described as checking whether
the unconditional P-SPN security level of the specified matrix
is no less than a given bound.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mdsecheck-method-getting-rid-of-the-matrix-powers">MDSECheck method: getting rid of the matrix powers<a href="https://vac.dev/rlog/mdsecheck-method#mdsecheck-method-getting-rid-of-the-matrix-powers" class="hash-link" aria-label="Direct link to MDSECheck method: getting rid of the matrix powers" title="Direct link to MDSECheck method: getting rid of the matrix powers">​</a></h2>
<p>The MDSECheck method, whose name is derived from
the words "MDS", "security", "elaborated" and "check",
has the same purpose as the sufficient test method,
but achieves it differently.
The differences of the first method from the latter
and approaches to implementing them can be described as follows:</p>
<ol>
<li>
<p>Computation and verification of minimal polynomials
of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">M^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">M^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">M^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is
the tested <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix over <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><mi>q</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mclose">)</span></span></span></span>
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> is the security level bound,
has been replaced with checks for the corresponding powers
of a root of the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
for non-presence in nontrivial subfields of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.</p>
<ol>
<li>
<p>The non-presence check is performed without
straightforward consideration of all nontrivial subfields of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.
The root is checked only for non-presence in the subfields
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mn>1</mn></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_1})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mn>2</mn></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_2})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>d</mi></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_d})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1512em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">p_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">p_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>d</mi></msub></mrow><annotation encoding="application/x-tex">p_d</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>
are all prime divisors of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>.</p>
</li>
<li>
<p>The non-presence check reuses some data computed during
the checking for irreducibility the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>,
which in this case coincides with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>
designating the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>.
The values of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2624em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> are saved
for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>1..</mn><mi>d</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">j \in [1..d]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">1..</span><span class="mord mathnormal">d</span><span class="mclose">]</span></span></span></span> during the irreducibility check
to replace exponentiations with sequential computations
of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>y</mi><mi>i</mi></msup><msup><mo stretchy="false">)</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(y^i)^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> from
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>y</mi><mrow><mo stretchy="false">(</mo><mi>i</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn><mo stretchy="false">)</mo></mrow></msup><msup><mo stretchy="false">)</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(y^{(i \: - \: 1)})^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mathnormal mtight">i</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span><span class="mclose mtight">)</span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>
as its product with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2624em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>.</p>
</li>
</ol>
</li>
<li>
<p>The check of the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
for irreducibility and maximum degree
is performed without unconditional computation of this polynomial.
This computation has been replaced with the Krylov method fragment,
which consists in building and solving
only one system of linear equations over <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><mi>q</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mclose">)</span></span></span></span>.
If <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> has an irreducible minimal polynomial of maximum degree,
then its coefficients are trivially determined from the system solution.
If the system is degenerate,
then the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> does not have such properties.</p>
</li>
</ol>
<p>The correctness of the first distinctive feature can be proven as follows.
Verifying that the minimal polynomial of a matrix
is of maximum degree and irreducible
is equivalent to verifying that
the characteristic polynomial of this matrix is irreducible,
because the minimal polynomial divides the characteristic one.
Also, it is trivially proven that for a matrix with such a minimal polynomial
it is equal to the characteristic polynomial.
Thus, the required checks for the matrices <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">M^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">M^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">M^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>
can be done by checking their characteristic polynomials for irreducibility.</p>
<p>Let <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> be <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix over <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><mi>q</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mclose">)</span></span></span></span>,
whose minimal polynomial is of maximum degree and irreducible.
The statements in the previous paragraph imply that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>,
which is the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-degree characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>, is irreducible.
Consider <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> over the extension field <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>,
which is the splitting field of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>.
Let <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>∈</mo><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">α \in GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> be a root of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>.
According to standard results from the Galois field theory,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">α</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>q</mi></msup></mrow><annotation encoding="application/x-tex">α^q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><msup><mi>q</mi><mn>2</mn></msup></msup></mrow><annotation encoding="application/x-tex">α^{q^2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9869em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup></msup></mrow><annotation encoding="application/x-tex">α^{q^{n \: - \: 1}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9869em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
are distinct roots of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> <a href="https://vac.dev/rlog/mdsecheck-method#references">[7]</a>.
Thus, these powers of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">α</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span> are <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> distinct eigenvalues of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>.
Hence, due to matrix similarity properties, there is some matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi></mrow><annotation encoding="application/x-tex">S</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span></span></span></span>
such that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>M</mi><msup><mi>S</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo>=</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">S M S^{-1} = D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord mathnormal" style="margin-right:0.10903em">SM</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> is the diagonal matrix,
whose nonzero elements are
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">α</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>q</mi></msup></mrow><annotation encoding="application/x-tex">α^q</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><msup><mi>q</mi><mn>2</mn></msup></msup></mrow><annotation encoding="application/x-tex">α^{q^2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9869em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup></msup></mrow><annotation encoding="application/x-tex">α^{q^{n \: - \: 1}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9869em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>.
Therefore, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><msup><mi>M</mi><mi>i</mi></msup><msup><mi>S</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo>=</mo><msup><mi>D</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">S M^i S^{-1} = D^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>,
so the roots of the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">M^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>
are <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>α</mi><mi>q</mi></msup><msup><mo stretchy="false">)</mo><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">(α^q)^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>α</mi><msup><mi>q</mi><mn>2</mn></msup></msup><msup><mo stretchy="false">)</mo><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">(α^{q^2})^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2369em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>α</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup></msup><msup><mo stretchy="false">)</mo><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">(α^{q^{n \: - \: 1}})^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2369em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>.
If the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> has degree less than <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>,
then the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">M^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> is divisible
by this minimal polynomial,
while <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> lies in some nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.
One of the fields isomorphic to this subfield is a residue class ring
of polynomials modulo the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> <a href="https://vac.dev/rlog/mdsecheck-method#references">[7]</a>.
If the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> is of degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>,
then the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">M^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>
equals this minimal polynomial and therefore is irreducible,
while <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> does not lie in any nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.
In this case, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>α</mi><mi>i</mi></msup><msup><mo stretchy="false">)</mo><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">(α^i)^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>α</mi><mi>i</mi></msup><msup><mo stretchy="false">)</mo><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup></mrow><annotation encoding="application/x-tex">(α^i)^{n \: - \: 1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span>
are linearly independent as distinct roots of
an irreducible polynomial over a finite field <a href="https://vac.dev/rlog/mdsecheck-method#references">[7]</a>,
so any field containing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> has at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>q</mi><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">q^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> elements
and therefore cannot be a trivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.
Thus, checking the characteristic polynomials of
the matrices <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">M^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">M^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">M^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span> for irreducibility
is equivalent to verifying that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">α^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">α^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">α^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>
do not lie in any nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.</p>
<p>The last sentences of the two previous paragraphs imply the following:
verifying that the minimal polynomials of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">M^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">M^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">M^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>
are of maximum degree and irreducible can be performed
by verifying that the corresponding powers of a root of
the characteristic polynomial of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> over <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><mi>q</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mclose">)</span></span></span></span>
do not belong to any nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>. <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">■</mi></mrow><annotation encoding="application/x-tex">\blacksquare</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.675em"></span><span class="mord amsrm">■</span></span></span></span></p>
<p>The approaches to implementing the first distinctive feature
can be explained and proven to be correct as follows.
Since <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>w</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^w)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> is a nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>u</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^u)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">u</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>
if and only if <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi></mrow><annotation encoding="application/x-tex">w</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.02691em">w</span></span></span></span> divides <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">u</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi><mo>&lt;</mo><mi>u</mi></mrow><annotation encoding="application/x-tex">w &lt; u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.02691em">w</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">&lt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">u</span></span></span></span> <a href="https://vac.dev/rlog/mdsecheck-method#references">[7]</a>,
the presence of some <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">ε</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">ε</span></span></span></span> in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>h</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^h)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">h</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>,
which is a nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>,
implies that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi><mo>∈</mo><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><mi>ν</mi></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">ε \in GF(q^{n \: / \: ν})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em"></span><span class="mord mathnormal">ε</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mathnormal mtight" style="margin-right:0.06366em">ν</span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> for some prime <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ν</mi></mrow><annotation encoding="application/x-tex">ν</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.06366em">ν</span></span></span></span> dividing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>,
because <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span> divides the quotient of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> and some of its prime factors.
Thus, checking that some value does not belong to subfields
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mn>1</mn></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_1})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mn>2</mn></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_2})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>d</mi></msub></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^{n \: / \: p_d})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3488em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1512em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">p_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">p_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mi>d</mi></msub></mrow><annotation encoding="application/x-tex">p_d</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">d</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> are all prime divisors of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>,
is equivalent to checking this value for
non-presence in nontrivial subfields of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.</p>
<p>Checking for irreducibility the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
is performed by means of Algorithm 2.2.9 in <a href="https://vac.dev/rlog/mdsecheck-method#references">[8]</a>
and consists in sequential computation of
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><mi>p</mi></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^p \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>p</mi><mn>2</mn></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^{p^2} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1814em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>p</mi><mrow><mo stretchy="false">⌊</mo><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><mn>2</mn><mo stretchy="false">⌋</mo></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^{p^{\lfloor n \: / \: 2 \rfloor}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2341em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0397em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9667em"><span style="top:-2.9667em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mopen mtight">⌊</span><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">2</span><span class="mclose mtight">⌋</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>
and checking that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>C</mi><mi>D</mi><mo stretchy="false">(</mo><msup><mi>y</mi><msup><mi>p</mi><mi>i</mi></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mtext> </mtext><mo>−</mo><mtext> </mtext><mi>y</mi><mo separator="true">,</mo><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">GCD(y^{p^i} \mod f(y) \: - \: y, f(y)) = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2445em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.07153em">GC</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9945em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9021em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">))</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>
for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>1..</mn><mo stretchy="false">⌊</mo><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><mn>2</mn><mo stretchy="false">⌋</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">i \in [1..\lfloor n \: / \: 2 \rfloor]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">1..</span><span class="mopen">⌊</span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">/</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord">2</span><span class="mclose">⌋]</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
and coincides with the minimal polynomial in this case.
The optimized root non-presence check is performed
by checking that for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>2..</mn><mi>l</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">i \in [2..l]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">2..</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mclose">]</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>j</mi><mo>∈</mo><mo stretchy="false">[</mo><mn>1..</mn><mi>d</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">j \in [1..d]</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">[</span><span class="mord">1..</span><span class="mord mathnormal">d</span><span class="mclose">]</span></span></span></span>
the value of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mo stretchy="false">(</mo><msup><mi>y</mi><mi>i</mi></msup><msup><mo stretchy="false">)</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mo>−</mo><msup><mi>y</mi><mi>i</mi></msup><mo stretchy="false">)</mo><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">((y^i)^{q^{n \: / \: p_j}} - y^i) \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">((</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is nonzero.
This approach is based on the following standard results
from the Galois field theory <a href="https://vac.dev/rlog/mdsecheck-method#references">[7]</a>:</p>
<ul>
<li>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> is isomorphic to the residue class ring of
univariate polynomials in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> modulo <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>,
because at this point <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is known to be irreducible,
and some root of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is mapped by this isomorphism to
the residue class the polynomial <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> in this ring.</p>
</li>
<li>
<p>All elements of a finite field <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>w</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^w)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> and only they
are roots of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>q</mi><mi>w</mi></msup></msup><mo>−</mo><mi>y</mi></mrow><annotation encoding="application/x-tex">y^{q^w} - y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0744em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.88em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.7385em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span>.</p>
</li>
</ul>
<p>The expression <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mo stretchy="false">(</mo><msup><mi>y</mi><mi>i</mi></msup><msup><mo stretchy="false">)</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mo>−</mo><msup><mi>y</mi><mi>i</mi></msup><mo stretchy="false">)</mo><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">((y^i)^{q^{n \: / \: p_j}} - y^i) \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">((</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>
can be rewritten as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mo stretchy="false">(</mo><msup><mi>y</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><msup><mo stretchy="false">)</mo><mi>i</mi></msup><mo>−</mo><msup><mi>y</mi><mi>i</mi></msup><mo stretchy="false">)</mo><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">((y^{q^{n \: / \: p_j}})^i - y^i) \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">((</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>,
which can be computed without exponentiation as the product of
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msup><mi>y</mi><mrow><mo stretchy="false">(</mo><mi>i</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn><mo stretchy="false">)</mo></mrow></msup><msup><mo stretchy="false">)</mo><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(y^{(i \: - \: 1)})^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.318em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mathnormal mtight">i</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span><span class="mclose mtight">)</span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> and
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><msup><mi>q</mi><mrow><mi>n</mi><mtext> </mtext><mi mathvariant="normal">/</mi><mtext> </mtext><msub><mi>p</mi><mi>j</mi></msub></mrow></msup></msup><mspace></mspace><mspace width="0.6667em"></mspace><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow><mtext> </mtext><mtext> </mtext><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y^{q^{n \: / \: p_j}} \mod f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2624em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.068em"><span style="top:-3.068em;margin-right:0.05em"><span class="pstrut" style="height:2.705em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.0071em"><span style="top:-3.0072em;margin-right:0.0714em"><span class="pstrut" style="height:2.5357em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight">/</span><span class="mspace mtight" style="margin-right:0.3271em"></span><span class="mord mtight"><span class="mord mathnormal mtight">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3448em;margin-left:0em;margin-right:0.1em"><span class="pstrut" style="height:2.6595em"></span><span class="mord mathnormal mtight" style="margin-right:0.05724em">j</span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.5092em"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mspace allowbreak"></span><span class="mspace" style="margin-right:0.6667em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord mathrm">mod</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>,
which has been saved during the irreducibility check.</p>
<p>The second distinctive feature can be
explained and proven to be correct in following way.
The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> does not have a minimal polynomial of maximum degree,
if some Krylov subspace of order <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> for it is not <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional.
Indeed, the minimal polynomial of the matrix is divisible
by the minimal polynomial of the restriction of
this linear operator to an arbitrary subspace,
and in the considered case the latter polynomial has degree less than <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>,
because the degree of the minimal polynomial of a linear operator cannot
exceed the dimension of the subspace the operator acts on.
Thus, an unconditional computation of the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>
is not required to determine
whether this polynomial is irreducible and has maximum degree.
Using this computation has been replaced with the Krylov method fragment,
which consists in choosing any nonzero <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>-dimensional column vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>
and solving the system of linear equations <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>A</mi><mi>X</mi><mo>=</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">A X = b</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">b</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal">A</span></span></span></span> is an <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> x <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> matrix,
whose columns are <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi><mi>v</mi></mrow><annotation encoding="application/x-tex">M v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mn>2</mn></msup><mi>v</mi></mrow><annotation encoding="application/x-tex">M^2 v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup><mi>v</mi></mrow><annotation encoding="application/x-tex">M^{n \: - \: 1} v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>,
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">b</span></span></span></span> is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>M</mi><mi>n</mi></msup><mi>v</mi></mrow><annotation encoding="application/x-tex">M^n v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">M</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.03588em">v</span></span></span></span>.
If <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal">A</span></span></span></span> is singular,
the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is reducible or does not have maximum degree,
so checking <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> has been accomplished;
otherwise, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span>, which is the minimal and characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>,
can be expressed as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>y</mi><mi>n</mi></msup><mo>−</mo><msub><mi>X</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msub><msup><mi>y</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow></msup><mo>−</mo><msub><mi>X</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>2</mn></mrow></msub><msup><mi>y</mi><mrow><mi>n</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>2</mn></mrow></msup><mo>−</mo><mo>…</mo><mo>−</mo><msub><mi>X</mi><mn>1</mn></msub><mi>y</mi><mo>−</mo><msub><mi>X</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">y^n - X_{n \: - \: 1} y^{n \: - \: 1} -
X_{n \: - \: 2} y^{n \: - \: 2} - … - X_1 y - X_0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0224em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0785em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0224em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0785em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">2</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mbin mtight">−</span><span class="mspace mtight" style="margin-right:0.2602em"></span><span class="mord mtight">2</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6667em;vertical-align:-0.0833em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0785em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0785em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.</p>
<p>The steps of MDSECheck method can be summarized as follows:</p>
<ol>
<li>
<p>The square MDS matrix <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> over <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><mi>q</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mclose">)</span></span></span></span>
and the unconditional P-SPN security level bound <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> are received as inputs.</p>
</li>
<li>
<p>The Krylov method fragment is used to compute the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>.
If the computation fails, then <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is not unconditionally secure,
so the check of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is complete.
If it succeeds, then the minimal polynomial has maximum degree
and, therefore, coincides with the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>.</p>
</li>
<li>
<p>Algorithm 2.2.9 is used
to check for irreducibility the minimal polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>,
which is also the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> in this case.
Some data computed during this step is saved to be reused at the next one.
If the polynomial is reducible, then the check of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is complete,
because <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> has been found to be not unconditionally secure.</p>
</li>
<li>
<p>The values of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">α^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mn>3</mn></msup></mrow><annotation encoding="application/x-tex">α^3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">...</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.1056em"></span><span class="mord">...</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>l</mi></msup></mrow><annotation encoding="application/x-tex">α^l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8491em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span></span></span></span></span></span></span></span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">α</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.0037em">α</span></span></span></span> is a root of the characteristic polynomial of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span>,
are sequentially checked for non-presence in
nontrivial subfields of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> as described above.
If <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>α</mi><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">α^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0037em">α</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> belongs to some nontrivial subfield of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>F</mi><mo stretchy="false">(</mo><msup><mi>q</mi><mi>n</mi></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">GF(q^n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.13889em">GF</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>,
then the unconditional P-SPN security level of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mtext> </mtext><mo>−</mo><mtext> </mtext><mn>1</mn></mrow><annotation encoding="application/x-tex">i \: - \: 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7429em;vertical-align:-0.0833em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>,
so the check of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">M</span></span></span></span> is complete.
If all the values do not belong to such a subfield,
then the unconditional P-SPN security level is at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi></mrow><annotation encoding="application/x-tex">l</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span>.</p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="mdsecheck-library-crate-implementation-in-rust">MDSECheck library crate: implementation in Rust<a href="https://vac.dev/rlog/mdsecheck-method#mdsecheck-library-crate-implementation-in-rust" class="hash-link" aria-label="Direct link to MDSECheck library crate: implementation in Rust" title="Direct link to MDSECheck library crate: implementation in Rust">​</a></h2>
<p>The library crate <a href="https://vac.dev/rlog/mdsecheck-method#references">[3]</a> provides tools for
generating random square Cauchy MDS matrices over prime finite fields
and applying the MDSECheck method
to check such matrices for unconditional security.
The used data types of field elements and polynomials are provided by
the crates ark-ff <a href="https://vac.dev/rlog/mdsecheck-method#references">[9]</a> and ark-poly <a href="https://vac.dev/rlog/mdsecheck-method#references">[10]</a>.
The auxiliary tools in the crate modules are accessible as well.</p>
<p>Generating by means of this crate a 10 x 10 MDS matrix,
which is defined over the BN254 scalar field <a href="https://vac.dev/rlog/mdsecheck-method#references">[11]</a>
and has unconditional P-SPN security level is 1000,
takes less than 60 milliseconds on average
for the laptop with the processor Intel® Core™ i9-14900HX,
whose maximum clock frequency is 5.8 GHz.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/mdsecheck-method#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>The MDSECheck method proposed in this article is a novel approach
to checking square MDS matrices for unconditional security
as the components of affine permutation layers of P-SPNs.
It has been implemented as a practical library crate
for generating unconditionally secure square MDS matrices
for P-SPNs over prime finite fields.</p>
<p>The future research directions may include
theoretical and experimental studies of performance of approaches,
which use the MDSECheck method
to generate unconditionally secure square MDS matrices for P-SPNs.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/mdsecheck-method#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ol>
<li>L. Grassi, D. Khovratovich, C. Rechberger, A. Roy, M. Schofnegger. "<a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer">POSEIDON: A New Hash Function for Zero-Knowledge Proof Systems (Updated Version)</a>".</li>
<li>L. Grassi, C. Rechberger, M. Schofnegger. "<a href="https://eprint.iacr.org/2020/500.pdf" target="_blank" rel="noopener noreferrer">Proving Resistance Against Infinitely Long Subspace Trails: How to Choose the Linear Layer</a>".</li>
<li>The page "<a href="https://crates.io/crates/mdsecheck" target="_blank" rel="noopener noreferrer">mdsecheck</a>" on crates.io.</li>
<li>Y. Kumar, P. Mishra, S. Samanta, K. Chand Gupta, A. Gaur. "<a href="https://arxiv.org/pdf/2403.10372" target="_blank" rel="noopener noreferrer">Construction of all MDS and involutory MDS matrices</a>".</li>
<li>The page "<a href="https://proofwiki.org/wiki/Value_of_Cauchy_Determinant" target="_blank" rel="noopener noreferrer">Value of Cauchy Determinant</a>" on proofwiki.org.</li>
<li>T. Silva, R. Dahab "<a href="https://www.ic.unicamp.br/~reltech/PFG/2021/PFG-21-43.pdf" target="_blank" rel="noopener noreferrer">MDS Matrices for Cryptography</a>".</li>
<li>S. Huczynska, M. Neunhöffer. "<a href="http://www.math.rwth-aachen.de/~Max.Neunhoeffer/Teaching/ff2012/ff2012.pdf" target="_blank" rel="noopener noreferrer">Finite Fields</a>"</li>
<li>R. Crandall, C. Pomerance. "<a href="http://thales.doa.fmph.uniba.sk/macaj/skola/teoriapoli/primes.pdf" target="_blank" rel="noopener noreferrer">Prime Numbers: A Computational Perspective</a>" (2nd edition).</li>
<li>The page "<a href="https://crates.io/crates/ark-ff" target="_blank" rel="noopener noreferrer">ark-ff</a>" on crates.io.</li>
<li>The page "<a href="https://crates.io/crates/ark-poly" target="_blank" rel="noopener noreferrer">ark-poly</a>" on crates.io.</li>
<li>The page "<a href="https://crates.io/crates/ark-bn254" target="_blank" rel="noopener noreferrer">ark-bn254</a>" on crates.io.</li>
</ol>]]></content>
<author>
<name>Aleksei Vambol</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Vac 2024 Year in Review]]></title>
<id>https://vac.dev/rlog/2024-recap</id>
<link href="https://vac.dev/rlog/2024-recap"/>
<updated>2025-01-09T18:30:00.000Z</updated>
<summary type="html"><![CDATA[In this post, we recap Vac's achievements in 2024 and look forward to 2025.]]></summary>
<content type="html"><![CDATA[<p>In this post, we recap Vac's achievements in 2024 and look forward to 2025.</p>
<!-- -->
<p>With 2024 now behind us and a new year ahead,
Vac is proud to reflect on the milestones and breakthroughs that defined another year of researching and developing free and open digital public goods for the <a href="https://free.technology/" target="_blank" rel="noopener noreferrer">Institute of Free Technology</a> and wider web3 ecosystem.</p>
<p>Vac comprises various subteams and service units, each with its own focus.
Below, we celebrate each unit's achievements and look forward to its 2025 plans.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nescience">Nescience<a href="https://vac.dev/rlog/2024-recap#nescience" class="hash-link" aria-label="Direct link to Nescience" title="Direct link to Nescience">​</a></h2>
<p>Nescience is our state separation architecture that aims to enable private transactions and provide a general-purpose execution environment for classical applications.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<p>This year, the Nescience state separation architecture moved from exploration to real progress,
taking significant steps towards building a functional and reliable system.
The team focused on turning ideas into something real,
testing the proposed architecture,
and understanding its strengths and weaknesses.</p>
<ul>
<li>ZkVM exploration and benchmarks<!-- -->
<ul>
<li>Published <a href="https://vac.dev/rlog/zkVM-explorations/" target="_blank" rel="noopener noreferrer">deep reviews of 23 existing zkVMs</a></li>
<li><a href="https://vac.dev/rlog/zkVM-testing/" target="_blank" rel="noopener noreferrer">Benchmarked the performance of the six zkVMs</a> that best fit Nescience</li>
</ul>
</li>
<li>Defined the NSSA architecture<!-- -->
<ul>
<li>Brought clarity to NSSA’s design and explained the system’s architecture <a href="https://vac.dev/rlog/Nescience-state-separation-architecture/" target="_blank" rel="noopener noreferrer">in a lengthy exploratory blog post</a></li>
</ul>
</li>
<li>Built the sandboxed testnet<!-- -->
<ul>
<li>Designed the first version of the node specification</li>
<li>All core components (execution types, UTXOs, cryptographic primitives) implemented and being tested</li>
<li>Testing the performance of all execution types in various scenarios</li>
</ul>
</li>
</ul>
<p>We also made progress on the essential parts of NSSA’s system, including:</p>
<ul>
<li>Key protocol for secure key management</li>
<li>Execution types and circuits for reliable computation</li>
<li>UTXO specification to manage state transitions effectively</li>
<li>Cryptography module to ensure privacy and security</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<p>In 2025, the Nescience team plans to double down on what works, fix what doesn’t, and push NSSA closer to real-world use.</p>
<ul>
<li>Sandboxed testnet data analysis – the sandboxed testnet will be our primary data source that we will analyse to identify issues, limitations, and areas for improvement.</li>
<li>Expanding the node – expand sandboxed components into a full node implementation with rigorous testing and iterative optimization (to bridge the gap between proof of concept and production readiness).</li>
<li>Finalizing the architecture and RFC – after completing NSSA’s architecture, we will draft an RFC to ensure transparency and enable greater collaboration with the broader ecosystem.</li>
<li>Testing real-life scenarios – applying NSSA to diverse, practical use cases to assess its adaptability, performance, and impact.</li>
<li>Ongoing optimization – ensure NSSA is robust, efficient, and ready to scale.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="token-economics-tke">Token Economics (TKE)<a href="https://vac.dev/rlog/2024-recap#token-economics-tke" class="hash-link" aria-label="Direct link to Token Economics (TKE)" title="Direct link to Token Economics (TKE)">​</a></h2>
<p>The TKE Service Unit works closely with IFT portfolio projects to design and implement crypto-economic incentive structures.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-1">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-1" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Formalized and implemented <a href="https://codex.storage/" target="_blank" rel="noopener noreferrer">Codex</a> economic incentives in the Litepaper and simulations</li>
<li>Orchestrated Status Network incentive structure and smart contract implementation</li>
<li>Started building <a href="https://nomos.tech/" target="_blank" rel="noopener noreferrer">Nomos’s</a> economic model</li>
<li>Consulted and provided analysis of incentives for the Logos Operators ordinals project</li>
<li>Drove discussions on the economic sustainability of <a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a>;
helped define RLN membership and its payment mechanism</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-1">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-1" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<p>In 2025, TKE will continue to support IFT portfolio projects,
working toward economic sustainability while strengthening relationships within the organization.
Additionally, the service unit aims to continue building its external reputation through partnerships and publications of relevant work on the <a href="https://forum.vac.dev/" target="_blank" rel="noopener noreferrer">Vac forum</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="quality-assurance-qa">Quality Assurance (QA)<a href="https://vac.dev/rlog/2024-recap#quality-assurance-qa" class="hash-link" aria-label="Direct link to Quality Assurance (QA)" title="Direct link to Quality Assurance (QA)">​</a></h2>
<p>The QA Service Unit focuses on the development and execution of comprehensive test plans,
including implementing unit and interoperability testing.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-2">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-2" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Matured Waku interoperability testing framework with coverage for all major protocols and features</li>
<li>Began collaboration with Nomos, contributing to unit and integration testing</li>
<li>Partnered with the <a href="https://status.app/" target="_blank" rel="noopener noreferrer">Status</a> team to test message reliability under unstable network conditions</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-2">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-2" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>Extend collaboration with the Waku team on go-waku bindings and message reliability testing</li>
<li>Cement working relationship with the Nomos team through the building of an E2E testing framework for higher-level node validation</li>
<li>Work closely with Status’s QA team to enhance the functional testing framework</li>
<li>Continue work on nim-libp2p testing</li>
<li>Expand collaboration to additional projects</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rfc">RFC<a href="https://vac.dev/rlog/2024-recap#rfc" class="hash-link" aria-label="Direct link to RFC" title="Direct link to RFC">​</a></h2>
<p>The RFC Service Unit takes on the responsibility of shepherding and editing specifications for IFT projects.
The unit acts as a linchpin for ensuring standardized and interoperable protocols within the IFT ecosystem.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-3">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-3" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Working to implement RFC culture across the IFT ecosystem</li>
<li>Began editorial work for several IFT portfolio projects: Status, Nomos, Waku, and Codex.</li>
<li>Reworked our standards with regard to writing RFCs to a consensus-oriented specification system</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-3">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-3" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>Continue to implement RFC culture across the IFT ecosystem</li>
<li>Broaden the number of RFCs produced
– particularly for IFT portfolio projects nearing public releases</li>
<li>Include new projects with the <a href="https://rfc.vac.dev/" target="_blank" rel="noopener noreferrer">rfc-index</a></li>
<li>Encourage external projects requiring RFCs to establish relationships with the service unit</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="applied-cryptography-and-zk-acz">Applied Cryptography and ZK (ACZ)<a href="https://vac.dev/rlog/2024-recap#applied-cryptography-and-zk-acz" class="hash-link" aria-label="Direct link to Applied Cryptography and ZK (ACZ)" title="Direct link to Applied Cryptography and ZK (ACZ)">​</a></h2>
<p>The ACZ Service Unit focuses on cryptographic solutions and zero-knowledge proofs,
enhancing the security, privacy, and trustworthiness of IFT portfolio projects
and contributing to the overall integrity and resilience of the decentralized web ecosystem.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-4">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-4" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Researched a libp2p mix protocol and first proof-of-concept implementation (including ping and GossipSub over mix)</li>
<li>Researched a decentralized version of MLS (message layer security) with a first proof of concept</li>
<li>Released Zerokit <a href="https://github.com/vacp2p/zerokit/releases/tag/v0.6.0" target="_blank" rel="noopener noreferrer">v0.6.0</a>
and <a href="https://github.com/vacp2p/zerokit/releases/tag/v0.5.0" target="_blank" rel="noopener noreferrer">v0.5.0</a></li>
<li>Added <a href="https://github.com/vacp2p/gnark-rln" target="_blank" rel="noopener noreferrer">gnark RLN implementation</a></li>
<li>Released Stealth Address Kit <a href="https://github.com/vacp2p/stealth-address-kit/releases/tag/v0.3.1" target="_blank" rel="noopener noreferrer">v0.3.1</a>,
<a href="https://github.com/vacp2p/stealth-address-kit/releases/tag/v0.2.0" target="_blank" rel="noopener noreferrer">v0.2.0</a>,
and <a href="https://github.com/vacp2p/stealth-address-kit/releases/tag/v0.1.0" target="_blank" rel="noopener noreferrer">v0.1.0</a></li>
<li>Published:<!-- -->
<ul>
<li><a href="https://vac.dev/rlog/rln-light-verifiers/" target="_blank" rel="noopener noreferrer">Verifying RLN Proofs in Light Clients with Subtrees</a></li>
<li><a href="https://vac.dev/rlog/rln-v3/" target="_blank" rel="noopener noreferrer">RLN-v3: Towards a Flexible and Cost-Efficient Implementation</a></li>
</ul>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-4">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-4" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>Ensure libp2p mix protocol is production-ready
and support with the publishing of a paper and blog posts</li>
<li>Ensure decentralized MLS is production-ready
and support with the publishing of a paper and blog posts</li>
<li>Begin explorations of additional research topics</li>
<li>Release <a href="https://github.com/vacp2p/zerokit/issues/271" target="_blank" rel="noopener noreferrer">Zerokit v0.7</a> and future versions</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="p2p">P2P<a href="https://vac.dev/rlog/2024-recap#p2p" class="hash-link" aria-label="Direct link to P2P" title="Direct link to P2P">​</a></h2>
<p>The P2P Service Unit specializes in peer-to-peer technologies
and develops nim-libp2p, improves the libp2p GossipSub protocol, and assists IFT portfolio projects with the integration of P2P network layers.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-5">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-5" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Analysis and work on libp2p GossipSub improvements</li>
<li>Published:<!-- -->
<ul>
<li><a href="https://vac.dev/rlog/gsub-idontwant-perf-eval/" target="_blank" rel="noopener noreferrer">Libp2p GossipSub IDONTWANT Message Performance Impact</a></li>
</ul>
</li>
<li>PR to libp2p specifications about specific lib2p GossipSub improvements we researched and tested <a href="https://github.com/libp2p/specs/pull/654" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/pull/654</a></li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-5">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-5" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>Add new features to nim-libp2p:
QUIC transport, web transport</li>
<li>Update specifications for libp2p GossipSub,
aiming to significantly improve its performance</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="distributed-systems-testing-dst">Distributed Systems Testing (DST)<a href="https://vac.dev/rlog/2024-recap#distributed-systems-testing-dst" class="hash-link" aria-label="Direct link to Distributed Systems Testing (DST)" title="Direct link to Distributed Systems Testing (DST)">​</a></h2>
<p>The DST Service Unit’s primary objective is to assist IFT portfolio projects in understanding the scaling behavior of their nodes within larger networks.
By conducting thorough regression testing, the DST unit helps ensure the reliability and stability of projects.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-6">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-6" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>DST compute resources transitioned from a hosted environment to a dedicated Vac Lab,
enabling better customization of resources and adding significantly more compute power
– enabled much higher and more stable simulations (several hundred nodes to several thousand) and enhanced environmental control.</li>
<li>Maintained monthly regression simulations for both Waku and Nim-libp2p,
helping us to detect several issues and ensure that future versions do not introduce new ones.</li>
<li>Successfully simulated and obtained results for all Waku protocols, relaying feedback to the team.</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-6">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-6" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>More testing and simulations for Codex and Nomos</li>
<li>Develop useful tools for all IFT portfolio projects – e.g. a Log Parser tool and data dashboard</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="nim">Nim<a href="https://vac.dev/rlog/2024-recap#nim" class="hash-link" aria-label="Direct link to Nim" title="Direct link to Nim">​</a></h2>
<p>Several IFT portfolio projects use the Nim ecosystem for its efficiency.
The Nim Service Unit is responsible for the development and maintenance of Nim tooling.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-7">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-7" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Released Nim-libp2p (<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.7.1" target="_blank" rel="noopener noreferrer">v1.7.1</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.7.0" target="_blank" rel="noopener noreferrer">v1.7.0</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.6.0" target="_blank" rel="noopener noreferrer">v1.6.0</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.5.0" target="_blank" rel="noopener noreferrer">v1.5.0</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.4.0" target="_blank" rel="noopener noreferrer">v1.4.0</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.3.0" target="_blank" rel="noopener noreferrer">v1.3.0</a>,
<a href="https://github.com/vacp2p/nim-libp2p/releases/tag/v1.2.0" target="_blank" rel="noopener noreferrer">v1.2.0</a>)</li>
<li>Introduced SAT solver to the Nimble package manager that significantly improves dependency resolution</li>
<li>Nim VSCode Extension</li>
<li>Stabilized Nim Language Server</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="smart-contracts-sc">Smart Contracts (SC)<a href="https://vac.dev/rlog/2024-recap#smart-contracts-sc" class="hash-link" aria-label="Direct link to Smart Contracts (SC)" title="Direct link to Smart Contracts (SC)">​</a></h2>
<p>Vac's Smart Contracts Service Unit ensures the smart contracts deployed across the various IFT portfolio projects are secure, robust, and aligned with project requirements.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="highlights-8">Highlights<a href="https://vac.dev/rlog/2024-recap#highlights-8" class="hash-link" aria-label="Direct link to Highlights" title="Direct link to Highlights">​</a></h3>
<ul>
<li>Deployed the SNT staking protocol testnet following Status's <a href="https://our.status.im/snt-vote-results/" target="_blank" rel="noopener noreferrer">governance vote</a> to develop SNT staking and Status Network</li>
<li>Wrote specifications for <a href="https://github.com/codex-storage/codex-contracts-eth/tree/master/certora/specs" target="_blank" rel="noopener noreferrer">Codex's architectural components</a> and <a href="https://github.com/vacp2p/staking-reward-streamer/tree/main/certora/specs" target="_blank" rel="noopener noreferrer">Status's staking contracts</a></li>
<li>Delivered several learn-up sessions on a variety of topics for IFT contributors, including:<!-- -->
<ul>
<li>Stealth addresses</li>
<li>Tokenized vaults</li>
<li>Rental NFTs</li>
<li>Merkle trees</li>
<li>Account abstraction</li>
<li>EVM deep dive</li>
</ul>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="looking-forward-7">Looking forward<a href="https://vac.dev/rlog/2024-recap#looking-forward-7" class="hash-link" aria-label="Direct link to Looking forward" title="Direct link to Looking forward">​</a></h3>
<ul>
<li>Deploy the SNT staking protocol on the Status Network testnet</li>
<li>Encourage community security audits via contests</li>
<li>Provide smart contract consultation services for IFT portfolio products</li>
<li>Engage in more learn-up sessions to promote org-wide knowledge sharing.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="heading-into-2025">Heading into 2025<a href="https://vac.dev/rlog/2024-recap#heading-into-2025" class="hash-link" aria-label="Direct link to Heading into 2025" title="Direct link to Heading into 2025">​</a></h2>
<p>This year has seen Vac involved with many research, development, and testing undertakings in support of IFT portfolio projects.
The digital public goods that emerge from our efforts not only support the organization itself but are open and free to use by any project that would benefit.</p>
<p>As we move into 2025, we aim to nurture a stronger RFC culture across the IFT to encourage greater collaboration and knowledge sharing among portfolio projects.
Our goal is to serve as an internal conduit of expertise within the organization, supported by a strong RFC culture, maintaining a repository of internal knowledge creation, and identifying and facilitating IFT project synergies.
Such an approach should lead to greater efficiencies across the organization.</p>
<p>We also aim to establish a diverse research community around Vac, and our efforts in this regard are already underway.
In the final quarter of 2024, Vac stepped up its collaboration with the libp2p community and made a concerted effort to engage the community on the <a href="https://forum.vac.dev/" target="_blank" rel="noopener noreferrer">Vac forum</a>.
In 2025, we aim to continue working closely with those communities to which we already have ties, such as the libp2p, Ethereum, and Nim ecosystems.</p>
<p>We look forward to continuing our journey with you!</p>
<p><em>Follow <a href="https://x.com/vacp2p" target="_blank" rel="noopener noreferrer">Vac on X</a>, join us in the <a href="https://discord.gg/FPSXQ9afJE" target="_blank" rel="noopener noreferrer">Vac Discord</a>, or take part in the discussions on the <a href="https://forum.vac.dev/" target="_blank" rel="noopener noreferrer">Vac forum</a> to stay up to date with our research and development progress.</em></p>]]></content>
<author>
<name>Vac</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Vac 101: Climbing Merkle Trees]]></title>
<id>https://vac.dev/rlog/climbing-merkle-trees</id>
<link href="https://vac.dev/rlog/climbing-merkle-trees"/>
<updated>2024-12-30T12:00:00.000Z</updated>
<summary type="html"><![CDATA[In this post, we introduce a crucial data structure used throughout web3.]]></summary>
<content type="html"><![CDATA[<p>In this post, we introduce a crucial data structure used throughout web3.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/climbing-merkle-trees#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>A large amount of data is swapped between users on a blockchain in the form of transactions.
Over the entire life of a blockchain,
the storage space required to maintain a copy of every transaction becomes untenable for most users.
However, the integrity of a blockchain relies on a large pool of users that can validate the blockchain's history from its inception to its present state.
The data representing the blockchain's state is compressed.
This compression addresses the issue of scalability that would otherwise greatly restrict the pool of users.</p>
<p>Data compression alone is not the end goal.
As mentioned, it is essential for users to be able to validate the blockchain's history.
The property of compression and validation was solved in Bitcoin by the use of Merkle trees.
Merkle trees were introduced first by Ralph Merkle in his dissertation [<a href="https://www.ralphmerkle.com/papers/Thesis1979.pdf" target="_blank" rel="noopener noreferrer">1</a>].
A Merkle tree is a data structure that compresses a digest of data to a constant size while still providing a method for proving membership of elements of the digest.
A previous rlog[<a href="https://vac.dev/rlog/rln-light-verifiers/" target="_blank" rel="noopener noreferrer">2</a>] described how Merkle trees with their proof of membership could be used for lightweight clients for RLN.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="tree-structure">Tree structure<a href="https://vac.dev/rlog/climbing-merkle-trees#tree-structure" class="hash-link" aria-label="Direct link to Tree structure" title="Direct link to Tree structure">​</a></h2>
<p>A tree is a special data structure that organizes nodes so that there is exactly one path between any two nodes.
The trees that we consider can be arranged in layers with multiple nodes (children) merged into a single node (parent) in the preceding layer.
A single node exists in the base layer;
this special node is called the root node.
The highest level of the tree consists of childless nodes called leaves.</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/vac101_tree-c39839d4050c3723ccde9d3622de2870.png" width="1017" height="456" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>A binary tree has one additional property:
each nonleaf node has exactly two children nodes.
That is, we assume that nodes in a binary tree are either a parent node with two children or a leaf.
As strange as it sounds, each child node has exactly one parental node.</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/vac101_binary_tree-f2600381c1537895a063761d315201ce.png" width="940" height="467" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>A binary tree with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">2^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> leaves consists of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">n+1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6667em;vertical-align:-0.0833em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> layers.
Additionally, such a tree has <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mrow><mi>n</mi><mo>+</mo><mn>1</mn></mrow></msup><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">2^{n+1}-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8974em;vertical-align:-0.0833em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> nodes.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="merkle-trees">Merkle trees<a href="https://vac.dev/rlog/climbing-merkle-trees#merkle-trees" class="hash-link" aria-label="Direct link to Merkle trees" title="Direct link to Merkle trees">​</a></h2>
<p>A Merkle tree is a specialized tree in which each node contains the evaluation of a hash function.
Merkle trees are usually taken to have a binary tree structure.
As such, the presentation we provide in this section will be for binary trees.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="construction">Construction<a href="https://vac.dev/rlog/climbing-merkle-trees#construction" class="hash-link" aria-label="Direct link to Construction" title="Direct link to Construction">​</a></h3>
<p>In this section, we show how Merkle trees are constructed to compress a digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.
Suppose that the digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> consists of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">2^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> entries;
we assume that the digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> has this many entries since a Merkle tree is a binary tree.
Additionally, each digest can be padded to ensure that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> has the desired number of entries.</p>
<p>Each leaf of the Merkle tree contains the hash of a digest entry.
Each parent node contains the hash of the concatenation of their child nodes.
Through this iterative construction, we reach the root of the tree.
The value contained in the root node is called the root hash.
The root hash is a compressed representation of the digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/vac101_merkle_tree-a7c86f78d5aa8016921924220d6005fc.png" width="1035" height="629" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>Each node in the Merkle tree is computed by taking a hash.
Since a binary tree with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">2^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> leaves has <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mrow><mi>n</mi><mo>+</mo><mn>1</mn></mrow></msup><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">2^{n+1}-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8974em;vertical-align:-0.0833em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> nodes,
then we need to evaluate <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mrow><mi>n</mi><mo>+</mo><mn>1</mn></mrow></msup><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">2^{n+1}-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8974em;vertical-align:-0.0833em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> hashes to construct the Merkle tree.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="merkle-tree-intregrity">Merkle tree intregrity<a href="https://vac.dev/rlog/climbing-merkle-trees#merkle-tree-intregrity" class="hash-link" aria-label="Direct link to Merkle tree intregrity" title="Direct link to Merkle tree intregrity">​</a></h3>
<p>A large quantity of data can be compressed to a single hash value.
A natural question to ask is: could a clever party find another digest that yields a Merkle tree with the same root hash?
If possible, this would compromise the ledger since the blockchain's history could be altered.
Fortunately, Merkle trees are quite secure.
In fact, Merkle trees can be used to both bind and hide a digest.</p>
<p>The Merkle tree is able to bind a digest with one of the properties of hash functions (see our previous Vac 101 [<a href="https://vac.dev/rlog/vac101-fiat-shamir#hash-functions" target="_blank" rel="noopener noreferrer">3</a>] for information on hash functions).
A hash function is collision resistant; it is infeasible for a malicious party to find two values share the same hash value.</p>
<p>This collision resistance property, essentially, fixes the input to each leaf and into their parent, their parent's parent, and so on.</p>
<p>In certain applications,
it may be desirable for the digest of a Merkle tree to be kept confidential.
This is achieved with the preimage resistant property of hash functions.
A hash function is preimage resistant provided that it is difficult to reverse the hashing operation.
It would be necessary for a malicious party to find preimages to each node starting from the root node to determine the original digest.</p>
<p>Now, we see that Merkle trees are secured structures that are tamper resistant.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-of-membership">Proof of membership<a href="https://vac.dev/rlog/climbing-merkle-trees#proof-of-membership" class="hash-link" aria-label="Direct link to Proof of membership" title="Direct link to Proof of membership">​</a></h3>
<p>An interesting and critical property of Merkle trees is their ability to prove that any piece of data is part of its digest.
This can be done with logarithmic storage and logarithmic computation time.</p>
<p>Suppose that we want to show that data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> is part of the Merkle tree's digest.
Additionally, suppose that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">hash</span></span></span></span></span> is the hash function used to construct the tree.
We assume that the hash function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">hash</span></span></span></span></span> can be computed in constant-time for any input.</p>
<p>Suppose that a prover provides data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> to a verifier,
and tells the verifier that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> corresponds to the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span>th leaf of the Merkle tree.
For the verifier to be convinced that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> is part of the digest, he needs to be able to construct the tree's root hash using <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">hash</span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> and some additional information from the prover.
Specifically, the prover must provide the sibling hashes for each value that the verifier can compute.
This enables the verifier to compute the parents of the siblings that the prover provides and the values that he was able to produce himself.
The last of the computed parents is the root.</p>
<p>The leaf index <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span> indicates whether a hash value provided by the prover is a left or right sibling.
This is done by looking at the binary expansion of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span></span></span></span>.</p>
<p>The verifier can compute the leaf <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>h</mi><mn>0</mn></msub><mo>=</mo><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi mathvariant="normal">ℓ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_0 = \mathsf{hash}(\ell)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">hash</span></span><span class="mopen">(</span><span class="mord">ℓ</span><span class="mclose">)</span></span></span></span>.
Next, using <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>h</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">h_0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>'s sibling, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>h</mi><mn>0</mn><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup></mrow><annotation encoding="application/x-tex">h'_0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.2481em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7519em"><span style="top:-2.4519em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span></span></span></span>, provided by the prover,
the verifier can compute <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>h</mi><mn>1</mn></msub><mo>=</mo><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><msub><mi>h</mi><mn>0</mn></msub><mi mathvariant="normal">∥</mi><msubsup><mi>h</mi><mn>0</mn><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_1 = \mathsf{hash}(h_0 \|h'_0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0019em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord">∥</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7519em"><span style="top:-2.4519em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>h</mi><mn>1</mn></msub><mo>=</mo><mrow><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><msubsup><mi>h</mi><mn>0</mn><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup><mi mathvariant="normal">∥</mi><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_1 = \mathsf{hash}(h'_0 \| h_0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0019em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7519em"><span style="top:-2.4519em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span><span class="mord">∥</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>
depending on whether <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>h</mi><mn>0</mn><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msubsup></mrow><annotation encoding="application/x-tex">h'_0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.2481em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7519em"><span style="top:-2.4519em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span></span></span></span> is a left or right sibling.
This pathing continues until the verifier either successfully computes the root hash (in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">n+1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6667em;vertical-align:-0.0833em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> hashes) or fails to do so.</p>
<p>The prover has to provide <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> sibling nodes for the proof of membership.</p>
<p>There is a key detail that is essential for the proof of membership to be secure.
The root hash has to be provided to the verifier prior to the selection of data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span>.
Otherwise, the prover could generate a series of hash values (with the corresponding root hash) to forge a proof of membership.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="capped-proof-of-membership">Capped proof of membership<a href="https://vac.dev/rlog/climbing-merkle-trees#capped-proof-of-membership" class="hash-link" aria-label="Direct link to Capped proof of membership" title="Direct link to Capped proof of membership">​</a></h4>
<p>Polygon provides an implementation [<a href="https://github.com/0xPolygonZero/plonky2/blob/main/plonky2/src/hash/merkle_tree.rs" target="_blank" rel="noopener noreferrer">4</a>] of a shortened proof of membership with a slight modification.
A specific layer of the Merkle tree is published instead of just the root hash.
By doing this, a capped proof of membership is just the path from leaf to the published layer.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="extensions-of-merkle-trees">Extensions of Merkle trees<a href="https://vac.dev/rlog/climbing-merkle-trees#extensions-of-merkle-trees" class="hash-link" aria-label="Direct link to Extensions of Merkle trees" title="Direct link to Extensions of Merkle trees">​</a></h2>
<p>Merkle trees can be extended in multiple ways.
In this section, we explore a select few of these extensions.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="sparse-merkle-trees">Sparse Merkle trees<a href="https://vac.dev/rlog/climbing-merkle-trees#sparse-merkle-trees" class="hash-link" aria-label="Direct link to Sparse Merkle trees" title="Direct link to Sparse Merkle trees">​</a></h3>
<p>A sparse Merkle tree (SMT) is a special Merkle tree that can be used to represent digests with nonconsecutive entries.
Specifically, each digest entry has a particular leaf index.
For simplicity, we assume that the index value is computed by taking the hash of the entry.
We note that this is a sorted SMT.</p>
<p>Let <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> denote the number of bits that a hash value can possess. This means that our SMT can have at most <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">2^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> leaves.</p>
<p>An SMT is treated as a Merkle tree in which each entry is placed in the leaf corresponding to its hash value, and the other entries have a <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">u</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi></mrow><annotation encoding="application/x-tex">\mathsf{null}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">null</span></span></span></span></span> marker inserted in.
This means that we can prove membership in the way described.
However, we can also prove nonmembership of an element by showing that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">u</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi></mrow><annotation encoding="application/x-tex">\mathsf{null}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">null</span></span></span></span></span> is located in the element's hash location.
The crucial difference between a sorted and unsorted SMT is that the unsorted variant cannot be used to prove nonmembership.</p>
<p>We can take advantage of the sparse nature of SMTs to provide shortened proofs.
Specifically, it is unlikely for entries to cluster together.
Thus, it is efficient to maintain a list of values:</p>
<table><thead><tr><th>Null values</th></tr></thead><tbody><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mn>0</mn></msub><mo>:</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi><mo stretchy="false">(</mo><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">u</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi><mo stretchy="false">)</mo></mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">d_0 := \mathsf{Hash(null)},</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span><span class="mopen">(</span><span class="mord mathsf">null</span><span class="mclose">)</span></span><span class="mpunct">,</span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mn>1</mn></msub><mo>:</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi><mo stretchy="false">(</mo><msub><mi mathvariant="sans-serif">d</mi><mn mathvariant="sans-serif">0</mn></msub><mi mathvariant="sans-serif">∥</mi><mi mathvariant="sans-serif">∥</mi><msub><mi mathvariant="sans-serif">d</mi><mn mathvariant="sans-serif">0</mn></msub><mo stretchy="false">)</mo></mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">d_1 := \mathsf{Hash(d_0 \|\| d_0)},</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span><span class="mopen">(</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathsf mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord">∥∥</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathsf mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span><span class="mpunct">,</span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mn>2</mn></msub><mo>:</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi><mo stretchy="false">(</mo><msub><mi mathvariant="sans-serif">d</mi><mn mathvariant="sans-serif">1</mn></msub><mi mathvariant="sans-serif">∥</mi><mi mathvariant="sans-serif">∥</mi><msub><mi mathvariant="sans-serif">d</mi><mn mathvariant="sans-serif">1</mn></msub><mo stretchy="false">)</mo></mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">d_2 := \mathsf{Hash(d_1 \|\| d_1)},</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span><span class="mopen">(</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathsf mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord">∥∥</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathsf mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span><span class="mpunct">,</span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">⋮</mi><mpadded height="0em" voffset="0em"><mspace mathbackground="black" width="0em" height="1.5em"></mspace></mpadded></mrow><annotation encoding="application/x-tex">\vdots</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.53em;vertical-align:-0.03em"></span><span class="mord"><span class="mord">⋮</span><span class="mord rule" style="border-right-width:0em;border-top-width:1.5em;bottom:0em"></span></span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mrow><mi>n</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>:</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi><mo stretchy="false">(</mo><msub><mi mathvariant="sans-serif">d</mi><mrow><mi mathvariant="sans-serif">n</mi><mo>−</mo><mn mathvariant="sans-serif">2</mn></mrow></msub><mi mathvariant="sans-serif">∥</mi><mi mathvariant="sans-serif">∥</mi><msub><mi mathvariant="sans-serif">d</mi><mrow><mi mathvariant="sans-serif">n</mi><mo>−</mo><mn mathvariant="sans-serif">2</mn></mrow></msub><mo stretchy="false">)</mo></mrow><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">d_{n-1} := \mathsf{Hash(d_{n-2} \|\| d_{n-2})}.</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9028em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span><span class="mopen">(</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathsf mtight">n</span><span class="mbin mtight">−</span><span class="mord mathsf mtight">2</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mord">∥∥</span><span class="mord"><span class="mord mathsf">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3089em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathsf mtight">n</span><span class="mbin mtight">−</span><span class="mord mathsf mtight">2</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mclose">)</span></span><span class="mord">.</span></span></span></span></td></tr></tbody></table>
<p>Each of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>d</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">d_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>'s represents the root hash of a Merkle tree with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>i</mi></msup></mrow><annotation encoding="application/x-tex">2^i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8247em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8247em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span></span></span></span></span></span></span></span> leaves containing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">u</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi></mrow><annotation encoding="application/x-tex">\mathsf{null}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">null</span></span></span></span></span>.
These values can be used to shorten the time needed to construct an SMT and the length of proofs.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-of-nonmembership">Proof of nonmembership<a href="https://vac.dev/rlog/climbing-merkle-trees#proof-of-nonmembership" class="hash-link" aria-label="Direct link to Proof of nonmembership" title="Direct link to Proof of nonmembership">​</a></h3>
<p>In the first Vac 101 [<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters" target="_blank" rel="noopener noreferrer">5</a>], we examined Bloom and Cuckoo filters that could be used for proof of membership and nonmembership.
However, the proof of membership may result in false positives due to collisions.
This would affect nonmembership proofs as well.
Sparse Merkle trees can be adapted to provide greater assurance that a given piece of data is not a member of the digest.</p>
<p>Why is sorting essential?
The sorting mechanism of data can be arbitrarily chosen.
However, it is essential that there are no gaps in the ordering.
The maximum number of elements that could ever exist in the digest must be known.
A simple method for this is to use a hash function to provide fingerprints to the data.
Each hash using either SHA-256 or Keccak has 256-bits.
Our entire digest could consist of a maximum of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>256</mn></msup></mrow><annotation encoding="application/x-tex">2^{256}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">256</span></span></span></span></span></span></span></span></span></span></span></span> entries.
This assumes that our digest does not contain collisions.</p>
<p>The fingerprint of a piece of data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> indicates which leaf of the SMT it is contained in.
This means that a nonmembership of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> in the SMT becomes a matter of proving that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">u</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi></mrow><annotation encoding="application/x-tex">\mathsf{null}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">null</span></span></span></span></span> is contained in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span>'s location.</p>
<p>It is crucial for the SMT to be sorted.
Otherwise, a malicious party can append the entry <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> to a random location.
This allows for the malicious party to provide contradictory proofs that prove both membership and nonmembership.
We note that the requirement that an SMT is sorted may be too strong of an assumption in centralized cases.
However, sortedness is a necessary property of SMTs for decentralized systems.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="verkle-trees">Verkle Trees<a href="https://vac.dev/rlog/climbing-merkle-trees#verkle-trees" class="hash-link" aria-label="Direct link to Verkle Trees" title="Direct link to Verkle Trees">​</a></h3>
<p>A proof of membership grows in length as the Merkle tree grows.
The most obvious approach to remedy this scalability issue is to use Merkle trees in which each node has more than two children.
However, this does not fix the issue.
A proof of membership in a <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span>-nary Merkle tree [<a href="https://math.mit.edu/research/highschool/primes/materials/2018/Kuszmaul.pdf" target="_blank" rel="noopener noreferrer">6</a>] (each node has <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> children) has a proof size <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>log</mi><mo>⁡</mo></mrow><mi>k</mi></msub><mo stretchy="false">(</mo><mi>n</mi><mo stretchy="false">)</mo><mo stretchy="false">(</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\log_k(n)(k-1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop"><span class="mop">lo<span style="margin-right:0.01389em">g</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2441em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mclose">)</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">)</span></span></span></span>.
The multiple <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">k-1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7778em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> is the number of silbings that a node has on each layer.
Hence, the proof size grows faster than a logarithmic function of the digest size.</p>
<p>An alternate approach is to use a different data structure: Verkle trees [<a href="https://math.mit.edu/research/highschool/primes/materials/2018/Kuszmaul.pdf" target="_blank" rel="noopener noreferrer">6</a>].
A Verkle tree replaces hash functions with polynomial commitments [<a href="https://ethresear.ch/t/using-polynomial-commitments-to-replace-state-roots/7095" target="_blank" rel="noopener noreferrer">7</a>, <a href="https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html" target="_blank" rel="noopener noreferrer">8</a>].
We will explore Verkle trees in a future Vac 101 edition.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/climbing-merkle-trees#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li>
<ol>
<li><a href="https://www.ralphmerkle.com/papers/Thesis1979.pdf" target="_blank" rel="noopener noreferrer">Secrecy, Authentication, and Public Key Systems</a></li>
</ol>
</li>
<li>
<ol start="2">
<li><a href="https://vac.dev/rlog/rln-light-verifiers/" target="_blank" rel="noopener noreferrer">Verifying RLN Proofs in Light Clients with Subtrees</a></li>
</ol>
</li>
<li>
<ol start="3">
<li><a href="https://vac.dev/rlog/vac101-fiat-shamir#hash-functions" target="_blank" rel="noopener noreferrer">Vac 101: Transforming an Interactive Protocol to a Noninteractive Argument</a></li>
</ol>
</li>
<li>
<ol start="4">
<li><a href="https://github.com/0xPolygonZero/plonky2/blob/main/plonky2/src/hash/merkle_tree.rs" target="_blank" rel="noopener noreferrer">Capped merkle tree in Plonky2</a></li>
</ol>
</li>
<li>
<ol start="5">
<li><a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters" target="_blank" rel="noopener noreferrer">Vac 101: Membership with Bloom Filters and Cuckoo Filters</a></li>
</ol>
</li>
<li>
<ol start="6">
<li><a href="https://math.mit.edu/research/highschool/primes/materials/2018/Kuszmaul.pdf" target="_blank" rel="noopener noreferrer">Verkle Trees</a></li>
</ol>
</li>
<li>
<ol start="7">
<li><a href="https://ethresear.ch/t/using-polynomial-commitments-to-replace-state-roots/7095" target="_blank" rel="noopener noreferrer">Using polynomial commitments to replace state roots</a></li>
</ol>
</li>
<li>
<ol start="8">
<li><a href="https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html" target="_blank" rel="noopener noreferrer">KZG polynomial commitments</a></li>
</ol>
</li>
<li>
<ol start="9">
<li><a href="https://github.com/o1-labs/verkle-tree" target="_blank" rel="noopener noreferrer">O1 labs' Verkle Tree repo</a></li>
</ol>
</li>
</ul>]]></content>
<author>
<name>Marvin</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Large Message Handling in GossipSub: Potential Improvements]]></title>
<id>https://vac.dev/rlog/gsub-largemsg-improvements</id>
<link href="https://vac.dev/rlog/gsub-largemsg-improvements"/>
<updated>2024-10-31T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Large Message Handling in GossipSub: Potential Improvements]]></summary>
<content type="html"><![CDATA[<p>Large Message Handling in GossipSub: Potential Improvements</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivation">Motivation<a href="https://vac.dev/rlog/gsub-largemsg-improvements#motivation" class="hash-link" aria-label="Direct link to Motivation" title="Direct link to Motivation">​</a></h2>
<p>The challenge of large message transmissions in GossipSub leads to longer than expected network-wide message dissemination times (and relatively higher fluctuations).
It is particularly relevant for applications that require on-time, network-wide dissemination of large messages,
such as Ethereum and Waku [<a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">1</a>,<a href="https://docs.waku.org/research/research-and-studies/message-propagation/" target="_blank" rel="noopener noreferrer">2</a>].</p>
<p>This matter has been extensively discussed in the libp2p community [<a href="https://github.com/libp2p/rust-libp2p/pull/3666" target="_blank" rel="noopener noreferrer">3</a>,
<a href="https://github.com/sigp/lighthouse/pull/4383" target="_blank" rel="noopener noreferrer">4</a>,
<a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw" target="_blank" rel="noopener noreferrer">5</a>,
<a href="https://github.com/status-im/nim-libp2p/issues/850" target="_blank" rel="noopener noreferrer">6</a>,
<a href="https://github.com/vacp2p/nim-libp2p/pull/911" target="_blank" rel="noopener noreferrer">7</a>,
<a href="https://github.com/vacp2p/nim-libp2p/issues/859" target="_blank" rel="noopener noreferrer">8</a>],
and numerous improvements have been considered (or even incorporated) for the GossipSub protocol to enable efficient large-message propagation
[<a href="https://github.com/libp2p/rust-libp2p/pull/3666" target="_blank" rel="noopener noreferrer">3</a>,
<a href="https://github.com/vacp2p/nim-libp2p/pull/911" target="_blank" rel="noopener noreferrer">7</a>,
<a href="https://hackmd.io/@gRwfloEASH6NWWS_KJxFGQ/B18wdnNDh" target="_blank" rel="noopener noreferrer">9</a>,
<a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52" target="_blank" rel="noopener noreferrer">10</a>].</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="problem-description">Problem description<a href="https://vac.dev/rlog/gsub-largemsg-improvements#problem-description" class="hash-link" aria-label="Direct link to Problem description" title="Direct link to Problem description">​</a></h2>
<p>Sending a message to N peers involves approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⌈</mo><msub><mrow><mi>log</mi><mo>⁡</mo></mrow><mi>D</mi></msub><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo stretchy="false">⌉</mo></mrow><annotation encoding="application/x-tex">\lceil \log_D(N) \rceil</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">⌈</span><span class="mop"><span class="mop">lo<span style="margin-right:0.01389em">g</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2342em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2441em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)⌉</span></span></span></span> rounds,
with approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>D</mi><mo>−</mo><mn>1</mn><msup><mo stretchy="false">)</mo><mrow><mi>X</mi><mo>−</mo><mn>1</mn></mrow></msup><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">(D-1)^{X-1} \times D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0913em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.07847em">X</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> transmissions in each round,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi><mo separator="true">,</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">X, D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> represent the round number and mesh size.</p>
<p>Transmitting to a higher number of peers (floodpublish) can theoretically reduce latency by increasing the transmissions in each round to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>D</mi><mo>−</mo><mn>1</mn><msup><mo stretchy="false">)</mo><mrow><mi>X</mi><mo>−</mo><mn>1</mn></mrow></msup><mo>×</mo><mo stretchy="false">(</mo><mi>F</mi><mo>+</mo><mi>D</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(D-1)^{X-1} \times (F+D)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0913em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.07847em">X</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">F</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mclose">)</span></span></span></span>,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi></mrow><annotation encoding="application/x-tex">F</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">F</span></span></span></span> represents the number of peers included in floodpublish.</p>
<p>This arrangement works fine for relatively small/moderate message sizes.
However, as message sizes increase, significant rises and fluctuations in network-wide message dissemination time are seen.</p>
<p>Interestingly, a higher <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi></mrow><annotation encoding="application/x-tex">F</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">F</span></span></span></span> can also degrade performance in this situation.</p>
<p>Several aspects contribute to this behavior:</p>
<ol>
<li>
<p>Ideally, a message transmission to a single peer concludes in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mn>1</mn></msub><mo>=</mo><mfrac><mi>L</mi><mi>R</mi></mfrac><mo>+</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">\tau_1 = \frac {L}{R}+P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">L</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span> (ignoring any message processing time),
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo separator="true">,</mo><mi>R</mi><mo separator="true">,</mo><mi>P</mi></mrow><annotation encoding="application/x-tex">L, R, P</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal">L</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">P</span></span></span></span> represent message size, data rate, and link latency.
Therefore, the time required for sending a message on a 100Mbps link with 100ms latency
jumps from <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>τ</mi><mn>1</mn><mrow><mn>10</mn><mi>K</mi><mi>B</mi></mrow></msubsup><mo>=</mo><mn>100.8</mn><mi>m</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">\tau_1^{10KB} = 100.8ms</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0894em;vertical-align:-0.2481em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-2.4519em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span><span class="mord mathnormal mtight" style="margin-right:0.07153em">K</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">B</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">100.8</span><span class="mord mathnormal">m</span><span class="mord mathnormal">s</span></span></span></span> for a 10KB message to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>τ</mi><mn>1</mn><mrow><mn>1</mn><mi>M</mi><mi>B</mi></mrow></msubsup><mo>=</mo><mn>180</mn><mi>m</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">\tau_1^{1MB} = 180ms</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0894em;vertical-align:-0.2481em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-2.4519em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">MB</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">180</span><span class="mord mathnormal">m</span><span class="mord mathnormal">s</span></span></span></span> for a 1MB message.
For <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> peers, the transmission time increases to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>τ</mi><mi>D</mi><mrow><mn>1</mn><mi>M</mi><mi>B</mi></mrow></msubsup><mo>=</mo><mo stretchy="false">(</mo><mn>80</mn><mo>×</mo><mi>D</mi><mo stretchy="false">)</mo><mo>+</mo><mn>100</mn><mi>m</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">\tau_D^{1MB} = (80 \times D) + 100ms</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1167em;vertical-align:-0.2753em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-2.4247em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">MB</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2753em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">80</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">100</span><span class="mord mathnormal">m</span><span class="mord mathnormal">s</span></span></span></span>,
triggering additional queuing delays (proportional to the transmission queue size) during each transmission round.</p>
</li>
<li>
<p>In practice, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>τ</mi><mn>1</mn><mrow><mn>1</mn><mi>M</mi><mi>B</mi></mrow></msubsup></mrow><annotation encoding="application/x-tex">\tau_1^{1MB}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0894em;vertical-align:-0.2481em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-2.4519em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1</span><span class="mord mathnormal mtight" style="margin-right:0.05017em">MB</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2481em"><span></span></span></span></span></span></span></span></span></span> sometimes rises to several hundred milliseconds,
further exaggerating the abovementioned queuing delays.
This rise is because TCP congestion avoidance algorithms usually limit maximum in-flight bytes in a round trip time (RTT)
based on the congestion window (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>) and maximum segment size (MSS) to approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub><mo>×</mo><mi>M</mi><mi>S</mi><mi>S</mi></mrow><annotation encoding="application/x-tex">{C_{wnd} \times MSS}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mord mathnormal" style="margin-right:0.05764em">MSS</span></span></span></span></span>,
with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> rising with the data transfer for each flow.
Consequently, sending the same message through a newly established (cold) connection takes longer.
The message transfer time lowers as the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> grows.
Therefore, performance-improvement practices such as floodpublish, frequent mesh adjustment, and lazy sending
typically result in longer than expected message dissemination times for large messages (due to cold connections).
It is also worth mentioning that some TCP variants reset their <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> after different periods of inactivity.</p>
</li>
<li>
<p>Theoretically, the message transmission time to D peers <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>τ</mi><mi>D</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(\tau_D)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> remains the same
even if the message is relayed sequentially to all peers or simultaneous transmissions are carried out,
i.e., <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub><mo>=</mo><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>D</mi></msubsup><msub><mi>τ</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D = \sum_{i=1}^{D} \tau_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2809em;vertical-align:-0.2997em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.9812em"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.2029em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2997em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>
However, sequential transmissions finish early for individual peers, allowing them to relay early.
This may result in quicker network-wide message dissemination.</p>
</li>
<li>
<p>A realistic network comprises nodes with dissimilar capabilities (bandwidth, link latency, compute, etc.).
As the message disseminates, it's not uncommon for some peers to receive it much earlier than others.
Early gossip (IHAVE announcements) may bring in many IWANT requests to the early receivers (even from peers already receiving the same message),
which adds to their workload.</p>
</li>
<li>
<p>A busy peer (with a sizeable outgoing message queue) will enqueue (or simultaneously transfer) newly scheduled outgoing messages.
As a result, already scheduled messages are prioritized over those published by the peer itself,
introducing a significant initial delay to the locally published messages.
Enqueuing IWANT replies to the outgoing message queue can further exaggerate the problem.
The lack of adaptiveness and standardization in outgoing message prioritization are key factors that can lead to noticeable inconsistency
in message dissemination latency at each hop, even in similar network conditions.</p>
</li>
<li>
<p>Message size directly contributes to peers' workloads in terms of processing and transmission time.
It also raises the probability of simultaneous redundant transmissions to the same peer,
resulting in bandwidth wastage, congestion, and slow message propagation to the network.
Moreover, the benefits of sequential message relaying can be compromised by prioritizing slow (or busy) peers.</p>
</li>
<li>
<p>Most use cases necessitate validating received messages before forwarding them to the next-hop peers.
For a higher message transfer time <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>τ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(\tau )</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="mclose">)</span></span></span></span>, this store-and-forward delay accumulates across the hops traveled by the message.</p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="possible-improvements">Possible improvements<a href="https://vac.dev/rlog/gsub-largemsg-improvements#possible-improvements" class="hash-link" aria-label="Direct link to Possible improvements" title="Direct link to Possible improvements">​</a></h2>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-minimizing-transfer-time-for-large-messages">1. Minimizing transfer time for large messages<a href="https://vac.dev/rlog/gsub-largemsg-improvements#1-minimizing-transfer-time-for-large-messages" class="hash-link" aria-label="Direct link to 1. Minimizing transfer time for large messages" title="Direct link to 1. Minimizing transfer time for large messages">​</a></h3>
<p>The impact of message size and achievable data rate on message transmit time <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>τ</mi></mrow><annotation encoding="application/x-tex">\tau</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.1132em">τ</span></span></span></span> is crucial
as this time accumulates due to the store-and-forward delay introduced at intermediate hops.</p>
<p>Some possible improvements to minimize overall message dissemination latency include:</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-message-fragmentation">a. Message fragmentation<a href="https://vac.dev/rlog/gsub-largemsg-improvements#a-message-fragmentation" class="hash-link" aria-label="Direct link to a. Message fragmentation" title="Direct link to a. Message fragmentation">​</a></h4>
<p>In a homogeneous network, network-wide message dissemination time (ignoring any processing delays)
can be simplified to roughly <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>δ</mi><mo>≈</mo><msub><mi>δ</mi><mrow><mi>T</mi><mi>x</mi></mrow></msub><mo>+</mo><msub><mi>P</mi><mi>h</mi></msub></mrow><annotation encoding="application/x-tex">\delta \approx \delta_{Tx} + P_h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03785em">δ</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03785em">δ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0379em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">h</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mrow><mi>T</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\delta_{Tx}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03785em">δ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0379em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> represents accumulative message transmit time denoted as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mrow><mi>T</mi><mi>x</mi></mrow></msub><mo>=</mo><mfrac><mi>S</mi><mi>R</mi></mfrac><mo>×</mo><mi>h</mi></mrow><annotation encoding="application/x-tex">\delta_{Tx} = \frac{S}{R} \times h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03785em">δ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0379em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em">S</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span>,
with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mo separator="true">,</mo><mi>R</mi></mrow><annotation encoding="application/x-tex">S, R</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span></span></span></span> being the data size and data rate,
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mo separator="true">,</mo><msub><mi>P</mi><mi>h</mi></msub></mrow><annotation encoding="application/x-tex">h, P_h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">h</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.1389em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">h</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> being the number of hops in the longest path and message propagation time through the longest path.</p>
<p>Partitioning a large message into n fragments reduces a single fragment transmit time to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><msub><mi>δ</mi><mrow><mi>T</mi><mi>x</mi></mrow></msub><mi>n</mi></mfrac></mrow><annotation encoding="application/x-tex">\frac{\delta_{Tx}}{n}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2414em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8964em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.4103em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03785em">δ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3567em;margin-left:-0.0379em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1433em"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>.
As a received fragment can be immediately relayed by the receiver (while the sender is still transmitting the remaining fragments),
it reduces the transmit time to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mrow><mi>T</mi><mi>x</mi></mrow></msub><mo>=</mo><mfrac><mi>S</mi><mi>R</mi></mfrac><mo>×</mo><mfrac><mrow><mn>2</mn><mi>h</mi><mo>−</mo><mn>1</mn></mrow><mi>n</mi></mfrac></mrow><annotation encoding="application/x-tex">\delta_{Tx} = \frac{S}{R} \times \frac{2h-1}{n}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03785em">δ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0379em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mord mathnormal mtight">x</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2173em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8723em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.05764em">S</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.2251em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8801em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">n</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.394em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">2</span><span class="mord mathnormal mtight">h</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>.</p>
<p>This time reduction is mainly attributed to the smaller store-and-forward delay involved in fragment transmissions.</p>
<p>However, it is worth noting that many applications require each fragment to be individually verifiable.
At the same time, message fragmentation allows a malicious peer to never relay some fragments of a message,
which can lead to a significant rise in the application's receive buffer size.</p>
<p>Therefore, message fragmentation requires a careful tradeoff analysis between time and risks.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-message-staggering">b. Message staggering<a href="https://vac.dev/rlog/gsub-largemsg-improvements#b-message-staggering" class="hash-link" aria-label="Direct link to b. Message staggering" title="Direct link to b. Message staggering">​</a></h4>
<p>Considering the same bandwidth, the time <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> required for sending a message to D peers stays the same,
even if we relay to all peers in parallel or send sequentially to the peers, i.e., <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mi>D</mi></msub><mo>=</mo><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>D</mi></msubsup><msub><mi>τ</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\tau_D = \sum_{i=1}^{D} \tau_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2809em;vertical-align:-0.2997em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.9812em"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.2029em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2997em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.</p>
<p>However, sequential relaying results in quicker message reception at individual peers (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>τ</mi><mn>1</mn></msub><mo>≈</mo><mfrac><msub><mi>τ</mi><mi>D</mi></msub><mi>D</mi></mfrac></mrow><annotation encoding="application/x-tex">\tau_1 \approx \frac{\tau_D}{D}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6331em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.1132em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0567em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7117em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.4103em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3567em;margin-left:-0.1132em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1433em"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span>) due to bandwidth concentration for a particular peer.
So, the receiver can start relaying early to its mesh members while the original sender is still sending the message to other peers.</p>
<p>As a result, after every <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><msub><mi>τ</mi><mi>D</mi></msub><mi>D</mi></mfrac></mrow><annotation encoding="application/x-tex">\frac{\tau_D}{D}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0567em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7117em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.4103em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3567em;margin-left:-0.1132em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1433em"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span> milliseconds,
the number of peers receiving the message increases by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mi>X</mi></msup><mtext>&nbsp;</mtext><mi mathvariant="normal">∀</mi><mtext>&nbsp;</mtext><mi>X</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mi>D</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">2^X\ \forall\ X \in \{0, D-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8804em;vertical-align:-0.0391em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.07847em">X</span></span></span></span></span></span></span></span><span class="mspace">&nbsp;</span><span class="mord">∀</span><span class="mspace">&nbsp;</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span> and by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mi>X</mi><mo>−</mo><mi>D</mi></mrow><mrow><mi>X</mi><mo>−</mo><mn>1</mn></mrow></msubsup><msub><mi>λ</mi><mi>k</mi></msub><mtext>&nbsp;</mtext><mi mathvariant="normal">∀</mi><mtext>&nbsp;</mtext><mi>X</mi><mo>≥</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">\sum_{k=X-D}^{X-1} \lambda_k\ \forall\ X \geq D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3393em;vertical-align:-0.358em"></span><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.9812em"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mrel mtight">=</span><span class="mord mathnormal mtight" style="margin-right:0.07847em">X</span><span class="mbin mtight">−</span><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span style="top:-3.2029em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.07847em">X</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.358em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal">λ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace">&nbsp;</span><span class="mord">∀</span><span class="mspace">&nbsp;</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≥</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.
Here, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> represents message transmission round <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi><mo>=</mo><mi>i</mi><mo>⋅</mo><mfrac><msub><mi>τ</mi><mi>D</mi></msub><mi>D</mi></mfrac><mo>∣</mo><mi>i</mi><mo>∈</mo><msub><mi mathvariant="double-struck">N</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">X = i \cdot \frac{\tau_D}{D} \mid i \in \mathbb{N}_0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6595em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.095em;vertical-align:-0.345em"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.7117em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.4103em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.1132em">τ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3448em"><span style="top:-2.3567em;margin-left:-0.1132em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1433em"><span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.345em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∣</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathbb">N</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>λ</mi><mi>k</mi></msub></mrow><annotation encoding="application/x-tex">\lambda_k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">λ</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> represents the number of peers that received the message in round <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span>.</p>
<p>It is worth noting that a realistic network imposes certain constraints on staggering for peers.
For instance, in a network with dissimilar peer capabilities,
placing a slow peer (also in cases where many senders simultaneously select the same peer) at the head of the transmission queue
may result in head-of-line blocking for the message queue.</p>
<p>At the same time, early receivers get many IWANT requests, increasing their workload.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="c-message-prioritization-for-slow-senders">c. Message prioritization for slow senders<a href="https://vac.dev/rlog/gsub-largemsg-improvements#c-message-prioritization-for-slow-senders" class="hash-link" aria-label="Direct link to c. Message prioritization for slow senders" title="Direct link to c. Message prioritization for slow senders">​</a></h4>
<p>A slow peer often struggles with a backlog of messages in the outgoing message queue(s) for mesh members.
Any new message transmission at this stage (especially the locally published messages) gets delayed.
Adaptive message-forwarding can help such peers prioritize traffic to minimize latency for essential message transfers.</p>
<p>For instance, any GossipSub peer will likely receive every message from multiple senders,
leading to redundant transmissions [<a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921" target="_blank" rel="noopener noreferrer">11</a>].
Implementing efficient strategies (only for slow senders) like lazy sending
and prioritizing locally published messages/IWANT replies over already queued messages
can help minimize outgoing message queue sizes and optimize bandwidth for essential message transfers.</p>
<p>A peer can identify itself as a slow peer by using any bandwidth estimation approach
or simply setting an outgoing message queue threshold for all mesh members.</p>
<p>Eliminating/deprioritizing some messages can lower a peer's score,
but it also earns the peer an overall better score by achieving some early message transfers.<br>
<!-- -->For instance, sending many near-first messages can only save a peer from a deficit penalty.
On the other hand, sending only one message (assuming MeshMessageDeliveriesThreshold defaults to 1)
as the first delivered message can add to the accumulative peer score.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-mitigating-transport-issues">2. Mitigating transport issues<a href="https://vac.dev/rlog/gsub-largemsg-improvements#2-mitigating-transport-issues" class="hash-link" aria-label="Direct link to 2. Mitigating transport issues" title="Direct link to 2. Mitigating transport issues">​</a></h3>
<p>Congestion avoidance algorithms used in various TCP versions directly influence achievable throughput and message transfer time
as maximum unacknowledged in-flight bytes are based on the congestion window <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(C_{wnd})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span> size.</p>
<p>Rapid adaptation of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to the available network conditions can help lower message dissemination latency.</p>
<p>Therefore, selecting a more suitable TCP variant like BBR,
which is known for its ability to dynamically adjust the congestion window based on network conditions,
can significantly enhance GossipSub's performance.</p>
<p>At the same time, parameters like receive window scaling and initial <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> also impact message transfer time,
but these are usually OS-specific system-wide choices.</p>
<p>One possible solution is to raise <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> by exchanging data over the newly established connection.
This data may involve useful details like peer exchange information and gossip to build initial trust,
or GossipSub can use some dummy data to raise <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to a reasonable level.</p>
<p>It's important to understand that some TCP variants reset <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> after specific periods of inactivity [<a href="https://datatracker.ietf.org/doc/html/rfc2581#section-4.1" target="_blank" rel="noopener noreferrer">12</a>].
This can lead to a decline in TCP's performance for applications
that generate traffic after intervals long enough to trigger the resetting of the congestion window.</p>
<p>Implementing straightforward measures like transport-level ping-pong messages can effectively mitigate this problem [<a href="https://github.com/libp2p/specs/pull/558" target="_blank" rel="noopener noreferrer">13</a>].</p>
<p>The limitations faced with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> scaling also impact some performance optimizations in GossipSub.
For instance, floodpublishing is an optimization relying on additional transmissions by the publisher to minimize message dissemination latency.</p>
<p>However, a small <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> value in (new/cold) TCP connections established with floodpublish peers significantly increases message transmission time
[<a href="https://github.com/sigp/lighthouse/pull/4383" target="_blank" rel="noopener noreferrer">4</a>].
Usually, these peers also receive the same message from other sources during this time, wasting the publisher's bandwidth.</p>
<p>The same is the case with IWANT replies.</p>
<p>Maintaining a bigger mesh (with warm TCP connections) and relaying to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> peers can be a better alternative to this problem.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-eliminating-redundant-transmissions">3. Eliminating redundant transmissions<a href="https://vac.dev/rlog/gsub-largemsg-improvements#3-eliminating-redundant-transmissions" class="hash-link" aria-label="Direct link to 3. Eliminating redundant transmissions" title="Direct link to 3. Eliminating redundant transmissions">​</a></h3>
<p>For every received packet, a peer makes roughly <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> transmissions to contribute its fair share to the spread of messages.
However, the fact that many recipients had already received the message (from some other peer)
makes this message propagation inefficient.</p>
<p>Although the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>-spread is attributed to quicker dissemination and resilience against non-conforming peers,
many potential solutions can still minimize redundant transmissions
while preserving the resilience of GossipSub.</p>
<p>These solutions, ranging from probabilistic to more knowledgeful elimination of messages from the outgoing message queue,
not only address the issue of redundancy but also provide an opportunity for bandwidth optimization,
especially for resource-constrained peers.</p>
<p>For instance, an IDONTWANT message, a key component of GossipSub (v1.2) [<a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52" target="_blank" rel="noopener noreferrer">10</a>],
can significantly reduce redundant transmissions.</p>
<p>It allows any node to notify its mesh members that it has already received a message,
thereby preventing them from resending the same message.
This functionality is useful when a node receives a message larger than a specified threshold.</p>
<p>In such cases, the node promptly informs its mesh peers about the successful reception of the message by sending IDONTWANT messages.</p>
<p>It's important to note that an IDONTWANT message is essentially an IHAVE message, but with a crucial difference,
i.e., IHAVEs are only transmitted during the heartbeat intervals, whereas IDONTWANTs are sent immediately after receiving a large message.</p>
<p>This prompt notification helps curtail redundant large message transmissions without compromising the GossipSub resilience.</p>
<p>However, the use of IDONTWANT messages alone has an inherent limitation.
For instance, a peer can only send an IDONTWANT after receiving the complete message.</p>
<p>A large message transmission consumes significant time.
For example, transmitting a 1MB message at 100 Mbps bandwidth may consume 80 to several hundred milliseconds (depending upon <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and latency).</p>
<p>As a result, other mesh members may also start transmitting the same message during this interval.
A few potential solutions include:</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-staggering-with-idontwant-messages">a. Staggering with IDONTWANT messages<a href="https://vac.dev/rlog/gsub-largemsg-improvements#a-staggering-with-idontwant-messages" class="hash-link" aria-label="Direct link to a. Staggering with IDONTWANT messages" title="Direct link to a. Staggering with IDONTWANT messages">​</a></h4>
<p>As previously discussed, <a href="https://vac.dev/rlog/gsub-largemsg-improvements#b-message-staggering">staggering</a> can significantly reduce network-wide message dissemination latency.
This is primarily due to the relatively smaller store-and-forward delays that are inherent in this approach.</p>
<p>Using both staggering and IDONTWANT messages can further enhance efficiency by reducing redundant transmissions.
This is because a node only saturates its bandwidth for a small subset of mesh peers,
leading to early transmissions and prompt IDONTWANT message notifications to the mesh members.</p>
<p>It is worth highlighting that staggering can be implemented in various ways.</p>
<p>For example, it can be applied to peers (peer staggering)
where a node sequentially relays the same message to all peers one by one.</p>
<p>Alternatively, a node can send a different message to every peer (message staggering or rotational sending),
allowing IDONTWANTs for other messages to arrive during this time.
The message staggering approach is beneficial when several messages are introduced to the network within a short interval of time.</p>
<p>As the peers in staggered sending are sequentially covered
(with a faster speed due to bandwidth concentration), this leads to another problem.</p>
<p>The early covered peers send IHAVE (during their heartbeat intervals) for the messages they have received.
IHAVE announcements for newly received large messages trigger IWANTs from nodes
(including those already receiving the same message),
leading to an additional workload for early receivers [<a href="https://github.com/vacp2p/nim-libp2p/issues/1101" target="_blank" rel="noopener noreferrer">14</a>].</p>
<p>Potential solutions to mitigate these problems include:</p>
<ol>
<li>Defering IHAVE announcements for large messages.</li>
</ol>
<p>Deferring IHAVE announcements can indirectly prioritize message transmission to the mesh peers over IWANT replies.
However, deciding on a suitable deferred interval is crucial for optimal performance.
One possible solution is to generate IHAVEs only after the message is relayed to all the mesh peers.</p>
<ol start="2">
<li>Defering IWANT requests for messages that are currently being received.</li>
</ol>
<p>This requires <a href="https://vac.dev/rlog/gsub-largemsg-improvements#b-imreceiving-message">prior knowledge of msgIDs</a> for the messages under reception.
Knowing the message length is also essential in deciding a suitable defer interval
to handle situations where a sender starts sending a message and never completes the transmission.</p>
<ol start="3">
<li>Not issuing IWANT for a message if at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> peers have transmitted IDONTWANT for the same message
(as this indicates that these peers will eventually relay this message).</li>
</ol>
<p>However, this approach can inadvertently empower a group of non-conforming mesh peers to send IDONTWANT for a message and never complete message transmission.
A delayed IWANT, along with negative peer scoring, can remedy this problem.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-imreceiving-message">b. IMReceiving message<a href="https://vac.dev/rlog/gsub-largemsg-improvements#b-imreceiving-message" class="hash-link" aria-label="Direct link to b. IMReceiving message" title="Direct link to b. IMReceiving message">​</a></h4>
<p>A peer can issue an IDONTWANT only after it has received the entire message.
However, a large message transmission may take several hundred milliseconds to complete.
During this time, many other mesh members may start relaying the same message.</p>
<p>Therefore, the probability of simultaneously receiving the same message from multiple senders increases with the message size,
significantly compromising the effectiveness of IDONTWANT messages.</p>
<p>Sending a short preamble (containing msgID and length) before the message transmission can provide valuable information about the message.
If a receiver is already receiving the same message from another sender,
the receiver can request to defer this transmission by sending a brief IMReceiving message.</p>
<p>An IDONTWANT from the receiver will indicate successful message reception. Otherwise, the waiting sender can initiate transmission after a specific wait interval.</p>
<p>However, waiting for IMReceiving after sending the preamble can delay the message transmission.
On the other hand, proceeding with message transfer (after sending the preamble) leads to another problem:
it is difficult to cancel ongoing message transmission after receiving IMReceiving for the same message.</p>
<p>To streamline this process, a peer can immediately send an IMReceiving message (for every received preamble),
urging other mesh peers to defer sending the same message [<a href="https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281" target="_blank" rel="noopener noreferrer">15</a>,
<a href="https://forum.vac.dev/t/idontwant-message-impact/283" target="_blank" rel="noopener noreferrer">16</a>].</p>
<p>The other peers can send this message if IDONTWANT is not received from the receiver during the wait interval.
This approach can boost IDONTWANT benefits by considering ongoing transmissions for large messages.</p>
<p>While IMReceiving messages can bring about substantial improvements in terms of latency and bandwidth utilization,
it's crucial to be aware of the potential risks.</p>
<p>A malicious user can exploit this approach to disrupt message transmission
either by never completing a message or by intentionally sending a message at an extremely slow rate to numerous peers.</p>
<p>This could ultimately result in network-wide slow message propagation.</p>
<p>However, carefully calibrating the deferring interval (based on message size) and negative peer scoring can help mitigate these risks.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="c-idontwant-message-with-reduced-forwarding">c. IDONTWANT message with reduced forwarding<a href="https://vac.dev/rlog/gsub-largemsg-improvements#c-idontwant-message-with-reduced-forwarding" class="hash-link" aria-label="Direct link to c. IDONTWANT message with reduced forwarding" title="Direct link to c. IDONTWANT message with reduced forwarding">​</a></h4>
<p>It is common for slow peers to pile up outgoing message queues,
especially for large message transfers.
This results in a significant queuing delay for outgoing messages.
Reduced message forwarding can help decrease the workload of slower peers.</p>
<p>On receiving a message longer than the specified threshold,
a slow peer can relay it to only <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>∈</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">K \in D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> peers and send an IDONTWANT message to all the peers in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.</p>
<p>In this arrangement, the IDONTWANT message serves an additional purpose:
to promptly announce data availability, reinforcing redundancy in the presence of adversaries.</p>
<p>When a peer receives an IDONTWANT for an unseen message,
it learns about the new message and can request it by sending an IWANT request without waiting for the heartbeat (gossip) interval.
As a result, a significantly smaller number of transmissions is sufficient for propagating the message to the entire network.</p>
<p>This approach conserves peer bandwidth by minimizing redundant transmissions
while ensuring GossipSub resilience at the cost of one RTT (for missing peers).</p>
<p>Interestingly, curtailing queuing delays can also help lower network-wide message dissemination latency (for huge messages).</p>
<p>However, finding an appropriate value for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> is crucial for optimal performance.
A smaller <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> saves peer bandwidth, while a larger <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> achieves quicker spread until outgoing message queues pile up.
Setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><msub><mi>D</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">K = D_{low}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> can be one option.</p>
<p>It is worth mentioning that such behavior may negatively impact peer scoring (by missing message delivery rewards from <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo>−</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">D-K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> peers).
However, a minimized workload enables early message dissemination to the remaining peers.
These early transmissions and randomized <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> set selection can help achieve an overall better peer score.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-message-prioritization">4. Message prioritization<a href="https://vac.dev/rlog/gsub-largemsg-improvements#4-message-prioritization" class="hash-link" aria-label="Direct link to 4. Message prioritization" title="Direct link to 4. Message prioritization">​</a></h3>
<p>Despite the standardized specifications of the GossipSub protocol,
the message forwarding mechanisms can significantly impact network-wide message dissemination latency and bandwidth utilization.</p>
<p>It is worth mentioning that every node is responsible for transmitting different types of packets,
including control messages, locally published messages, messages received from mesh members, IWANT replies, etc.</p>
<p>As long as traffic volume is lower than the available data rate,
the message forwarding mechanisms yield similar results due to negligible queuing delays.</p>
<p>However, when the traffic volume increases and exceeds the available peer bandwidth (even for short traffic bursts),
the outgoing message queue(s) sizes rise, potentially impacting the network's performance.</p>
<p>In this scenario, FIFO-based traffic forwarding can lead to locally published messages being placed at the end of the outgoing message queue,
introducing a queuing delay proportional to the queue size.
The same applies to other delay-sensitive messages like IDONTWANT, PRUNE, etc.</p>
<p>On the other hand, the segregation of traffic into priority and non-priority queues can potentially starve low-priority messages.
One possible solution is to use weighted queues for a fair spread of messages.</p>
<p>Message prioritization can be a powerful tool to ensure that important messages reach their intended recipients on time
and allow for customizable message handling.</p>
<p>For example, staggering between peers and messages can be better managed by using priority queues.
However, it is important to note that message prioritization also introduces additional complexity to the system,
necessitating sophisticated algorithms for better message handling.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-maximizing-benefits-from-iwant-messages">5. Maximizing benefits from IWANT messages<a href="https://vac.dev/rlog/gsub-largemsg-improvements#5-maximizing-benefits-from-iwant-messages" class="hash-link" aria-label="Direct link to 5. Maximizing benefits from IWANT messages" title="Direct link to 5. Maximizing benefits from IWANT messages">​</a></h3>
<p>During heartbeat intervals, GossipSub nodes transmit IHAVE messages (carrying IDs of seen messages) to the peers not included in the full-message mesh.
These peers can use IWANT messages to request any missing messages.
A budget counter ensures these messages never exceed a specified threshold during each heartbeat interval.</p>
<p>The IHAVE/IWANT messages are a crucial tool in maintaining network connectivity.
They bridge the information gap between nearby and far-off peers,
ensuring that information can be disseminated to peers outside the mesh.
This function is essential in protecting against network partitions and indirectly aids in safeguarding against Sybil and eclipse attacks.</p>
<p>However, it is essential to understand that high transmission times for large messages
require careful due diligence when using IWANT messages for reasons not limited to:</p>
<ol>
<li>
<p>A large message reception may take several hundred milliseconds to complete.
During this time, an IHAVE message announcing the same message ID will trigger an IWANT request.</p>
</li>
<li>
<p>A peer can send IWANT requests for the same message to multiple nodes,
leading to simultaneous transmissions of the same message.</p>
</li>
<li>
<p>Replying to (potentially many) IWANT requests can delay the transmission of the same message to mesh peers,
resulting in lower peer scores and slower message propagation.</p>
</li>
</ol>
<p>A few possible solutions to mitigate this problem may include:</p>
<ol>
<li>
<p>Issuing IHAVE announcements only after the message is delivered to many mesh peers.</p>
</li>
<li>
<p>Allocating a volume-based budget to service IWANT requests during each heartbeat interval.</p>
</li>
<li>
<p>Deferring IWANT requests for messages that are currently being received.</p>
</li>
<li>
<p>Deferring IWANT requests if at least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> IDONTWANTs are received for the same message.</p>
</li>
<li>
<p>A large message transmission can yield high <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>; preferring such peers during mesh maintenance can be helpful.</p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary">Summary<a href="https://vac.dev/rlog/gsub-largemsg-improvements#summary" class="hash-link" aria-label="Direct link to Summary" title="Direct link to Summary">​</a></h2>
<p>This study investigates the pressing issue of considerable fluctuations and rises in network-wide dissemination times for large messages.</p>
<p>We delve into multiple factors,
such as increased message transmit times, store-and-forward delays, congestion avoidance mechanisms, and prioritization between messages,
to establish a comprehensive understanding of the problem.</p>
<p>The study also explores the performance of optimization efforts
like floodpublishing, IHAVE/IWANT messages, and message forwarding strategies in the wake of large message transmissions.</p>
<p>A key finding is that most congestion avoidance algorithms lack optimization for peer-to-peer networks.
Coupling this constraint with increased message transmission times
results in notable store-and-forward delays accumulating at each hop.</p>
<p>Furthermore, the probabilistic message-forwarding nature of GossipSub further exacerbates the situation
by utilizing a considerable share of available bandwidth on redundant transmissions.</p>
<p>Therefore, approaches focused on eliminating redundant transmissions
(IDONTWANT, IMReceiving, lazy sending, etc.) can prove helpful.
At the same time, strategies aimed at reducing store-and-forward delays
(fragmentation, staggering, prioritization, etc.) can prove beneficial.</p>
<p>It is worth mentioning that many of the strategies suggested in this post are ideas at different stages.
Some of these have already been explored and discussed to some extent [<a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw" target="_blank" rel="noopener noreferrer">5</a>,
<a href="https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366" target="_blank" rel="noopener noreferrer">17</a>,
<a href="https://vac.dev/rlog/gsub-idontwant-perf-eval/" target="_blank" rel="noopener noreferrer">18</a>].
We are nearing the completion of a comprehensive performance evaluation of these approaches and will soon share the results of our findings.</p>
<p>Please feel free to join the discussion and leave feedback regarding this post in the
<a href="https://forum.vac.dev/t/large-message-handling-in-gossipsub-potential-improvements/375" target="_blank" rel="noopener noreferrer">VAC forum</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/gsub-largemsg-improvements#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<p>[1] EIP-4844: Shard Blob Transactions. Retrieved from <a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">https://eips.ethereum.org/EIPS/eip-4844</a></p>
<p>[2] Message Propagation Times With Waku-RLN. Retrieved from <a href="https://docs.waku.org/research/research-and-studies/message-propagation/" target="_blank" rel="noopener noreferrer">https://docs.waku.org/research/research-and-studies/message-propagation/</a></p>
<p>[3] Lenient Flood Publishing. Retrieved from <a href="https://github.com/libp2p/rust-libp2p/pull/3666" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/rust-libp2p/pull/3666</a></p>
<p>[4] Disable Flood Publishing. Retrieved from <a href="https://github.com/sigp/lighthouse/pull/4383" target="_blank" rel="noopener noreferrer">https://github.com/sigp/lighthouse/pull/4383</a></p>
<p>[5] GossipSub for Big Messages. Retrieved from <a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw" target="_blank" rel="noopener noreferrer">https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw</a></p>
<p>[6] GossipSub: Lazy Sending. Retrieved from <a href="https://github.com/status-im/nim-libp2p/issues/850" target="_blank" rel="noopener noreferrer">https://github.com/status-im/nim-libp2p/issues/850</a></p>
<p>[7] GossipSub: Limit Flood Publishing. Retrieved from <a href="https://github.com/vacp2p/nim-libp2p/pull/911" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/nim-libp2p/pull/911</a></p>
<p>[8] GossipSub: Lazy Prefix Detection. Retrieved from <a href="https://github.com/vacp2p/nim-libp2p/issues/859" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/nim-libp2p/issues/859</a></p>
<p>[9] Potential Gossip Improvement List for EIP4844. Retrieved from <a href="https://hackmd.io/@gRwfloEASH6NWWS_KJxFGQ/B18wdnNDh" target="_blank" rel="noopener noreferrer">https://hackmd.io/@gRwfloEASH6NWWS_KJxFGQ/B18wdnNDh</a></p>
<p>[10] GossipSub Specifications v1.2: IDONTWANT Message. Retrieved from <a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52</a></p>
<p>[11] Number of Duplicate Messages in Ethereum’s GossipSub Network. Retrieved from <a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921" target="_blank" rel="noopener noreferrer">https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921</a></p>
<p>[12] TCP Congestion Control: Re-starting Idle Connections. Retrieved from <a href="https://datatracker.ietf.org/doc/html/rfc2581#section-4.1" target="_blank" rel="noopener noreferrer">https://datatracker.ietf.org/doc/html/rfc2581#section-4.1</a></p>
<p>[13] PING/PONG Control Messages. Retrieved from <a href="https://github.com/libp2p/specs/pull/558" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/pull/558</a></p>
<p>[14] IHAVE/IWANT Message Impact. Retrieved from <a href="https://github.com/vacp2p/nim-libp2p/issues/1101" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/nim-libp2p/issues/1101</a></p>
<p>[15] Large Message Handling IDONTWANT + IMReceiving Messages. Retrieved from <a href="https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281</a></p>
<p>[16] IDONTWANT Message Impact. Retrieved from <a href="https://forum.vac.dev/t/idontwant-message-impact/283" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/idontwant-message-impact/283</a></p>
<p>[17] IWANT Message Impact. Retrieved from <a href="https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366" target="_blank" rel="noopener noreferrer">https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366</a></p>
<p>[18] IDONTWANT Message Performance. Retrieved from <a href="https://vac.dev/rlog/gsub-idontwant-perf-eval/" target="_blank" rel="noopener noreferrer">https://vac.dev/rlog/gsub-idontwant-perf-eval/</a></p>]]></content>
<author>
<name>Umar Farooq</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Libp2p GossipSub IDONTWANT Message Performance Impact]]></title>
<id>https://vac.dev/rlog/gsub-idontwant-perf-eval</id>
<link href="https://vac.dev/rlog/gsub-idontwant-perf-eval"/>
<updated>2024-10-28T12:00:00.000Z</updated>
<summary type="html"><![CDATA[This post provides quick insights into the IDONTWANT message performance and highlights minor tweaks that can further contribute to performance gains.]]></summary>
<content type="html"><![CDATA[<p>This post provides quick insights into the IDONTWANT message performance and highlights minor tweaks that can further contribute to performance gains.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="overview">Overview<a href="https://vac.dev/rlog/gsub-idontwant-perf-eval#overview" class="hash-link" aria-label="Direct link to Overview" title="Direct link to Overview">​</a></h2>
<p><a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52" target="_blank" rel="noopener noreferrer">IDONTWANT</a> messages are introduced to curtail redundant transmissions without compromising resilience.
Cutting down on duplicates can potentially render two significant advantages:</p>
<ol>
<li>
<p>Reducing bandwidth requirements</p>
</li>
<li>
<p>Reducing message dissemination time (latency)</p>
</li>
</ol>
<p>For IDONTWANTs to be effective, they must be received and processed by the sender before the sender starts relaying the respective message.</p>
<p><a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921#arrival-time-of-duplicates-9" target="_blank" rel="noopener noreferrer">Duplicates investigation</a> reveals that
the average time difference between the first message arrival and the first duplicate arrival is higher than the average round trip time in Ethereum's GossipSub network.</p>
<p>This allows for timely IDONTWANT reception and canceling of many duplicate transmissions,
showing a potential for a significant drop in bandwidth utilization.
On the other hand, lowering message dissemination time is only possible by minimizing queuing delays at busy peers.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="experiments">Experiments<a href="https://vac.dev/rlog/gsub-idontwant-perf-eval#experiments" class="hash-link" aria-label="Direct link to Experiments" title="Direct link to Experiments">​</a></h2>
<p>We conducted a series of experiments with different arrangements (changing heartbeat_interval and message size)
to precisely identify the impact of IDONTWANT messages on bandwidth utilization and message dissemination time.</p>
<p>The experiments are performed on nim-libp2p using the <a href="https://github.com/vacp2p/dst-gossipsub-test-node/pull/4" target="_blank" rel="noopener noreferrer">shadow simulator</a>.
The peer bandwidth and link latency are uniformly set between 50-150 Mbps and 40-130 milliseconds in five stages.</p>
<p>In all experiments, ten messages are transmitted to the network, i.e.,
ten peers (publishers) are selected as the message transmitters.
Every publisher transmits exactly one message,
and inter-packet spacing (delay) is set to four seconds for each published message.
For a fair assessment, we ensure that the publishers are uniformly selected from each bandwidth class.</p>
<p>At the start of each experiment, two additional messages are transmitted to increase the TCP <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>C</mi><mrow><mi>w</mi><mi>n</mi><mi>d</mi></mrow></msub></mrow><annotation encoding="application/x-tex">C_{wnd}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span><span class="mord mathnormal mtight">n</span><span class="mord mathnormal mtight">d</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>.
These messages are not included in latency computations.</p>
<p>The simulation details are presented in the table below.</p>
<table><thead><tr><th><strong>Parameter</strong></th><th><strong>Value</strong></th><th><strong>Parameter</strong></th><th><strong>Value</strong></th></tr></thead><tbody><tr><td>Peers</td><td>2000</td><td>Publishers</td><td>10</td></tr><tr><td>Peer bandwidth</td><td>50-150 Mbps</td><td>Link latency</td><td>40-130 ms</td></tr><tr><td>Message size</td><td>1KB, 50KB, 500KB, 1MB</td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span></td><td>8</td></tr><tr><td>Heartbeat interval</td><td>700ms, 1000ms, 1500ms</td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{low}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.01968em">l</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></td><td>6</td></tr><tr><td>FloodPublish</td><td>False</td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{high}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">hi</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></td><td>12</td></tr><tr><td>Gossip factor</td><td>0.05</td><td>Muxer</td><td>yamux</td></tr></tbody></table>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="findings">Findings<a href="https://vac.dev/rlog/gsub-idontwant-perf-eval#findings" class="hash-link" aria-label="Direct link to Findings" title="Direct link to Findings">​</a></h2>
<p>We use bandwidth utilization and latency as evaluation metrics.
Bandwidth utilization represents total network-wide traffic (including gossip and other control messages).
Latency refers to network-wide message dissemination time.
The total number of IWANT requests and the number of message transmissions saved by IDONTWANT messages are also presented for detailed insights.</p>
<p>Experiments reveal that IDONTWANT messages yield a noticeable (up to 21%) drop in bandwidth utilization.
A higher drop is seen with a higher heartbeat interval.
Interestingly, a relatively low bandwidth reduction (12-20%) is seen for 1MB messages,
compared to 500KB messages (18-21%).</p>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/BW_700ms-54baea410c768c9ccbe8313c7ab3f992.png" width="1982" height="1157" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/BW_1000ms-340307cdf866c54fd52becb4df316fdf.png" width="1982" height="1157" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/BW_1500ms-d3b9c0f60549d0c6fabe47b548024f41.png" width="1982" height="1157" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<p>This is because downloading a large message may consume several hundred milliseconds.
During this time, a receiver will likely
<a href="https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366" target="_blank" rel="noopener noreferrer">generate multiple IWANT requests</a>
for the same message, increasing bandwidth utilization.</p>
<p>Moreover, a peer can generate
<a href="https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281" target="_blank" rel="noopener noreferrer">IDONTWANTs only after it has finished downloading the message</a>.
A longer download time will result in simultaneous reception of the same message from other mesh members.</p>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/IWANT_Requests-a19c04fc0a361e98075caa8e7cb1885a.png" width="1982" height="1157" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/IDONTWANT_Saves-463b248e2a1ee7995919cec733576159.png" width="1982" height="1157" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<p>These IWANT requests mainly overwhelm early message receivers,
which can negatively impact message dissemination time on some occasions.
Therefore, a similar message dissemination time is seen with and without IDONTWANT messages.</p>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/Lat_700ms-8fc202f87796b38baae0b623fcea4b57.png" width="2052" height="1155" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/Lat_1000ms-6a2af695a929c61c40d169a7d390606d.png" width="2052" height="1155" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/Lat_1500ms-42ca1f7a5f110002ed960be4fb811457.png" width="2052" height="1155" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<p>Similar results are seen on our large-scale deployment runs
(<a href="https://zealous-polka-dc7.notion.site/Nim-libp2p-v1-5-0-regression-testing-August-2024-25edba733c704ccaa411919555c5db1a" target="_blank" rel="noopener noreferrer">running Waku nodes in Kubernetes</a>).</p>
<p>Please feel free to join the discussion and leave feedback regarding this post in the
<a href="https://forum.vac.dev/t/libp2p-gossipsub-idontwant-message-performance-impact/374" target="_blank" rel="noopener noreferrer">VAC forum</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/gsub-idontwant-perf-eval#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li><a href="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md" target="_blank" rel="noopener noreferrer">GossipSub Specifications v1.2</a></li>
<li><a href="https://github.com/libp2p/specs/pull/548" target="_blank" rel="noopener noreferrer">GossipSub v1.2: IDONTWANT Control Message</a></li>
<li><a href="https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921" target="_blank" rel="noopener noreferrer">Number Duplicate Messages in Ethereum’s Gossipsub Network</a></li>
<li><a href="https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366" target="_blank" rel="noopener noreferrer">IWANT Messages Impact on Latency </a></li>
<li><a href="https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281" target="_blank" rel="noopener noreferrer">Large Message Handling (IDONTWANT + IMReceiving)</a></li>
<li><a href="https://forum.vac.dev/t/idontwant-message-impact/283" target="_blank" rel="noopener noreferrer">IDONTWANT Message Impact Before/After Message Validation</a></li>
<li><a href="https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw#2" target="_blank" rel="noopener noreferrer">GossipSub for Big Messages</a></li>
<li><a href="https://zealous-polka-dc7.notion.site/Nim-libp2p-v1-5-0-regression-testing-August-2024-25edba733c704ccaa411919555c5db1a" target="_blank" rel="noopener noreferrer">Regression Test Results: nim-libp2p</a></li>
</ul>]]></content>
<author>
<name>Umar Farooq</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Vac 101: Transforming an Interactive Protocol to a Noninteractive Argument]]></title>
<id>https://vac.dev/rlog/vac101-fiat-shamir</id>
<link href="https://vac.dev/rlog/vac101-fiat-shamir"/>
<updated>2024-10-15T12:00:00.000Z</updated>
<summary type="html"><![CDATA[In this post, we introduce a common technique used to convert interactive protocols to their noninteractive variant.]]></summary>
<content type="html"><![CDATA[<p>In this post, we introduce a common technique used to convert interactive protocols to their noninteractive variant.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/vac101-fiat-shamir#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>The set of interactive protocols form a class of protocols that consist of communication between two parties: the Prover and the Verifier.
The Prover tries to convince the Verifier of a given claim.
For example, the Prover may want to convince the Verifier that she owns a specific Unspent Transaction Output (UTXO);
that is, the Prover possesses the ability to spend the UTXO.
In many instances, there is information that the Prover does not wish to reveal to the Verifier.
In our example, it is critical that the Prover does not provide the Verifier with the spending key associated with her UTXO.
In addition to the Prover's claim and secret data, there is additional data, public parameters, that the claimed statement is expressed in terms of.
The public parameters can be thought of as the basis for all similar claims.</p>
<p>In an interactive protocol, the Prover and the Verifier are in active communication.
Specifically, the Prover and the Verifier exchange messages so that the Verifier can validate the Prover's claim.
However, this communication is not practical for many applications.
It is necessary that any party can verify the Prover's claim in decentralized systems.
It is impractical for the Prover to be in active communication with a large number of verifying parties.
Instead, it is desirable for the Prover to generate a proof on their own that can convince any party.
To achieve this, it is necessary for the Prover to generate the Verifier's messages in such a way
that the Prover cannot manipulate the Verifier's messages for her benefit.
The Fiat-Shamir heuristic <a href="https://dl.acm.org/doi/10.5555/36664.36676" target="_blank" rel="noopener noreferrer">1</a> is used for this purpose.
Even though much of our discussion will focus on <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Σ</span></span></span></span>-protocols,
the Fiat-Shamir heuristic is not limited to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Σ</span></span></span></span>-protocols.
The Fiat-Shamir heuristic has been applied to zk-SNARKs, but the security in this setting has been the subject of discussion and research in recent years.
Block et al. <a href="https://eprint.iacr.org/2023/1071" target="_blank" rel="noopener noreferrer">2</a> provide the first formal analysis of Fiat-Shamir heuristic in zk-SNARKs.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="sigma-protocols">Sigma Protocols<a href="https://vac.dev/rlog/vac101-fiat-shamir#sigma-protocols" class="hash-link" aria-label="Direct link to Sigma Protocols" title="Direct link to Sigma Protocols">​</a></h2>
<p>A <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Σ</span></span></span></span>-protocol is a family of interactive protocols that consists of three publicly transmitted messages between the Prover and the Verifier.
In particular, the protocol has the following framework:</p>
<table><thead><tr><th>Prover</th><th></th><th>Verifier</th></tr></thead><tbody><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mrow><mi mathvariant="sans-serif">c</mi><mi mathvariant="sans-serif">o</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">i</mi><mi mathvariant="sans-serif">t</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">t</mi></mrow></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{\mathsf{commitment}}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2976em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.2866em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathsf mtight">commitment</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟵</mo></mo><mrow><mi mathvariant="sans-serif">c</mi><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">g</mi><mi mathvariant="sans-serif">e</mi></mrow></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{\mathsf{challenge}}{\longleftarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3552em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.3442em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟵</span></span></span><span style="top:-3.7581em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathsf mtight">challenge</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mrow><mi mathvariant="sans-serif">r</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">p</mi><mi mathvariant="sans-serif">o</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">e</mi></mrow></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{\mathsf{response}}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1802em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.1692em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.7581em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathsf mtight">response</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr></tbody></table>
<p>These three messages form the protocol's transcript: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mrow><mi mathvariant="sans-serif">c</mi><mi mathvariant="sans-serif">o</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">i</mi><mi mathvariant="sans-serif">t</mi><mi mathvariant="sans-serif">m</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">t</mi></mrow><mo separator="true">,</mo><mrow><mi mathvariant="sans-serif">c</mi><mi mathvariant="sans-serif">h</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">l</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">g</mi><mi mathvariant="sans-serif">e</mi></mrow><mo separator="true">,</mo><mrow><mi mathvariant="sans-serif">r</mi><mi mathvariant="sans-serif">e</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">p</mi><mi mathvariant="sans-serif">o</mi><mi mathvariant="sans-serif">n</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">e</mi></mrow><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(\mathsf{commitment}, \mathsf{challenge}, \mathsf{response})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathsf">commitment</span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathsf">challenge</span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathsf">response</span></span><span class="mclose">)</span></span></span></span>.
The Verifier uses all three of these messages to validate the Prover's original claim.
The Verifier's challenge should be selected uniform random from all possible challenges.
Based on this selection, a dishonest Prover can only convince the Verifier with a negligible probability.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-schnorr-protocol">The Schnorr Protocol<a href="https://vac.dev/rlog/vac101-fiat-shamir#the-schnorr-protocol" class="hash-link" aria-label="Direct link to The Schnorr Protocol" title="Direct link to The Schnorr Protocol">​</a></h3>
<p>The Schnorr protocol <a href="https://link.springer.com/chapter/10.1007/0-387-34805-0_22" target="_blank" rel="noopener noreferrer">3</a> is usually the first <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Σ</span></span></span></span>-protocol that one studies.
Additionally, the Schnorr protocol can be used as an efficient signature scheme.
The Schnorr protocol provides a framework that enables the Prover to convince the Verifier that: for group elements <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span>,
the Prover knows the power to raise <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> to obtain <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span>.
Specifically, the Prover possesses some integer <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi><mo>=</mo><msup><mi>g</mi><mi>x</mi></msup></mrow><annotation encoding="application/x-tex">X = g^x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span></span></span></span></span></span></span></span>.
Cryptographic resources may use either multiplicative or additive notation for groups;
we will use multiplicative notation.
Briefly, the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> being combined with itself in multiplicative notation is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>⋅</mo><mi>g</mi><mo>=</mo><msup><mi>g</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">g \cdot g = g^2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0085em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span></span></span></span></span></span></span></span>,
while in additive notation it is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>+</mo><mi>g</mi><mo>=</mo><mn>2</mn><mi>g</mi></mrow><annotation encoding="application/x-tex">g + g = 2g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em"></span><span class="mord">2</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>.
We assume that our group is of prime order <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">p</span></span></span></span>, and is sufficiently large to satisfy the discrete logarithm assumption.</p>
<p>The Schnorr protocol proceeds as follows:</p>
<table><thead><tr><th>Prover</th><th></th><th>Verifier</th></tr></thead><tbody><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">t \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>:</mo><mo>=</mo><msup><mi>g</mi><mi>t</mi></msup></mrow><annotation encoding="application/x-tex">T := g^t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.988em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.7936em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span></span></span></span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mi>T</mi></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{T}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3003em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.2893em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟵</mo></mo><mi>c</mi></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{c}{\longleftarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1234em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.1124em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟵</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">c \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi><mo>:</mo><mo>=</mo><mi>t</mi><mo>+</mo><mi>x</mi><mi>c</mi></mrow><annotation encoding="application/x-tex">z := t + xc</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6984em;vertical-align:-0.0833em"></span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span><span class="mord mathnormal">c</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mi>z</mi></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{z}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1234em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.1124em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td></td><td>output 1 provided <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mi>z</mi></msup><mo><mover><mo><mo>=</mo></mo><mo stretchy="false" lspace="0em" rspace="0em">?</mo></mover></mo><mi>T</mi><msup><mi>X</mi><mi>c</mi></msup></mrow><annotation encoding="application/x-tex">g^z \stackrel{?}{=} T X^c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3474em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.153em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">=</span></span></span><span style="top:-3.5669em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mclose mtight">?</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span></span></span></span></td></tr></tbody></table>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="chaum-pedersen-protocol">Chaum-Pedersen protocol<a href="https://vac.dev/rlog/vac101-fiat-shamir#chaum-pedersen-protocol" class="hash-link" aria-label="Direct link to Chaum-Pedersen protocol" title="Direct link to Chaum-Pedersen protocol">​</a></h3>
<p>A tuple of group elements <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(U,V,W)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span></span></span></span> is a DH-triple if and only if there exists some <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>∈</mo><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">x \in \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em"></span><span class="mord mathnormal">x</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>=</mo><msup><mi>g</mi><mi>x</mi></msup></mrow><annotation encoding="application/x-tex">V = g^x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi><mo>=</mo><msup><mi>U</mi><mi>x</mi></msup></mrow><annotation encoding="application/x-tex">W = U^x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span></span></span></span></span></span></span></span>.
The Chaum-Pedersen protocol provides a framework that enables a Prover to convince a Verifier that she possesses such a <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> for a claimed DH-triple <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(U,V,W)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span></span></span></span>.
The Chaum-Pedersen protocol proceeds as follows:</p>
<table><thead><tr><th>Prover</th><th></th><th>Verifier</th></tr></thead><tbody><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">t \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7651em;vertical-align:-0.15em"></span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>:</mo><mo>=</mo><msup><mi>g</mi><mi>t</mi></msup></mrow><annotation encoding="application/x-tex">T := g^t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.988em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.7936em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mo>:</mo><mo>=</mo><msup><mi>U</mi><mi>t</mi></msup></mrow><annotation encoding="application/x-tex">S := U^t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7936em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.7936em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span></span></span></span></span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mrow><mi>T</mi><mo separator="true">,</mo><mi>S</mi></mrow></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{T,S}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3474em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.3364em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.7581em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em">T</span><span class="mpunct mtight">,</span><span class="mord mathnormal mtight" style="margin-right:0.05764em">S</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟵</mo></mo><mi>c</mi></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{c}{\longleftarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1234em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.1124em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟵</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">c \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi><mo>:</mo><mo>=</mo><mi>t</mi><mo>+</mo><mi>x</mi><mi>c</mi></mrow><annotation encoding="application/x-tex">z := t + xc</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6984em;vertical-align:-0.0833em"></span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span><span class="mord mathnormal">c</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo><mover><mo><mo>⟶</mo></mo><mi>z</mi></mover></mo></mrow><annotation encoding="application/x-tex">\stackrel{z}{\longrightarrow}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1234em;vertical-align:-0.011em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.1124em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">⟶</span></span></span><span style="top:-3.711em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.011em"><span></span></span></span></span></span></span></span></span></span></td><td></td></tr><tr><td></td><td></td><td>output 1 provided <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mi>z</mi></msup><mo><mover><mo><mo>=</mo></mo><mo stretchy="false" lspace="0em" rspace="0em">?</mo></mover></mo><mi>T</mi><msup><mi>V</mi><mi>c</mi></msup></mrow><annotation encoding="application/x-tex">g^z \stackrel{?}{=} T V^c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.3474em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.153em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">=</span></span></span><span style="top:-3.5669em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mclose mtight">?</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>U</mi><mi>z</mi></msup><mo><mover><mo><mo>=</mo></mo><mo stretchy="false" lspace="0em" rspace="0em">?</mo></mover></mo><mi>S</mi><msup><mi>W</mi><mi>c</mi></msup></mrow><annotation encoding="application/x-tex">U^z \stackrel{?}{=} SW^c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.153em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mop op-limits"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:1.153em"><span style="top:-3em"><span class="pstrut" style="height:3em"></span><span><span class="mop">=</span></span></span><span style="top:-3.5669em;margin-left:0em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mclose mtight">?</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span></span></span></span></td></tr></tbody></table>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="hash-functions">Hash Functions<a href="https://vac.dev/rlog/vac101-fiat-shamir#hash-functions" class="hash-link" aria-label="Direct link to Hash Functions" title="Direct link to Hash Functions">​</a></h2>
<p>Cryptographic hash functions serve as the backbone to the Fiat-Shamir heuristic.
A hash function, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span>, is a special function that takes in an arbitrary binary string and outputs a binary string of a predetermined fixed length.
Specifically,
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo>:</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mo>∗</mo></msup><mo>→</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">\mathsf{Hash} : \{0,1\}^* \rightarrow \{0,1\}^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6887em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mbin mtight">∗</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">→</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span>.</p>
<p>The security of cryptographic hash functions will rely on certain tasks being computationally infeasible.
A task is computationally infeasible provided that there is no deterministic algorithm that can conclude the task in polynomial-time.</p>
<p>A cryptographic hash function satisfies the following properties:</p>
<ul>
<li><strong>Succinct</strong>: The hash function should be easy to compute; the hash <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi mathvariant="bold">b</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\mathsf{Hash}({\bf{b}})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mclose">)</span></span></span></span> can be efficiently computed for any binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">b</mi></mrow><annotation encoding="application/x-tex">{\bf{b}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span></span></span></span>.</li>
<li><strong>Preimage Resistance</strong>: It should be computationally infeasible to work backwards given the output of a hash function. Let <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">y</mi></mrow><annotation encoding="application/x-tex">{\bf{y}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">y</span></span></span></span></span></span></span> be a binary string of length <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>.
It should be 'impossible' to find some binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">x</mi></mrow><annotation encoding="application/x-tex">{\bf{x}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4444em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span></span></span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">y</mi><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi mathvariant="bold">x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{\bf{y}} = \mathsf{Hash}({\bf{x}})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">y</span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span></span></span><span class="mclose">)</span></span></span></span>.</li>
<li><strong>Collision Resistance</strong>: It should be difficult to find two strings that hash to the same value.
It should be computationally infeasible to find two binary strings <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">x</mi><mn mathvariant="bold">1</mn></msub></mrow><annotation encoding="application/x-tex">{\bf{x}_1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5944em;vertical-align:-0.15em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathbf mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">x</mi><mn mathvariant="bold">2</mn></msub></mrow><annotation encoding="application/x-tex">{\bf{x}_2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5944em;vertical-align:-0.15em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathbf mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><msub><mi mathvariant="bold">x</mi><mn mathvariant="bold">1</mn></msub><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><msub><mi mathvariant="bold">x</mi><mn mathvariant="bold">2</mn></msub><mo stretchy="false">)</mo><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}({\bf{x}_1}) = \mathsf{Hash}({\bf{x}_2}).</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathbf mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">x</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathbf mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span><span class="mclose">)</span><span class="mord">.</span></span></span></span></li>
</ul>
<p>A related class of functions is one-way functions.
A one-way function satisfies the first two conditions of a cryptographic hash function (succinct and preimage resistance).
All cryptographic hash functions are a one-way functions.
However, one-way functions do not necessarily satisfy collision-resistance.
We will simply refer to cryptographic hash functions as hash functions for the rest of this blog.
Commonly used hash functions include SHA-256 <a href="https://www.cs.princeton.edu/~appel/papers/verif-sha.pdf" target="_blank" rel="noopener noreferrer">5</a>,
Keccak <a href="https://keccak.team/keccak_specs_summary.html" target="_blank" rel="noopener noreferrer">6</a>, and Poseidon <a href="https://eprint.iacr.org/2019/458" target="_blank" rel="noopener noreferrer">7</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-fiat-shamir-heuristic">The Fiat-Shamir heuristic<a href="https://vac.dev/rlog/vac101-fiat-shamir#the-fiat-shamir-heuristic" class="hash-link" aria-label="Direct link to The Fiat-Shamir heuristic" title="Direct link to The Fiat-Shamir heuristic">​</a></h2>
<p>The Fiat-Shamir heuristic is the technique used to convert an interactive protocol to a noninteractive protocol.
This is done by replacing each of the Verifier's messages with a hashed value.
Specifically, the Prover generates the Verifier's message by evaluating the hash function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span>
with the concatenation of all public values that appear in the protocol thus far.
If <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mn>0</mn></msub><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><msub><mi>m</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">m_0, \dots, m_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> denote the public values in the protocol thus far,
then the Verifier's message is computed as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>:</mo><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><msub><mi>m</mi><mn>0</mn></msub><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mo>⋯</mo><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><msub><mi>m</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">m_{t+1} := \mathsf{Hash}(m_0|| \cdots || m_t)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mord">∣∣</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">⋯</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">∣∣</span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>.</p>
<p>Since <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span> can be efficiently computed, and the messages <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mn>0</mn></msub><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><msub><mi>m</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">m_0, \dots, m_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> are public, then any verifying party can compute <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow><annotation encoding="application/x-tex">m_{t+1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span></span></span></span>.
Critically, since <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span> is preimage resistant and collision resistant,
the Prover cannot manipulate her choices of the messages <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mn>0</mn></msub><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><msub><mi>m</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">m_0,\dots, m_t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to influence the message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow><annotation encoding="application/x-tex">m_{t+1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span></span></span></span>.
Hence, verifying parties can trust that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>m</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow><annotation encoding="application/x-tex">m_{t+1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6389em;vertical-align:-0.2083em"></span><span class="mord"><span class="mord mathnormal">m</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mbin mtight">+</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2083em"><span></span></span></span></span></span></span></span></span></span> is sufficiently random with respect to the preceding messages.</p>
<p>There are two variants of the Fiat-Shamir heuristic: weak and strong.
The weak variant uses all of the publicly traded messages in computing the Verifier's messages but does not include the public parameters.
However, in the strong variant all of the publicly traded messages and public parameters are used to compute the Verifier's messages.
We will provide a discussion on issues that can arise from using the weak Fiat-Shamir heuristic.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="schnorr-protocol-with-the-strong-fiat-shamir">Schnorr Protocol with the strong Fiat-Shamir<a href="https://vac.dev/rlog/vac101-fiat-shamir#schnorr-protocol-with-the-strong-fiat-shamir" class="hash-link" aria-label="Direct link to Schnorr Protocol with the strong Fiat-Shamir" title="Direct link to Schnorr Protocol with the strong Fiat-Shamir">​</a></h3>
<p>When the strong Fiat-Shamir heuristic is applied to the Schnorr protocol, the message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi>g</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>X</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>T</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">c = \mathsf{Hash}(g||X||T)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mclose">)</span></span></span></span>.
This choice of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> provides security since it should be computationally infeasible to find collisions for the outputs of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span>.
Thus, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> fixes the group elements <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span></span></span></span>.</p>
<p>The elements that would be omitted in the hash by applying weak Fiat-Shamir heuristic are <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span>.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="chaum-pedersen-protocol-with-the-strong-fiat-shamir">Chaum-Pedersen Protocol with the strong Fiat-Shamir<a href="https://vac.dev/rlog/vac101-fiat-shamir#chaum-pedersen-protocol-with-the-strong-fiat-shamir" class="hash-link" aria-label="Direct link to Chaum-Pedersen Protocol with the strong Fiat-Shamir" title="Direct link to Chaum-Pedersen Protocol with the strong Fiat-Shamir">​</a></h3>
<p>The message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mi>H</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>g</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>U</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>V</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>W</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>T</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>S</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">c = Hash(g||U||V||W||T||S)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.08125em">H</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mclose">)</span></span></span></span> when the Prover applies the strong Fiat-Shamir heuristic to the Chaum-Pedersen protocol.
The properties of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><annotation encoding="application/x-tex">\mathsf{Hash}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord"><span class="mord mathsf">Hash</span></span></span></span></span> fixes the generator <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> and the Prover's statement <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(U,V,W)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span></span></span></span>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="improper-use-of-the-fiat-shamir-heuristic">Improper use of the Fiat-Shamir heuristic<a href="https://vac.dev/rlog/vac101-fiat-shamir#improper-use-of-the-fiat-shamir-heuristic" class="hash-link" aria-label="Direct link to Improper use of the Fiat-Shamir heuristic" title="Direct link to Improper use of the Fiat-Shamir heuristic">​</a></h2>
<p>The Fiat-Shamir heuristic appears to be a fairly straightforward technique to implement.
However, a subtle but serious issue that can occur in the application of the Fiat-Shamir heuristic has been a point of discussion for the past few years.
The issue concerns what messages are included in the hash.
In particular, are the public parameters used to compute the hash value?</p>
<p>Bernhard et al. <a href="https://eprint.iacr.org/2016/771.pdf" target="_blank" rel="noopener noreferrer">8</a> provide a discussion of Fiat-Shamir heuristic restricted to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Σ</mi></mrow><annotation encoding="application/x-tex">\Sigma</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord">Σ</span></span></span></span>-protocols.
In particular, Bernhard et al. discuss the pitfalls of the weak Fiat-Shamir heuristic.
Recall that the strong Fiat-Shamir heuristic requires that the public parameters are included in the calculations of the Verifier's messages while the weak version does not.
The inclusion of the public parameters in the hash evaluations fixes these public values for the entire protocol.
This means that the Prover cannot retroactively change them.</p>
<p>The issues with the differences in the variants of the Fiat-Shamir heuristics has persisted since Bernhard et al.'s paper.
In recent years, auditors from <a href="https://www.trailofbits.com/" target="_blank" rel="noopener noreferrer">Trail of Bits</a> and <a href="https://www.openzeppelin.com/" target="_blank" rel="noopener noreferrer">OpenZeppelin</a> have
released blogs (<a href="https://blog.trailofbits.com/2022/04/13/part-1-coordinated-disclosure-of-vulnerabilities-affecting-girault-bulletproofs-and-plonk/" target="_blank" rel="noopener noreferrer">9</a>,
<a href="https://blog.trailofbits.com/2022/04/14/the-frozen-heart-vulnerability-in-giraults-proof-of-knowledge/" target="_blank" rel="noopener noreferrer">10</a>,
<a href="https://blog.trailofbits.com/2022/04/15/the-frozen-heart-vulnerability-in-bulletproofs/" target="_blank" rel="noopener noreferrer">11</a>, <a href="https://blog.trailofbits.com/2022/04/18/the-frozen-heart-vulnerability-in-plonk/" target="_blank" rel="noopener noreferrer">12</a>, <a href="https://blog.openzeppelin.com/the-last-challenge-attack" target="_blank" rel="noopener noreferrer">13</a>)
and papers (<a href="https://eprint.iacr.org/2023/691" target="_blank" rel="noopener noreferrer">14</a>, <a href="https://eprint.iacr.org/2024/398" target="_blank" rel="noopener noreferrer">15</a>)
describing specific vulnerabilities in zero-knowledge papers and repositories associated with the use of the weak Fiat-Shamir heuristic.</p>
<p>Trail of Bits coined the term <strong>FROZEN Heart</strong> to describe the use of weak Fiat-Shamir heuristic.
Frozen comes from the phrase "FoRging Of ZEro kNowledge proofs",
and Fiat-Shamir is the "heart" of transforming an interactive protocol to noninteractive protocol.</p>
<p>Now, we examine how weak Fiat-Shamir affects the Schnorr protocol and Chaum-Pedersen protocol.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="schnorr-protocol-with-the-weak-fiat-shamir-heuristic">Schnorr protocol with the weak Fiat-Shamir heuristic<a href="https://vac.dev/rlog/vac101-fiat-shamir#schnorr-protocol-with-the-weak-fiat-shamir-heuristic" class="hash-link" aria-label="Direct link to Schnorr protocol with the weak Fiat-Shamir heuristic" title="Direct link to Schnorr protocol with the weak Fiat-Shamir heuristic">​</a></h3>
<p>For Schnorr, we will examine two variants:
the first where we only include the Prover's claim <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> but not the public parameter <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>, and
the second where we include the public parameter <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> but not the Prover's claim <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span>.</p>
<p>Since we omit the generator <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>∈</mo><mi mathvariant="double-struck">G</mi></mrow><annotation encoding="application/x-tex">g \in \mathbb{G}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7335em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6889em"></span><span class="mord mathbb">G</span></span></span></span> from the computation for the message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> in our first approach,
then <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi>X</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>T</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">c = \mathsf{Hash}(X||T)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mclose">)</span></span></span></span>.</p>
<p>Now, a malicious Prover can complete the transcript for the Schnorr protocol by selecting any <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">z \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.
Since, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> is not fixed as it was not included in the computation of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span>.
But, the malicious Prover needs the transcript <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>T</mi><mo separator="true">,</mo><mi>c</mi><mo separator="true">,</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(T,c,z)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">c</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mclose">)</span></span></span></span> to satisfy <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mi>z</mi></msup><mo>=</mo><mi>T</mi><msup><mi>X</mi><mi>c</mi></msup></mrow><annotation encoding="application/x-tex">g^z = TX^c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span></span></span></span>.
Hence, the malicious Prover can compute the generator <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mo stretchy="false">(</mo><mi>T</mi><msup><mi>X</mi><mi>c</mi></msup><msup><mo stretchy="false">)</mo><msup><mi>z</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup></msup><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">g = (TX^c)^{z^{-1}}.</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2369em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="mord">.</span></span></span></span></p>
<p>In our second approach, we omit the group element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi><mo>∈</mo><mi mathvariant="double-struck">G</mi></mrow><annotation encoding="application/x-tex">X \in \mathbb{G}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6889em"></span><span class="mord mathbb">G</span></span></span></span> from the computation for the challenge <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span>.
That is, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mrow><mi mathvariant="sans-serif">H</mi><mi mathvariant="sans-serif">a</mi><mi mathvariant="sans-serif">s</mi><mi mathvariant="sans-serif">h</mi></mrow><mo stretchy="false">(</mo><mi>g</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>T</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">c = \mathsf{Hash}(g||T)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathsf">Hash</span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord">∣∣</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mclose">)</span></span></span></span>.</p>
<p>As with the previous example, the malicious Prover takes a Schnorr transcript <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>T</mi><mo separator="true">,</mo><mi>c</mi><mo separator="true">,</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(T,c,z)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">c</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mclose">)</span></span></span></span> where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi><msub><mo>∈</mo><mi>R</mi></msub><msub><mi mathvariant="double-struck">Z</mi><mi>p</mi></msub></mrow><annotation encoding="application/x-tex">z \in_R \mathbb{Z}_p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6891em;vertical-align:-0.15em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel"><span class="mrel">∈</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.00773em">R</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.975em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathbb">Z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">p</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.
It is necessary for the malicious Prover to find a value <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>g</mi><mi>z</mi></msup><mo>=</mo><mi>T</mi><msup><mi>X</mi><mi>c</mi></msup></mrow><annotation encoding="application/x-tex">g^z = TX^c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8588em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span></span></span></span>.
This can be acheived by computing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi><mo>=</mo><mo stretchy="false">(</mo><msup><mi>g</mi><mi>z</mi></msup><msup><mi>T</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><msup><mo stretchy="false">)</mo><msup><mi>c</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup></msup></mrow><annotation encoding="application/x-tex">X = (g^z T^{-1})^{c^{-1}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.2369em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.9869em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight">c</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em"><span style="top:-2.931em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="chaum-pedersen-protocol-with-the-fiat-shamir-heuristic">Chaum-Pedersen protocol with the Fiat-Shamir heuristic<a href="https://vac.dev/rlog/vac101-fiat-shamir#chaum-pedersen-protocol-with-the-fiat-shamir-heuristic" class="hash-link" aria-label="Direct link to Chaum-Pedersen protocol with the Fiat-Shamir heuristic" title="Direct link to Chaum-Pedersen protocol with the Fiat-Shamir heuristic">​</a></h3>
<p>The Verifier's message <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mi>H</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>T</mi><mo separator="true">,</mo><mi>S</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">c = Hash(T,S)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.08125em">H</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mclose">)</span></span></span></span> when weak Fiat-Shamir heuristic is applied.
The Prover's triple <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(U,V,W)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span></span></span></span> and the generator <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> are not fixed by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span>.
As such, a malicious Prover can generate values for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi></mrow><annotation encoding="application/x-tex">U,V,W</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> that satisfy the Verifier's identity checks.
In the case of a malicious Prover, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">T</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi></mrow><annotation encoding="application/x-tex">S</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.05764em">S</span></span></span></span> are randomly group elements instead of being computed using a value <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span> that the Prover selected.
This means a malicious Prover must randomly select the value <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi></mrow><annotation encoding="application/x-tex">z</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal" style="margin-right:0.04398em">z</span></span></span></span> as well.</p>
<p>Given the values that have been fixed so far, each of the Verifier's identities consists of two unknowns.
Hence, it is necessary to select one of these unknowns from each identity so that a malicious Prover can compute the last value.
For instances, suppose that the malicious Prover randomly selects <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi></mrow><annotation encoding="application/x-tex">V</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>W</mi></mrow><annotation encoding="application/x-tex">W</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span></span></span></span>.
The malicious Prover can compute <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mo>=</mo><mo stretchy="false">(</mo><mi>T</mi><msup><mi>V</mi><mi>c</mi></msup><msup><mo stretchy="false">)</mo><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>z</mi></mrow></msup></mrow><annotation encoding="application/x-tex">g = (T V^c)^{1/z}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.13889em">T</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1/</span><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mo>=</mo><mo stretchy="false">(</mo><mi>S</mi><msup><mi>W</mi><mi>c</mi></msup><msup><mo stretchy="false">)</mo><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>z</mi></mrow></msup></mrow><annotation encoding="application/x-tex">V = (SW^c)^{1/z}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">c</span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">1/</span><span class="mord mathnormal mtight" style="margin-right:0.04398em">z</span></span></span></span></span></span></span></span></span></span></span></span>.
Thus, the malicious Prover has a claimed statement <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(U,V,W)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mclose">)</span></span></span></span> for generator <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> that passes the Verifier's identities using weak Fiat-Shamir heuristic.</p>
<p>The omission of any of the values <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>U</mi><mo separator="true">,</mo><mi>V</mi><mo separator="true">,</mo><mi>W</mi><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">U,V,W,</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.22222em">V</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.13889em">W</span><span class="mpunct">,</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> in the computation of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">c</span></span></span></span> allows a malicious Prover to forge a proof.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/vac101-fiat-shamir#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>The Fiat-Shamir heuristic is an essential technique to convert an interactive protocol to a variant that does not require communication.
Additionally, careful application of this technique is necessary to maintain the integrity of the system.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/vac101-fiat-shamir#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h3>
<ul>
<li>
<ol>
<li><a href="https://dl.acm.org/doi/10.5555/36664.36676" target="_blank" rel="noopener noreferrer">How to Prove Yourself: Practical Solutions to Identification and Signature Problems</a></li>
</ol>
</li>
<li>
<ol start="2">
<li><a href="https://eprint.iacr.org/2023/1071" target="_blank" rel="noopener noreferrer">Fiat-Shamir Security of FRI and Related SNARKs</a></li>
</ol>
</li>
<li>
<ol start="3">
<li><a href="https://link.springer.com/chapter/10.1007/0-387-34805-0_22" target="_blank" rel="noopener noreferrer">Efficient Identification and Signatures for Smart Cards</a></li>
</ol>
</li>
<li>
<ol start="4">
<li><a href="https://link.springer.com/content/pdf/10.1007/3-540-48071-4_7.pdf" target="_blank" rel="noopener noreferrer">Wallet Databases with Observers</a></li>
</ol>
</li>
<li>
<ol start="5">
<li><a href="https://www.cs.princeton.edu/~appel/papers/verif-sha.pdf" target="_blank" rel="noopener noreferrer">Verification of a Cryptographic Primitive: SHA-256</a></li>
</ol>
</li>
<li>
<ol start="6">
<li><a href="https://keccak.team/keccak_specs_summary.html" target="_blank" rel="noopener noreferrer">Keccak specifications summary</a></li>
</ol>
</li>
<li>
<ol start="7">
<li><a href="https://eprint.iacr.org/2019/458" target="_blank" rel="noopener noreferrer">Poseidon: A New Hash Function for Zero-Knowledge Proof Systems</a></li>
</ol>
</li>
<li>
<ol start="8">
<li><a href="https://eprint.iacr.org/2016/771.pdf" target="_blank" rel="noopener noreferrer">How not to Prove Yourself: Pitfalls of the Fiat-Shamir Heuristic and Applications to Helios</a></li>
</ol>
</li>
<li>
<ol start="9">
<li><a href="https://blog.trailofbits.com/2022/04/13/part-1-coordinated-disclosure-of-vulnerabilities-affecting-girault-bulletproofs-and-plonk/" target="_blank" rel="noopener noreferrer">Frozen Heart - Part 1</a></li>
</ol>
</li>
<li>
<ol start="10">
<li><a href="https://blog.trailofbits.com/2022/04/14/the-frozen-heart-vulnerability-in-giraults-proof-of-knowledge/" target="_blank" rel="noopener noreferrer">Frozen Heart - Part 2</a></li>
</ol>
</li>
<li>
<ol start="11">
<li><a href="https://blog.trailofbits.com/2022/04/15/the-frozen-heart-vulnerability-in-bulletproofs/" target="_blank" rel="noopener noreferrer">Frozen Heart - Part 3</a></li>
</ol>
</li>
<li>
<ol start="12">
<li><a href="https://blog.trailofbits.com/2022/04/18/the-frozen-heart-vulnerability-in-plonk/" target="_blank" rel="noopener noreferrer">Frozen Heart - Part 4</a></li>
</ol>
</li>
<li>
<ol start="13">
<li><a href="https://blog.openzeppelin.com/the-last-challenge-attack" target="_blank" rel="noopener noreferrer">The Last Challenge Attack Blog</a></li>
</ol>
</li>
<li>
<ol start="14">
<li><a href="https://eprint.iacr.org/2023/691" target="_blank" rel="noopener noreferrer">Weak Fiat-Shamir Attacks on Modern Proof Systems</a></li>
</ol>
</li>
<li>
<ol start="15">
<li><a href="https://eprint.iacr.org/2024/398" target="_blank" rel="noopener noreferrer">The Last Challenge Attack</a></li>
</ol>
</li>
</ul>]]></content>
<author>
<name>Marvin</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[zkVM Testing Report: Evaluating Zero-Knowledge Virtual Machines for Nescience]]></title>
<id>https://vac.dev/rlog/zkVM-testing</id>
<link href="https://vac.dev/rlog/zkVM-testing"/>
<updated>2024-09-26T12:00:00.000Z</updated>
<summary type="html"><![CDATA[{/ truncate /}]]></summary>
<content type="html"><![CDATA[
<p>Following our initial exploration of zkVMs in our previous blog post [<a href="https://vac.dev/rlog/zkVM-explorations/" target="_blank" rel="noopener noreferrer">1</a>],
we have conducted a series of tests to identify the most suitable zkVM for the Nescience architecture [<a href="https://vac.dev/rlog/Nescience-state-separation-architecture" target="_blank" rel="noopener noreferrer">2</a>].
This post outlines the testing process, results, and conclusions. Additionally, the full test suite and scripts can be found
on our GitHub page [<a href="https://github.com/vacp2p/nescience-zkvm-testing" target="_blank" rel="noopener noreferrer">3</a>], allowing others to replicate the results or explore the candidates further.
Please note that we chose not to use hardware acceleration in our benchmarks, as our project is aimed at a broad audience.
Particularly, we cannot assume AVX512 support by default, as it is typically available only in high-end CPUs.</p>
<p>We've shortlisted the following zkVMs for testing:</p>
<ul>
<li>SP1 [<a href="https://blog.succinct.xyz/introducing-sp1/" target="_blank" rel="noopener noreferrer">4</a>]</li>
<li>RISC0 [<a href="https://www.risczero.com/zkvm" target="_blank" rel="noopener noreferrer">5</a>]</li>
<li>Nexus [<a href="https://docs.nexus.xyz/" target="_blank" rel="noopener noreferrer">6</a>]</li>
<li>ZkMIPS [<a href="https://docs.zkm.io/zkm-architecture" target="_blank" rel="noopener noreferrer">7</a>]</li>
<li>ZkWASM [<a href="https://delphinuslab.com/zk-wasm/" target="_blank" rel="noopener noreferrer">8</a>]</li>
<li>Valida [<a href="https://delendum.xyz/writings/2023-05-10-zkvm-design.html" target="_blank" rel="noopener noreferrer">9</a>]</li>
</ul>
<h1>Why these candidates?</h1>
<p>When narrowing down the zkVMs, we focused on key factors:</p>
<ul>
<li>True zero-knowledge functionality: The zkVMs had to demonstrate or be close to demonstrating the ability to generate and verify zero-knowledge proofs.</li>
<li>Performance baselines: We sought zkVMs with solid benchmarks in performance, particularly in speed and efficiency.</li>
<li>Specific functionalities: For Nescience, functionalities like lookup tables, precompiles, and recursive capabilities are critical.</li>
</ul>
<p>We need a zkVM that supports these to enable robust project development.</p>
<h1>Preliminary information on the candidates</h1>
<ol>
<li>
<p>SP1 is a performant, open-source zkVM that verifies the execution of arbitrary Rust (or any LLVM-compiled language) programs.
SP1 utilizes Plonky3, enabling recursive proofs and supporting a wide range of cryptographic algorithms, including ECC-based ones like Groth16.
While it supports aggregation, it appears not to support zero knowledge in a conventional manner.</p>
</li>
<li>
<p>RISC0 zkVM allows one to prove the correct execution of arbitrary Rust code. Built on a RISC-V architecture, it is inherently adaptable
for implementing standard cryptographic hash functions such as SHA-256 and ECDSA. RISC0 employs STARKs, providing a security level of 98 bits.
It supports multiple programming languages, including C and Rust, thanks to its compatibility with LLVM and WASM.</p>
</li>
<li>
<p>Nexus is a modular, extensible, open-source, highly parallelized, prover-optimized, and contributor-friendly zkVM written in Rust.
It focuses on performance and security, using the Nova folding scheme, which is particularly effective for recursive proofs.
Nexus also supports precompiles and targeted compilation, and besides Rust, it offers C++ support.</p>
</li>
<li>
<p>ZkMIPS is a general verifiable computing infrastructure based on Plonky2 and the MIPS microarchitecture, aiming to empower Ethereum
as a global settlement layer. It can run arbitrary Rust code as well. Notably, zkMIPS is the only zkVM in this list that utilizes the MIPS opcode set.</p>
</li>
<li>
<p>ZkWASM adheres to and supports the unmodified standard WASM bytecode specification. Since Rust code can be compiled to WASM bytecode,
one could theoretically run any Rust code on a zkWASM machine, providing flexibility and broad language support.</p>
</li>
<li>
<p>Valida is a STARK-based virtual machine aiming to improve upon the state of the art in several categories:</p>
<ul>
<li>Code reuse: The VM has a RISC-inspired instruction set, simplifying the targeting of conventional programming languages.
A backend compiler is being developed to compile LLVM IR to the Valida ISA, enabling the proving of programs written in Rust,
Go, C++, and others with minimal to no changes in source code.</li>
<li>Prover performance: Engineered to maximize prover performance, Valida is compatible with a 31-bit field, restricted to degree 3 constraints,
and features minimal instruction decoding. It operates directly on memory without general-purpose registers or a dedicated stack,
utilizing newer lookup arguments to reduce trace overhead involved in cross-chip communication.</li>
<li>Extensibility: Designed to be customizable, Valida can easily be extended to include an arbitrary number of user-defined instructions.
Procedural macros are used to construct the desired machine at compile time, avoiding any runtime penalties.</li>
</ul>
</li>
</ol>
<p>Valida appears to be in the early stages of development but already showcases respectable performance metrics.</p>
<h1>Testing plan</h1>
<p>To thoroughly evaluate each zkVM, we devised a two-stage testing process:</p>
<ul>
<li>
<p>Stage 1: Arithmetic operations</p>
<p>The first phase focused on evaluating the zkVMs’ ability to handle basic arithmetic operations: addition, subtraction, multiplication,
division, modulus division, and square root calculations. We designed the test around heptagonal numbers, which required zkVMs to process
multiple arithmetic operations simultaneously. By using this method, we could measure efficiency and speed in handling complex mathematical calculations –
a crucial element for zkVM performance.</p>
</li>
<li>
<p>Stage 2: Memory consumption</p>
<p>For the second phase, we evaluated each zkVM’s ability to manage memory under heavy loads. We tested several data structures, including lists,
hash maps, deques, queues, BTreeMaps, hash sets, and binary heaps. Each zkVM underwent tests for the following operations:</p>
<ul>
<li>Insert: How quickly can the zkVM add data to structures?</li>
<li>Delete: Does the zkVM handle memory release effectively?</li>
<li>Append: Can the zkVM efficiently grow data structures?</li>
<li>Search: How fast and efficient is the zkVM when retrieving stored data?</li>
</ul>
</li>
</ul>
<p>The purpose of this stage was to identify any memory bottlenecks and to determine whether a zkVM could manage high-intensity tasks efficiently,
something vital for the Nescience project’s complex, data-heavy processes.</p>
<h1>Machine specifications</h1>
<p>The tests were conducted on the following hardware configuration:</p>
<ul>
<li>CPU: AMD EPYC 7713 "Milan" 64-core processor (128 threads total)</li>
<li>RAM: 600GiB DDR4 3200MHz ECC RAM, distributed across 16 DIMMs</li>
<li>Host OS: Proxmox 8.3</li>
<li>Hypervisor: KVM</li>
<li>Network layer: Open vSwitch</li>
<li>Machine model: Supermicro AS-2024US-TRT</li>
</ul>
<h1>Results</h1>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-sp1">1. SP1<a href="https://vac.dev/rlog/zkVM-testing#1-sp1" class="hash-link" aria-label="Direct link to 1. SP1" title="Direct link to 1. SP1">​</a></h3>
<p>SP1 does not provide zero-knowledge capability in its proofs but delivers respectable performance, though slightly behind its main competitor.
Memory leaks were minimal, staying below the 700 KB threshold. Interestingly, SP1 consumed more RAM during the basic arithmetic
test than in memory allocation tests, showcasing the team's effective handling of memory under load. In the basic test,
allocations were primarily in the 9-16 B, 33-64 B, and 65-128 B ranges. For memory allocations, most fell into the 129-256 B range.</p>
<ul>
<li>Stage 1: Hept 100 test<!-- -->
<ul>
<li>Proof size: 3.108 MB</li>
<li>Proof time: 16.95 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general11-51932659ec4e58ad9f1b20013b3abdda.png" width="1318" height="778" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc11-5e8896fbfcf04b3abe1b53fd63b4a04d.png" width="1310" height="797" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc11-2ba6018b7760dfd150567d789283ffdf.png" width="1307" height="796" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed11-9cb04fe0b2e8a8a6e24ae048041099d4.png" width="1308" height="796" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes11-6ed8118385ec2b2570e7aaeee1f6541e.png" width="1308" height="798" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<ul>
<li>Stage 2: Vec 10000 test<!-- -->
<ul>
<li>Proof size: 3.17 MB</li>
<li>Proof time: 20.85 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general12-aa03eb35a9936b02b34ff2ae3dc2a764.png" width="1316" height="777" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc12-f0d03e2eb102436dd8d14827ffeee782.png" width="1320" height="794" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc12-1e7c0754f86c80cf83b4d58183816de6.png" width="1317" height="799" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed12-afda980a23ad27bab9dfb32a95a97a3f.png" width="1319" height="798" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes12-a05d9e5b04bc487f75d5ec3322619645.png" width="1324" height="793" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-risc0">2. RISC0<a href="https://vac.dev/rlog/zkVM-testing#2-risc0" class="hash-link" aria-label="Direct link to 2. RISC0" title="Direct link to 2. RISC0">​</a></h3>
<p>RISC0 stands out with exceptional performance in proof size and generation time, ranking among the best
(with the exception of Valida and zkWASM's basic test). It also handles memory well, with minor leaks under 0.5 MB
and controlled RAM consumption staying below 2.2 GB. RISC0's memory allocations were consistent, primarily in the 17-32 B and 33-64 B ranges.</p>
<ul>
<li>Stage 1: Hept 100 test<!-- -->
<ul>
<li>Proof size: 217.4 KB</li>
<li>Proof time: 9.73 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general21-06d52d151e217cbc9ebe65b1dee0fd76.png" width="1324" height="759" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc21-f0c07620d2c2a6dcc1cb8dd53d8bf33e.png" width="1314" height="799" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc21-875ab00f360822c237156c64609b1367.png" width="1315" height="798" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed21-c4091416c0c2cbd0effc9d0e349308ec.png" width="1315" height="801" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes21-b404c76c47b45312f15afe077e97c5d8.png" width="1322" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<ul>
<li>Stage 2: Vec 10000 test<!-- -->
<ul>
<li>Proof size: 217.4 KB</li>
<li>Proof time: 16.63 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general22-4c59fa28bca8c2b2cbd3d5f787e48489.png" width="1322" height="773" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc22-0f0788c13a8f29a95543b44ffc3f7e5c.png" width="1317" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc22-214f62229a8d204d44e83dfcf6a69c19.png" width="1318" height="801" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed22-f8b42af736f124afd52887d14e5df7d9.png" width="1319" height="799" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes22-e801d2de7a6ae71ed02556af421d17b4.png" width="1316" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<p>Based on these results, RISC0 is a solid candidate for Nescience.</p>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-nexus">3. Nexus<a href="https://vac.dev/rlog/zkVM-testing#3-nexus" class="hash-link" aria-label="Direct link to 3. Nexus" title="Direct link to 3. Nexus">​</a></h3>
<p>Nexus' performance offers interesting insights into folding scheme-based zkVMs. Surprisingly, proof sizes remained consistent
regardless of workload, with no significant memory leaks (under 700 KB). However, while RAM consumption increased slightly with higher
workloads (up to 1.2 GB), Nexus performed poorly during memory allocation tests, making it unsuitable for our use case.</p>
<ul>
<li>
<p>Allocation details:</p>
<ul>
<li>Basic test: Most allocations concentrated in 65-128 B</li>
<li>Memory-heavy test: Allocations in the 129-256 B range</li>
</ul>
</li>
<li>
<p>Stage 1: Hept 100 test</p>
<ul>
<li>Proof size: 46 MB</li>
<li>Proof time: 12.06 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general31-127358c1aa2715173141d55c78c79d70.png" width="1325" height="776" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc31-b33243d1b3e859704fa649c3cca423ae.png" width="1321" height="807" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc31-7b52143b8f6199186fb0ae7c66486365.png" width="1316" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed31-d75cf19d4acef0f2cafe8eb19a3605c4.png" width="1313" height="801" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes31-9af6876cd32a8486431c0859f5c15e7c.png" width="1321" height="793" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<ul>
<li>Stage 2: Vec 10000 test<!-- -->
<ul>
<li>Proof size: 46 MB</li>
<li>Proof time: 56 minutes</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general32-eb289f0be9cc090fc455d823c26bd310.png" width="1318" height="776" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc32-b4511228d11e730b80e97dbfe14f1b32.png" width="1320" height="804" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc32-0a2c79a5578806df5da0f97b15eb1c56.png" width="1315" height="798" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed32-44dfd430560af8658b94a3ef9f7e6e6f.png" width="1322" height="795" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes32-3011b5545d0899d6c3bf6a3c7f0c1304.png" width="1312" height="804" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-zkmips">4. ZkMIPS<a href="https://vac.dev/rlog/zkVM-testing#4-zkmips" class="hash-link" aria-label="Direct link to 4. ZkMIPS" title="Direct link to 4. ZkMIPS">​</a></h3>
<p>ZkMIPS presents an intriguing case. While it shows good results in terms of proof size and generation time during the basic test,
these come at the cost of significant RAM usage and memory leaks. The memory allocation test revealed a concerning 6.7 GB memory leak,
with 0.5 GB leaked during the basic test. Despite this, RAM consumption (while high at 17+ GB) remains stable under higher workloads.
Allocation sizes are spread across several ranges, with notable concentrations in the 17-32 B, 65-128 B, and 257-512 B slots.</p>
<ul>
<li>Stage 1: Hept 100 test<!-- -->
<ul>
<li>Proof size: 4.3 MB</li>
<li>Proof time: 9.32 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general41-949405deaef610fd9742055a23363f7e.png" width="1323" height="779" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc41-7df1382edc6f2440f4becb9306679308.png" width="1321" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc41-3174820f91664038d6fdd966f07bd90d.png" width="1316" height="802" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed41-362c4850d936a407e75ccd58283d88d1.png" width="1317" height="806" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes41-2eca56b296b460aa98b34ff0e3642a67.png" width="1307" height="806" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<ul>
<li>Stage 2: Vec 10000 test<!-- -->
<ul>
<li>Proof size: 4.898 MB</li>
<li>Proof time: 42.57 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general42-39f202b1dfdcecc289d3582e20cde498.png" width="1324" height="776" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc42-29736c27a94ac18072ccad4ba523374d.png" width="1312" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc42-6122a1bdf8c5db3a03fa4249ebb52e1f.png" width="1314" height="801" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed42-a28992ae3211a5afb2458cf153e160be.png" width="1310" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes42-5024bdc66d052e6b6636040cef38bd93.png" width="1305" height="800" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<p>This zkVM provides mixed results with strong proof generation but concerning memory management issues.</p>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-zkwasm">5. ZkWASM<a href="https://vac.dev/rlog/zkVM-testing#5-zkwasm" class="hash-link" aria-label="Direct link to 5. ZkWASM" title="Direct link to 5. ZkWASM">​</a></h3>
<p>ZkWASM, unfortunately, performed poorly in both stages regarding proof size and generation time. RAM consumption was particularly high,
exceeding 7 GB in the basic test, and an astounding 57 GB during memory allocation tests. Despite its impressive memory usage,
the proof sizes were relatively large at 18 KB and 334 KB respectively. Allocation sizes were mainly concentrated in the 33-64 B range,
with neighboring slots contributing small but notable amounts.</p>
<ul>
<li>Stage 1: Hept 100 test<!-- -->
<ul>
<li>Proof size: 18 KB</li>
<li>Proof time: 42.7 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general51-72f8449fb89dfdd31ab4eeef2bfa8ebf.png" width="1321" height="778" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc51-987d88b8264639cb4c1edf757b48b8f4.png" width="1314" height="802" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc51-5b0868395d26e76dac7744b216e4949f.png" width="1314" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed51-75711af15f7936d72c04975e332935a7.png" width="1313" height="808" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes51-129c42f705ff833ed745045e7803cb6f.png" width="1304" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<ul>
<li>Stage 2: Vec 10000 test<!-- -->
<ul>
<li>Proof size: 334 KB</li>
<li>Proof time: 323 seconds</li>
</ul>
</li>
</ul>
<table><thead><tr><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 1" src="https://vac.dev/assets/images/general52-3903edbdaf25478fbcabf8ec390ac257.png" width="1322" height="773" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 2" src="https://vac.dev/assets/images/alloc52-ebf1d882e709a00714dce2fd122428eb.png" width="1324" height="791" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 3" src="https://vac.dev/assets/images/tempalloc52-ceccc6d4166dbdedebcdb4370acb9650.png" width="1315" height="799" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 4" src="https://vac.dev/assets/images/consumed52-51cbe29ddaf2bed5d9fc9018b549a00c.png" width="1324" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th><th><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="Image 5" src="https://vac.dev/assets/images/sizes52-505ad4d2e61ad6462a0e9d100fcf234b.png" width="1322" height="803" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div></th></tr></thead></table>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="6-valida">6. Valida<a href="https://vac.dev/rlog/zkVM-testing#6-valida" class="hash-link" aria-label="Direct link to 6. Valida" title="Direct link to 6. Valida">​</a></h3>
<p>Valida delivered impressive results in proof generation speed and size, with a proof size of 280 KB and a proof time of &lt; 1 second.
However, profiling was not possible due to Valida's limited Rust support. Valida currently compiles Rust using the LLVM backend,
transpiling LLVM IR to leverage its C/C++ implementation, which leads to errors when handling Rust-specific data structures or dependencies.
As a result, complex memory interactions couldn't be tested, and using Valida with Rust code is currently not advisable.
A GitHub issue has been opened to address this.</p>
<hr>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary-table">Summary table<a href="https://vac.dev/rlog/zkVM-testing#summary-table" class="hash-link" aria-label="Direct link to Summary table" title="Direct link to Summary table">​</a></h2>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="stage-1">Stage 1<a href="https://vac.dev/rlog/zkVM-testing#stage-1" class="hash-link" aria-label="Direct link to Stage 1" title="Direct link to Stage 1">​</a></h3>
<table><thead><tr><th>zkVM</th><th>Proof time</th><th>Proof size</th><th>Peak RAM consumption</th><th>Memory leaked</th></tr></thead><tbody><tr><td>SP1</td><td>16.95 s</td><td>3.108 MB</td><td>2.1 GB</td><td>656.8 KB</td></tr><tr><td>RISC0</td><td>9.73 s</td><td>217.4 KB</td><td>1.9 GB</td><td>470.5 KB</td></tr><tr><td>Nexus</td><td>12.06 s</td><td>46 MB</td><td>9.7 MB</td><td>646.5 KB</td></tr><tr><td>ZkMIPS</td><td>9.32 s</td><td>4.3 MB</td><td>17.3 GB</td><td>453.8 MB</td></tr><tr><td>ZkWASM</td><td>42.7 s</td><td>18 KB</td><td>8.2 GB</td><td>259.4 KB</td></tr><tr><td>Valida</td><td>&lt; 1 s</td><td>280 KB</td><td>N/A</td><td>N/A</td></tr></tbody></table>
<hr>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="stage-2">Stage 2<a href="https://vac.dev/rlog/zkVM-testing#stage-2" class="hash-link" aria-label="Direct link to Stage 2" title="Direct link to Stage 2">​</a></h3>
<table><thead><tr><th>zkVM</th><th>Proof time</th><th>Proof size</th><th>Peak RAM consumption</th><th>Memory leaked</th></tr></thead><tbody><tr><td>SP1</td><td>20.85 s</td><td>3.17 MB</td><td>1.9 GB</td><td>616 KB</td></tr><tr><td>RISC0</td><td>16.63 s</td><td>217.4 KB</td><td>2.3 GB</td><td>485.3 KB</td></tr><tr><td>Nexus</td><td>56 m</td><td>46 MB</td><td>1.9 GB</td><td>616 KB</td></tr><tr><td>ZkMIPS</td><td>42.57 s</td><td>4.898 MB</td><td>18.9 GB</td><td>6.9 GB</td></tr><tr><td>ZkWASM</td><td>323 s</td><td>334 KB</td><td>58.8 GB</td><td>259.4 KB</td></tr><tr><td>Valida</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td></tr></tbody></table>
<hr>
<h1>Summary</h1>
<p>After an extensive evaluation of six zkVM candidates for the Nescience project, RISC0 emerged as the top choice.
It excels in both proof generation time and size while maintaining a reasonable memory footprint. With strong zero-knowledge
proof capabilities and support for multiple programming languages, it aligns well with our project's needs for privacy,
performance, and flexibility. Its overall balance between performance and efficiency makes it the most viable zkVM at this stage.</p>
<p>Valida, while promising with its potential for high prover performance, is still in early development and suffers from Rust integration issues.
The current LLVM IR transpilation limitations mean it cannot handle complex memory interactions, disqualifying it for now.
However, once its development matures, Valida could become a strong alternative, and we plan to revisit it as it evolves.</p>
<p>SP1, though initially interesting, failed to meet the zero-knowledge proof requirement. Its performance in arithmetic operations was
respectable but insufficient to justify further consideration given its lack of ZK functionality – critical for our privacy-first objectives.</p>
<p>Nexus demonstrated consistent proof sizes and manageable memory usage, but its lackluster performance during memory-intensive tasks and
its proof size (especially for larger workloads) disqualified it from being a top contender. While zkMIPS delivered solid proof times,
the memory issues were too significant to ignore, making it unsuitable.</p>
<p>Finally, zkWASM exhibited the poorest results, struggling both in proof size and generation time. Despite its potential for WASM bytecode support,
the excessive RAM consumption (up to 57 GB in the memory test) rendered it impractical for Nescience’s use case.</p>
<p>In conclusion, RISC0 is the best fit for Nescience at this stage, but Valida remains a future candidate as its development progresses.</p>
<p>In the future, we plan to compare RISC0 and SP1 with CUDA acceleration. Ideally, by that time, more zkVMs will include similar acceleration capabilities,
enabling a fairer and more comprehensive comparison across platforms.</p>
<p>We’d love to hear your thoughts on our zkVM testing process and results! Do you agree with our conclusions, or do you think we missed a promising zkVM?
We’re always open to feedback, insights, and suggestions from the community.</p>
<p>Join the discussion and share your perspectives on
<a href="https://forum.vac.dev/t/zkvm-testing-report-evaluating-zero-knowledge-virtual-machines-for-nescience/" target="_blank" rel="noopener noreferrer">our forum</a> or try out the
tests yourself through our <a href="https://github.com/vacp2p/nescience-zkvm-testing" target="_blank" rel="noopener noreferrer">GitHub page</a>!</p>
<h1>References</h1>
<p>[1] Exploring zkVMs: Which Projects Truly Qualify as Zero-Knowledge Virtual Machines? Retrieved from <a href="https://vac.dev/rlog/zkVM-explorations/" target="_blank" rel="noopener noreferrer">https://vac.dev/rlog/zkVM-explorations/</a></p>
<p>[2] Nescience: A User-Centric State-Separation Architecture. Retrieved from <a href="https://vac.dev/rlog/Nescience-state-separation-architecture" target="_blank" rel="noopener noreferrer">https://vac.dev/rlog/Nescience-state-separation-architecture</a></p>
<p>[3] Our GitHub Page for zkVM Testing. Retrieved from <a href="https://github.com/vacp2p/nescience-zkvm-testing" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/nescience-zkvm-testing</a></p>
<p>[4] Introducing SP1: A performant, 100% open-source, contributor-friendly zkVM. Retrieved from <a href="https://blog.succinct.xyz/introducing-sp1/" target="_blank" rel="noopener noreferrer">https://blog.succinct.xyz/introducing-sp1/</a></p>
<p>[5] The first general purpose zkVM. Retrieved from <a href="https://www.risczero.com/zkvm" target="_blank" rel="noopener noreferrer">https://www.risczero.com/zkvm</a></p>
<p>[6] The Nexus 2.0 zkVM. Retrieved from <a href="https://docs.nexus.xyz/" target="_blank" rel="noopener noreferrer">https://docs.nexus.xyz/</a></p>
<p>[7] ZKM Architecture. Retrieved from <a href="https://docs.zkm.io/zkm-architecture" target="_blank" rel="noopener noreferrer">https://docs.zkm.io/zkm-architecture</a></p>
<p>[8] ZK-WASM. Retrieved from <a href="https://delphinuslab.com/zk-wasm/" target="_blank" rel="noopener noreferrer">https://delphinuslab.com/zk-wasm/</a></p>
<p>[9] Valida zkVM Design. Retrieved from <a href="https://delendum.xyz/writings/2023-05-10-zkvm-design.html" target="_blank" rel="noopener noreferrer">https://delendum.xyz/writings/2023-05-10-zkvm-design.html</a></p>]]></content>
<author>
<name>Moudy</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Exploring zkVMs: Which Projects Truly Qualify as Zero-Knowledge Virtual Machines?]]></title>
<id>https://vac.dev/rlog/zkVM-explorations</id>
<link href="https://vac.dev/rlog/zkVM-explorations"/>
<updated>2024-08-27T12:00:00.000Z</updated>
<summary type="html"><![CDATA[{/ truncate /}]]></summary>
<content type="html"><![CDATA[
<p>The blockchain space is rapidly evolving, and with it, new technologies are emerging that promise enhanced privacy, scalability, and security.
As decentralized systems grow in complexity and usage, the need for secure and private computation has never been greater.
Zero-knowledge virtual machines (zkVMs) are one such innovation, allowing for computations to be proven correct without revealing the underlying data.
ZkVMs have enormous implications for privacy-preserving applications, decentralized finance (DeFi), and other blockchain-based use cases.
However, as the term "zkVM" becomes more widely adopted, it is critical to distinguish between projects that truly satisfy the stringent requirements of a zkVM and those that do not.</p>
<h1>What is a zkVM?</h1>
<p>A zkVM is a virtual machine that combines the principles of cryptographic proof generation and privacy preservation with the computational model
of traditional virtual machines. Essentially, a zkVM enables the execution of arbitrary programs while generating cryptographic proofs—specifically, zero-knowledge proofs (ZKPs)—that
can verify the correctness of these computations without revealing any sensitive information. This ensures that computations can be trusted while protecting the privacy of the data involved.
The key characteristics of a zkVM include:</p>
<ul>
<li>Proof generation: The ability to produce ZKPs that verify the correct execution of programs. There are several types of cryptographic techniques used in zkVMs to
generate these proofs, such as zk-SNARKs, zk-STARKs, and recursive proofs. A zkVM’s ability to generate these proofs determines how effectively it can ensure the integrity of computations
in a privacy-preserving manner.</li>
<li>Privacy preservation: The system must maintain privacy, ensuring that only the proof is revealed, not the underlying computation or data. Privacy-preserving zkVMs allow users to maintain
confidentiality without compromising the security or verifiability of their operations. However, not all zkVMs achieve the same level of privacy. Some may focus more on proof generation
and scalability while deprioritizing privacy features, which can limit their use in certain privacy-sensitive applications.</li>
<li>Scalability and performance: zkVMs should offer scalable and efficient computation, leveraging advanced cryptographic techniques like zk-SNARKs, zk-STARKs, or recursive proofs.
A zkVM's performance must also be measured in terms of latency (time to generate and verify a proof) and throughput (number of computations processed within a certain time frame).</li>
<li>Verifiable computation: The zkVM should be able to prove the execution of arbitrary programs in a secure and verifiable manner. Verifiable computation ensures that zkVMs can be deployed
across a wide range of applications, from DeFi to private data-sharing platforms and more.</li>
</ul>
<h1>Why zkVMs matter</h1>
<p>The rise of zkVMs is a crucial development for the future of blockchain and decentralized technologies. As more systems require the ability to scale while maintaining privacy and trust,
zkVMs provide a powerful solution. They offer the potential to reshape the way decentralized applications (dapps) handle sensitive information, enabling them to be both efficient and private.</p>
<p>It is essential to distinguish between projects that fully realize the potential of zkVMs and those that do not. In the remainder of this post, we evaluate several zkVM projects, analyzing
whether they satisfy the criteria for being classified as zkVMs based on our research.</p>
<h1>Our methodology</h1>
<p>We analyzed each project’s documentation, source code, and available benchmarks to determine whether they meet the definition of a zkVM.
Our criteria focus on the key capabilities of zkVMs—proof generation, privacy, scalability, and integration with existing systems.</p>
<h1>ZkVM project analysis</h1>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-sp1">1. [SP1]<a href="https://vac.dev/rlog/zkVM-explorations#1-sp1" class="hash-link" aria-label="Direct link to 1. [SP1]" title="Direct link to 1. [SP1]">​</a></h2>
<ul>
<li>Overview: SP1 [<a href="https://blog.succinct.xyz/introducing-sp1/" target="_blank" rel="noopener noreferrer">1</a>] is a developer-friendly zkVM designed to enable ZKP execution for LLVM-based languages like C, C++, Rust, and others. It supports a RISC-V-like instruction set architecture (ISA),
which makes it compatible with various programming languages compiled through LLVM.</li>
<li>Main focus: The main focus of SP1 is scalability, open-source contributions, and accessibility for developers. It prioritizes performance over privacy,
making it a good fit for environments where privacy isn't the primary concern.</li>
<li>Privacy: Not explicitly mentioned, making it less suitable for privacy-preserving applications.</li>
<li>Performance: SP1 has demonstrated up to 5.4x better performance than similar zkVMs like RISC0 for specific computations such as Fibonacci sequence generation.</li>
<li>Integration: SP1 is highly adaptable for rollups, light client verifiers, oracles, and even web2 projects like verifying the originality of images.</li>
<li>Conclusion: Yes, SP1 is a zkVM, but it does not prioritize zero-knowledge privacy, focusing more on scalability and performance.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-nexus">2. [Nexus]<a href="https://vac.dev/rlog/zkVM-explorations#2-nexus" class="hash-link" aria-label="Direct link to 2. [Nexus]" title="Direct link to 2. [Nexus]">​</a></h2>
<ul>
<li>Overview: Nexus [<a href="https://docs.nexus.xyz/" target="_blank" rel="noopener noreferrer">2</a>] is a highly modular zkVM designed to process up to a trillion CPU cycles per second. It relies on RISC-V instructions for computation, making it extensible and scalable.
However, it currently lacks full ZKP capabilities due to its use of Spartan proofs.</li>
<li>Main focus: Nexus focuses on high performance and scalability, aiming to create an efficient execution environment for computationally intensive tasks.</li>
<li>Privacy: Although zero-knowledge privacy isn't the primary feature of Nexus, the project hints at potential privacy enhancements in the future.</li>
<li>Performance: Nexus has a high theoretical throughput, but it has yet to demonstrate benchmarks on zero-knowledge privacy.</li>
<li>Integration: Nexus is a good fit for high-performance environments that do not necessarily require full privacy.</li>
<li>Conclusion: Yes, Nexus qualifies as a zkVM in terms of scalability and proof generation, but it does not yet achieve full zero-knowledge privacy.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-risc0">3. [RISC0]<a href="https://vac.dev/rlog/zkVM-explorations#3-risc0" class="hash-link" aria-label="Direct link to 3. [RISC0]" title="Direct link to 3. [RISC0]">​</a></h2>
<ul>
<li>Overview: Risc0 [<a href="https://www.risczero.com/zkvm" target="_blank" rel="noopener noreferrer">3</a>] is a general-purpose zkVM with strong developer support. It allows for the execution of Rust and C code on a RISC-V virtual machine
and generates zk-SNARK and zk-STARK proofs for these computations.</li>
<li>Main focus: Risc0 is focused on ease of use for developers by abstracting away the complexities of circuit generation, making it accessible for a wide range of use cases.</li>
<li>Privacy: Full zero-knowledge privacy is supported via zk-SNARK and zk-STARK proofs, with Groth16 used for constant-size proof generation.</li>
<li>Performance: Risc0 offers strong benchmarks across different hardware setups, making it one of the most versatile zkVMs in terms of performance and scalability.</li>
<li>Integration: Risc0 integrates with several ecosystems, including Ethereum, and supports verifiable execution of Rust-based programs.</li>
<li>Conclusion: Yes, Risc0 qualifies as a zkVM, offering a balance of developer usability, scalability, and privacy.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-powdr">4. [Powdr]<a href="https://vac.dev/rlog/zkVM-explorations#4-powdr" class="hash-link" aria-label="Direct link to 4. [Powdr]" title="Direct link to 4. [Powdr]">​</a></h2>
<ul>
<li>Overview: Powdr [<a href="https://docs.powdr.org/" target="_blank" rel="noopener noreferrer">4</a>] is a toolkit for creating custom zkVMs. It allows developers to select from various front-end and back-end components to create zkVMs tailored to specific needs.</li>
<li>Main focus: Powdr is focused on providing a modular architecture for zkVM creation. It enables flexibility by allowing the combination of different ZK-proof backends like Halo2 or Valida.</li>
<li>Privacy: Powdr itself does not generate ZKPs, but it facilitates the creation of zkVMs that do.</li>
<li>Performance: The performance depends on the components chosen by the developer, as Powdr itself is more of a framework.</li>
<li>Integration: Powdr is highly customizable and can integrate with existing zkVM frameworks to extend their capabilities.</li>
<li>Conclusion: No, Powdr is not a zkVM itself, but it is a powerful tool for building customized zkVMs with different privacy and performance needs.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="5-zkmips">5. [ZkMIPS]<a href="https://vac.dev/rlog/zkVM-explorations#5-zkmips" class="hash-link" aria-label="Direct link to 5. [ZkMIPS]" title="Direct link to 5. [ZkMIPS]">​</a></h2>
<ul>
<li>Overview: ZkMIPS [<a href="https://docs.zkm.io/zkm-architecture" target="_blank" rel="noopener noreferrer">5</a>] uses zk-STARKs to ensure privacy during computation, ensuring that private inputs are preserved while still proving correctness.</li>
<li>Performance: ZkMIPS is built for scalability, though explicit benchmarks are not widely published.</li>
<li>Integration: ZkMIPS can be integrated into systems that rely on MIPS architecture, making it versatile for legacy codebases that require privacy.</li>
<li>Conclusion: Yes, zkMIPS is a zkVM focused on scalability and privacy for MIPS-based architectures.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="6-valida">6. [Valida]<a href="https://vac.dev/rlog/zkVM-explorations#6-valida" class="hash-link" aria-label="Direct link to 6. [Valida]" title="Direct link to 6. [Valida]">​</a></h2>
<ul>
<li>Overview: Valida [<a href="https://delendum.xyz/writings/2023-05-10-zkvm-design.html" target="_blank" rel="noopener noreferrer">6</a>] is a performance-oriented zkVM that generates proofs for programs using a custom ISA designed to optimize zkVM implementation.
It uses Plonky3 for its proof system.</li>
<li>Main focus: Valida is centered around optimizing prover performance and extensibility, making it a valuable tool for generating proofs efficiently.</li>
<li>Privacy: While Valida is focused on performance, it does not prioritize zero-knowledge privacy as much as other zkVMs.</li>
<li>Performance: Valida has benchmarks indicating its performance advantages in proving computations quickly, particularly through parallel processing.</li>
<li>Integration: Valida is specialized and may not integrate as seamlessly into general-purpose systems, as it is optimized for performance over broad applicability.</li>
<li>Conclusion: Yes, Valida qualifies as a zkVM based on proof generation, but its lack of focus on privacy makes it less suitable for privacy-first use cases.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="7-jolt">7. [Jolt]<a href="https://vac.dev/rlog/zkVM-explorations#7-jolt" class="hash-link" aria-label="Direct link to 7. [Jolt]" title="Direct link to 7. [Jolt]">​</a></h2>
<ul>
<li>Overview: Jolt [<a href="https://a16zcrypto.com/posts/article/building-jolt/" target="_blank" rel="noopener noreferrer">7</a>] is a zkVM built to optimize prover performance using a modified Hyrax polynomial commitment system. It relies on RISC-V instructions for computation
but falls short of full zero-knowledge capabilities.</li>
<li>Main focus: Jolt's main goal is to optimize the speed of proving program execution, making it suitable for high-performance applications where privacy isn't the primary concern.</li>
<li>Privacy: Jolt does not fully achieve zero-knowledge privacy due to the choice of polynomial commitment schemes.</li>
<li>Performance: Jolt offers strong performance, with benchmarks highlighting its ability to process proofs efficiently.</li>
<li>Integration: Jolt can be integrated with systems that prioritize speed over privacy, particularly where rapid proof generation is essential.</li>
<li>Conclusion: Yes, Jolt qualifies as a zkVM based on proof generation, though it does not provide full zero-knowledge privacy.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="8-zkwasm">8. [ZkWASM]<a href="https://vac.dev/rlog/zkVM-explorations#8-zkwasm" class="hash-link" aria-label="Direct link to 8. [ZkWASM]" title="Direct link to 8. [ZkWASM]">​</a></h2>
<ul>
<li>Overview: ZkWASM [<a href="https://delphinuslab.com/zk-wasm/" target="_blank" rel="noopener noreferrer">8</a>] is a zkVM designed to execute WebAssembly (WASM) code in a privacy-preserving and scalable manner. It uses zk-SNARKs to prove the correctness of WASM
program execution while ensuring privacy.</li>
<li>Main focus: ZkWASM focuses on scalability and privacy for WebAssembly, making it ideal for dapps that require verifiable computation without compromising privacy.</li>
<li>Privacy: Full zero-knowledge privacy is provided through zk-SNARKs, ensuring that the execution of WASM programs remains confidential.</li>
<li>Performance: ZkWASM is optimized for running WASM programs efficiently, with offchain computation and onchain verification to enhance performance.</li>
<li>Integration: ZkWASM is ideal for dapps, particularly those that use WebAssembly and require verifiable execution.</li>
<li>Conclusion: Yes, zkWASM qualifies as a zkVM, providing strong privacy, scalability, and verifiable execution for WebAssembly code.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="9-aleo">9. [Aleo]<a href="https://vac.dev/rlog/zkVM-explorations#9-aleo" class="hash-link" aria-label="Direct link to 9. [Aleo]" title="Direct link to 9. [Aleo]">​</a></h2>
<ul>
<li>Overview: Aleo's [<a href="https://aleo.org/blog/" target="_blank" rel="noopener noreferrer">9</a>] snarkVM converts code into Aleo instructions, which are then compiled into bytecode executable on its zkVM. Aleo emphasizes building private, scalable dapps.</li>
<li>Main focus: Aleo prioritizes privacy and scalability for dapps, providing a robust framework for developers building private dapps.</li>
<li>Privacy: Aleo offers full privacy through zk-SNARK proofs, making it suitable for building fully private applications.</li>
<li>Performance: Aleo focuses on scalability through efficient proof systems, though detailed performance benchmarks are not widely available.</li>
<li>Integration: Aleo is built for privacy-first dapps and integrates with other zkVM-based systems.</li>
<li>Conclusion: Yes, Aleo qualifies as a zkVM, offering a comprehensive solution for private and scalable dapps.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="10-ola">10. [Ola]<a href="https://vac.dev/rlog/zkVM-explorations#10-ola" class="hash-link" aria-label="Direct link to 10. [Ola]" title="Direct link to 10. [Ola]">​</a></h2>
<ul>
<li>Overview: Ola [<a href="https://github.com/Sin7Y/olavm-whitepaper-v2/tree/master" target="_blank" rel="noopener noreferrer">10</a>] is a ZK-friendly, high-performance layer-2 (L2) rollup platform that is still under development. It is designed to execute computations offchain while generating
validity proofs for these computations, ensuring that they are correctly executed without compromising security.</li>
<li>Privacy: Ola does not specifically prioritize privacy in the same way that zkVMs do. While it leverages ZKPs for scalability, its focus is on proving the correctness of
transactions and computations rather than ensuring that the data remains private.</li>
<li>Performance: Ola is designed to achieve high performance, particularly in terms of transaction throughput.</li>
<li>Integration: Ola is designed to be interoperable with various layer-1 blockchains. The platform supports a hybrid ZK-rollup architecture and is expected to include bridges for cross-chain
interoperability, enabling assets and data to move seamlessly between the layer-1 blockchain and the Ola rollup.</li>
<li>Conclusion: No, Ola is not a zkVM. While it leverages ZKPs (in the form of ZK-rollups) to ensure the validity of offchain computations, its primary focus is on scalability and performance
rather than privacy or verifiable execution of arbitrary programs. Ola is more accurately described as a ZK-rollup platform aimed at improving transaction throughput and reducing transaction costs on
layer-1 blockchains.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="11-miden">11. [Miden]<a href="https://vac.dev/rlog/zkVM-explorations#11-miden" class="hash-link" aria-label="Direct link to 11. [Miden]" title="Direct link to 11. [Miden]">​</a></h2>
<ul>
<li>Overview: Miden zkVM [<a href="https://0xpolygonmiden.github.io/miden-vm/intro/main.html" target="_blank" rel="noopener noreferrer">11</a>] is a zk-STARK-based virtual machine that converts code into Miden VM instructions and proves the execution of these instructions with zero-knowledge privacy.</li>
<li>Main focus: Miden focuses on scalability and privacy for ZK-rollups, offering efficient proof generation for dapps.</li>
<li>Privacy: Miden ensures privacy for transactions and programs via zk-STARK proofs, making it suitable for private dapps.</li>
<li>Performance: Miden is optimized for scalability, with benchmarks showing its ability to handle up to 1,000 transactions per second (TPS).</li>
<li>Integration: Miden integrates well with ZK-rollup solutions, making it ideal for L2 scaling solutions on blockchains like Ethereum.</li>
<li>Conclusion: Yes, Miden qualifies as a zkVM, providing strong privacy and scalability for dapps and ZK-rollups.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="12-zkos">12. [ZkOS]<a href="https://vac.dev/rlog/zkVM-explorations#12-zkos" class="hash-link" aria-label="Direct link to 12. [ZkOS]" title="Direct link to 12. [ZkOS]">​</a></h2>
<ul>
<li>Overview: ZkOS [<a href="https://osblog.stephenmarz.com/index.html" target="_blank" rel="noopener noreferrer">12</a>] is a verifiable operating system focused on running zkApps in a decentralized manner. It is built on the RISC-V architecture and aims to create
a world computer where all untrusted executions can be verified.</li>
<li>Main focus: ZkOS is primarily designed to offer a proof-of-concept operating system where all executions can be verified in a trustless manner.
However, its focus is more on the infrastructure for verifiable applications rather than being a traditional zkVM.</li>
<li>Privacy: ZkOS does not focus on privacy guarantees such as those found in zkVMs that generate ZKPs.</li>
<li>Performance: ZkOS focuses on the efficient execution of dapps, but performance benchmarks specific to ZKP generation are not provided.</li>
<li>Integration: ZkOS supports the execution of zkApps, but it is more of a verifiable operating system rather than a zkVM, making it distinct in its functionality.</li>
<li>Conclusion: No, zkOS is not a zkVM. It is a verifiable operating system focused on the infrastructure to support zkApps but does not directly generate ZKPs or focus on privacy preservation.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="13-triton">13. [Triton]<a href="https://vac.dev/rlog/zkVM-explorations#13-triton" class="hash-link" aria-label="Direct link to 13. [Triton]" title="Direct link to 13. [Triton]">​</a></h2>
<ul>
<li>Overview: Triton [<a href="https://triton-vm.org/spec/" target="_blank" rel="noopener noreferrer">13</a>] is a domain-specific language (DSL) and compiler designed primarily for high-performance GPU kernels, particularly those used in deep learning applications.</li>
<li>Main focus: The primary goal of Triton is to optimize computation for machine learning and GPU workloads. It is focused on enhancing performance and efficiency in processing data
rather than on ZKPs or verifiable computation.</li>
<li>Privacy: Triton does not provide ZKPs or privacy features typically associated with zkVMs. Its focus is on high-performance computation rather than cryptographic verifiability.</li>
<li>Performance: Triton is highly optimized for GPU execution, offering significant improvements in performance for computationally intensive tasks such as those found in deep learning.</li>
<li>Integration: Triton is integrated with GPU-based computation environments and is highly specialized for optimizing low-level operations on hardware rather
than being a general-purpose virtual machine.</li>
<li>Conclusion: No, Triton is not a zkVM. It is a specialized tool for optimizing GPU workloads, focusing on performance rather than privacy or ZKPs.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="14-cairo">14. [Cairo]<a href="https://vac.dev/rlog/zkVM-explorations#14-cairo" class="hash-link" aria-label="Direct link to 14. [Cairo]" title="Direct link to 14. [Cairo]">​</a></h2>
<ul>
<li>Overview: Cairo zkVM [<a href="https://github.com/lambdaclass/cairo-vm/blob/main/docs/python_vm/README.md" target="_blank" rel="noopener noreferrer">14</a>] uses a custom language that compiles to an optimized STARK-based proof system, ensuring verifiable computation. It is primarily used in systems like Starknet.</li>
<li>Main focus: Cairo focuses on scalability and performance, using zk-STARK proofs to ensure the verifiable and secure execution of programs.</li>
<li>Privacy: Cairo provides privacy through zk-STARKs, but it focuses more on scalability and performance than privacy-first use cases.</li>
<li>Performance: Cairo is highly optimized for performance, making it well-suited for scalable applications on Starknet.</li>
<li>Integration: Cairo integrates deeply with systems like Starknet, supporting verifiable computation in a highly scalable and efficient manner.</li>
<li>Conclusion: Yes, Cairo qualifies as a zkVM, focusing on performance and verifiable execution while being ZK-friendly.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="15-snarkos">15. [SnarkOS]<a href="https://vac.dev/rlog/zkVM-explorations#15-snarkos" class="hash-link" aria-label="Direct link to 15. [SnarkOS]" title="Direct link to 15. [SnarkOS]">​</a></h2>
<ul>
<li>Overview: SnarkOS [<a href="https://aleo.org/post/aleo-completes-security-audits-of-snarkos-and-snarkvm/" target="_blank" rel="noopener noreferrer">15</a>] is a decentralized operating system designed to power Aleo's network, enabling secure and private dapps.
It manages transactions and consensus, making it a critical infrastructure component for Aleo's zkVM-based ecosystem.</li>
<li>Main focus: SnarkOS primarily focuses on securing Aleo's network through consensus mechanisms and privacy-preserving transactions rather than acting as a
zkVM that directly proves program execution.</li>
<li>Privacy: SnarkOS supports zero-knowledge privacy through its integration with Aleo's zkVM, but the operating system itself does not generate ZKPs for arbitrary computations.</li>
<li>Performance: SnarkOS is optimized for managing dapps on the Aleo network and handling private transactions, but its focus is more on infrastructure
and consensus than on proof generation.</li>
<li>Integration: SnarkOS integrates seamlessly with Aleo's zkVM to support private dapps and transactions, but its primary role is as a consensus layer.</li>
<li>Conclusion: No, SnarkOS is not a zkVM. It serves as an operating system for Aleo's decentralized network, focusing on privacy and consensus rather than on generating ZKPs for computations.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="16-lurk">16. [Lurk]<a href="https://vac.dev/rlog/zkVM-explorations#16-lurk" class="hash-link" aria-label="Direct link to 16. [Lurk]" title="Direct link to 16. [Lurk]">​</a></h2>
<ul>
<li>Overview: Lurk [<a href="https://github.com/lurk-lab" target="_blank" rel="noopener noreferrer">16</a>] is a Turing-complete programming language designed for recursive zk-SNARKs. It focuses on enabling developers to build complex,
recursive ZKPs efficiently through a custom language tailored for verifiable computation.</li>
<li>Main focus: Lurk is centered around recursive proof generation rather than serving as a traditional virtual machine. Its purpose is to facilitate the creation of complex zk-SNARK-based proofs,
making it a specialized tool for cryptographic proofs rather than general-purpose computation.</li>
<li>Privacy: Lurk is built for generating zk-SNARKs, which inherently provide privacy. However, Lurk itself is a language and not a zkVM that executes arbitrary programs and generates ZKPs for them.</li>
<li>Performance: Lurk is optimized for recursive zk-SNARK generation, but specific performance metrics are tied to its proof-generation capabilities rather than traditional execution environments.</li>
<li>Integration: Lurk is specialized for zk-SNARKs and may not easily integrate with other general-purpose systems, as it focuses on specific cryptographic tasks.</li>
<li>Conclusion: No, Lurk is not a zkVM. It is a programming language designed for recursive zk-SNARKs and focuses on proof generation rather than program execution in a virtual machine environment.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="17-piecrust">17. [Piecrust]<a href="https://vac.dev/rlog/zkVM-explorations#17-piecrust" class="hash-link" aria-label="Direct link to 17. [Piecrust]" title="Direct link to 17. [Piecrust]">​</a></h2>
<ul>
<li>Overview: Piecrust [<a href="https://docs.rs/piecrust/latest/piecrust/" target="_blank" rel="noopener noreferrer">17</a>] is a WASM-based zkVM designed to run on the Dusk Network. It supports concurrent execution and focuses on providing privacy and scalability for smart contracts.</li>
<li>Main focus: Piecrust is designed to provide private and efficient execution of smart contracts through the use of ZKPs.</li>
<li>Privacy: Piecrust supports ZK-friendly computations and enhances privacy through cryptographic primitives such as Merkle trees.</li>
<li>Performance: Piecrust is designed to be scalable and concurrent, allowing multiple sessions to run simultaneously, which improves overall performance.</li>
<li>Integration: Piecrust integrates with the Dusk Network and supports private smart contracts, making it ideal for dapps.</li>
<li>Conclusion: Yes, Piecrust qualifies as a zkVM, offering scalability, privacy, and support for succinct proof generation.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="18-ceno">18. [Ceno]<a href="https://vac.dev/rlog/zkVM-explorations#18-ceno" class="hash-link" aria-label="Direct link to 18. [Ceno]" title="Direct link to 18. [Ceno]">​</a></h2>
<ul>
<li>Overview: Ceno [<a href="https://eprint.iacr.org/2024/387" target="_blank" rel="noopener noreferrer">18</a>] is a zkVM that provides a theoretical framework for reducing proving time by grouping common portions of code together. It uses recursive proofs to enhance prover efficiency.</li>
<li>Main focus: Ceno aims to optimize prover performance through recursive proofs, making it a powerful tool for handling complex computations efficiently.</li>
<li>Privacy: Ceno supports zero-knowledge privacy through recursive proofs and is designed to handle large-scale computations securely.</li>
<li>Performance: Ceno's recursive proof framework ensures that it can efficiently prove the execution of programs, reducing the time required for proof generation.</li>
<li>Integration: Ceno can be integrated into systems that require high efficiency and privacy, particularly those handling complex, repeated computations.</li>
<li>Conclusion: Yes, Ceno qualifies as a zkVM, providing efficient and private computation through the use of recursive proofs.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="19-stellar">19. [Stellar]<a href="https://vac.dev/rlog/zkVM-explorations#19-stellar" class="hash-link" aria-label="Direct link to 19. [Stellar]" title="Direct link to 19. [Stellar]">​</a></h2>
<ul>
<li>Overview: Stellar [<a href="https://stellar.org/blog/developers/zkvm-a-new-design-for-fast-confidential-smart-contracts" target="_blank" rel="noopener noreferrer">19</a>] is a decentralized protocol designed to facilitate cross-border transactions between digital and fiat currencies.</li>
<li>Main focus: Stellar's primary goal is to improve financial transactions by enabling decentralized, low-cost currency transfers. It does not aim to provide ZKPs or run verifiable computations
like a zkVM.</li>
<li>Privacy: Stellar focuses on confidentiality and security for financial transactions, but it does not employ ZKPs in the way zkVMs do for verifying computation without revealing data.</li>
<li>Performance: Stellar prioritizes the performance of financial transactions, ensuring low latency and high throughput across its decentralized network.
However, this performance focus is specific to transactions rather than general-purpose program execution.</li>
<li>Integration: Stellar is designed for integration with financial systems, enabling currency conversions and transfers, but it is not built for executing smart contracts or verifiable computations.</li>
<li>Conclusion: No, Stellar is not a zkVM. It is a decentralized financial protocol focused on facilitating cross-border payments rather than verifiable or privacy-preserving computation.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="20-novanet">20. [NovaNet]<a href="https://vac.dev/rlog/zkVM-explorations#20-novanet" class="hash-link" aria-label="Direct link to 20. [NovaNet]" title="Direct link to 20. [NovaNet]">​</a></h2>
<ul>
<li>Overview: NovaNet [<a href="https://www.novanet.xyz/blog" target="_blank" rel="noopener noreferrer">20</a>] is an open peer-to-peer network that aims to build upon concepts of non-uniform incremental verifiable computation.</li>
<li>Main focus: NovaNet's focus is on peer-to-peer networking and decentralized computing rather than on proving the execution of programs in a zero-knowledge manner.</li>
<li>Privacy: NovaNet does not provide ZKPs or privacy features typically associated with zkVMs. Its focus is on decentralized networking and computation.</li>
<li>Performance: NovaNet prioritizes efficient decentralized computation but does not focus on privacy or performance benchmarks related to ZKPs.</li>
<li>Integration: NovaNet is built for decentralized networks but is not designed to integrate with systems requiring verifiable computation or ZKP generation.</li>
<li>Conclusion: No, NovaNet is not a zkVM. It is a decentralized peer-to-peer network focused on distributed computing rather than zero-knowledge computation.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="21-zkllvm">21. [ZkLLVM]<a href="https://vac.dev/rlog/zkVM-explorations#21-zkllvm" class="hash-link" aria-label="Direct link to 21. [ZkLLVM]" title="Direct link to 21. [ZkLLVM]">​</a></h2>
<ul>
<li>Overview: ZkLLVM [<a href="https://github.com/NilFoundation/zkLLVM" target="_blank" rel="noopener noreferrer">21</a>] is a compiler that transforms C++ or Rust code into circuits for use in zk-SNARK or zk-STARK systems. Its primary purpose is to bridge high-level programming
languages with ZKP systems by compiling code into arithmetic circuits that can be used to generate and verify proofs.</li>
<li>Main focus: ZkLLVM focuses on making ZKPs accessible to developers by enabling them to write code in familiar languages (C++, Rust) and then compile that code into ZK circuits.</li>
<li>Privacy: ZkLLVM enables the generation of ZKPs by compiling high-level code into ZK-compatible circuits. It plays a crucial role in privacy-preserving applications but does not act
as a zkVM itself.</li>
<li>Performance: ZkLLVM allows for the performance of ZKPs to be closely tied to the complexity of the compiled circuits. The performance depends on the underlying
zk-SNARK or zk-STARK system used.</li>
<li>Integration: ZkLLVM integrates with zk-SNARK and zk-STARK proof systems, making it useful for a variety of privacy-focused applications, but it does not serve as a zkVM
for general-purpose computation.</li>
<li>Conclusion: No, zkLLVM is not a zkVM. It is a compiler that transforms high-level code into ZK circuits, enabling ZKPs but not acting as a virtual machine for executing and proving programs.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="22-zkmove">22. [ZkMove]<a href="https://vac.dev/rlog/zkVM-explorations#22-zkmove" class="hash-link" aria-label="Direct link to 22. [ZkMove]" title="Direct link to 22. [ZkMove]">​</a></h2>
<ul>
<li>Overview: ZkMove [<a href="https://www.zkmove.net/2023-06-20-zkMove-0.2.0-Achieving-Full-Bytecode-Compatibility-with-Move/" target="_blank" rel="noopener noreferrer">22</a>] is a zkVM designed to execute smart contracts written in the Move language. It utilizes ZKPs to ensure that the execution of these contracts remains verifiable and secure.</li>
<li>Main focus: ZkMove focuses on privacy and verifiable execution for Move-based smart contracts, providing a framework for ZK-friendly computation.</li>
<li>Privacy: ZkMove ensures that smart contract execution remains private through ZKPs, making it suitable for privacy-preserving applications.</li>
<li>Performance: ZkMove is optimized for verifiable execution, ensuring that contracts can be proven correct while preserving privacy.</li>
<li>Integration: ZkMove integrates well with systems that use the Move language, particularly in environments that require private smart contract execution.</li>
<li>Conclusion: Yes, zkMove qualifies as a zkVM, offering ZK-friendly execution and privacy for smart contracts written in the Move language.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="23-o1vm">23. [O1VM]<a href="https://vac.dev/rlog/zkVM-explorations#23-o1vm" class="hash-link" aria-label="Direct link to 23. [O1VM]" title="Direct link to 23. [O1VM]">​</a></h2>
<ul>
<li>Overview: O1VM [<a href="https://github.com/o1-labs/proof-systems/tree/master/o1vm" target="_blank" rel="noopener noreferrer">23</a>] is a general-purpose zkVM developed by o1Labs. It is designed to prove the execution of MIPS programs efficiently through a combination of zk-SNARKs
and specialized techniques like folding schemes and RAMLookups.</li>
<li>Main focus: O1VM focuses on scalability and verifiable computation for MIPS-based programs, making it a strong contender for executing and proving complex programs efficiently.</li>
<li>Privacy: O1VM ensures privacy through zk-SNARK proofs, keeping the details of the computation private while proving its correctness.</li>
<li>Performance: O1VM is optimized for handling long execution traces and complex computations, making it highly scalable.</li>
<li>Integration: O1VM integrates well with MIPS-based architectures and systems that require privacy-preserving computation.</li>
<li>Conclusion: Yes, o1VM qualifies as a zkVM, providing privacy, scalability, and strong proof generation for MIPS programs.</li>
</ul>
<h1>Summary of findings</h1>
<table><thead><tr><th>Project name</th><th>ZkVM status</th><th>Zero knowledge</th><th>Reasoning/comments</th></tr></thead><tbody><tr><td><strong>SP1</strong></td><td>Yes</td><td>No</td><td>Proves execution of LLVM-based programs but lacks privacy features.</td></tr><tr><td><strong>Nexus</strong></td><td>Yes</td><td>No</td><td>Strong proof generation but lacks zero-knowledge privacy due to Spartan.</td></tr><tr><td><strong>Risc0</strong></td><td>Yes</td><td>Yes</td><td>Supports full ZKP generation for Rust programs.</td></tr><tr><td><strong>Powdr</strong></td><td>No</td><td>Yes</td><td>Toolkit for creating custom zkVMs, not a zkVM itself.</td></tr><tr><td><strong>ZkMIPS</strong></td><td>Yes</td><td>Yes</td><td>Supports MIPS-like architecture with full zero-knowledge and proof generation.</td></tr><tr><td><strong>Valida</strong></td><td>Yes</td><td>No</td><td>Performance-focused zkVM, lacks privacy guarantees.</td></tr><tr><td><strong>Jolt</strong></td><td>Yes</td><td>No</td><td>Performance-focused zkVM, does not achieve zero-knowledge privacy.</td></tr><tr><td><strong>ZkWASM</strong></td><td>Yes</td><td>Yes</td><td>Full zero-knowledge and verifiable execution of WebAssembly code.</td></tr><tr><td><strong>Aleo</strong></td><td>Yes</td><td>Yes</td><td>Fully private and scalable dapps.</td></tr><tr><td><strong>Ola</strong></td><td>No</td><td>No</td><td>Primarily a ZK-rollup platform, not a zkVM, focusing on scalability and performance rather than privacy.</td></tr><tr><td><strong>Miden</strong></td><td>Yes</td><td>Yes</td><td>Zk-STARK-based zkVM with strong privacy and scalability.</td></tr><tr><td><strong>ZkOS</strong></td><td>No</td><td>No</td><td>Verifiable operating system focused on zkApps, not a zkVM.</td></tr><tr><td><strong>Triton</strong></td><td>No</td><td>No</td><td>Optimizes GPU workloads but not designed for ZKPs.</td></tr><tr><td><strong>Cairo</strong></td><td>Yes</td><td>ZK-friendly</td><td>Custom Rust-based language with zk-STARK proof generation.</td></tr><tr><td><strong>SnarkOS</strong></td><td>No</td><td>Yes</td><td>Decentralized OS for Aleo's network, focuses on consensus rather than verifiable computation.</td></tr><tr><td><strong>Lurk</strong></td><td>No</td><td>No</td><td>Programming language for recursive zk-SNARKs, not a zkVM.</td></tr><tr><td><strong>Piecrust</strong></td><td>Yes</td><td>ZK-friendly</td><td>ZkVM with recursive SNARK capabilities, focused on succinct proof generation.</td></tr><tr><td><strong>Ceno</strong></td><td>Yes</td><td>Yes</td><td>Theoretical zkVM improving prover efficiency through recursive proofs.</td></tr><tr><td><strong>Stellar</strong></td><td>No</td><td>No</td><td>Focuses on cross-border transactions, not ZK-proof generation or verifiable computation.</td></tr><tr><td><strong>NovaNet</strong></td><td>No</td><td>No</td><td>Peer-to-peer network focused on distributed computing, not zero-knowledge computation.</td></tr><tr><td><strong>ZkLLVM</strong></td><td>No</td><td>Yes, in some cases</td><td>Compiler for generating ZK-circuits, not a zkVM.</td></tr><tr><td><strong>ZkMove</strong></td><td>Yes</td><td>ZK-friendly</td><td>ZkVM supporting Move language with ZKP execution.</td></tr><tr><td><strong>O1VM</strong></td><td>Yes</td><td>Yes</td><td>MIPS-based zkVM with strong privacy, scalability, and proof generation.</td></tr></tbody></table>
<h1>Insights and conclusions</h1>
<p>Our analysis reveals that many of the projects labeled as zkVMs do meet the core criteria for zkVMs, offering verifiable computation and proof generation
as foundational features. However, a number of these projects fall short of delivering full zero-knowledge privacy. Projects like Risc0, Aleo, and Miden stand out as leading zkVM frameworks
that balance proof generation, privacy, and scalability, offering strong platforms for developers seeking to build privacy-preserving applications.</p>
<p>Conversely, projects like SP1 and Nexus excel in generating verifiable proofs but currently lack comprehensive zero-knowledge privacy mechanisms. These platforms are excellent for
scenarios where proof generation and scalability are paramount, but privacy is not a primary concern.</p>
<p>As zkVM technology continues to evolve, we expect to see more projects integrating enhanced privacy-preserving mechanisms while simultaneously improving performance and scalability.
This ongoing development will likely broaden the application of zkVMs across the blockchain ecosystem, particularly in privacy-sensitive sectors such as finance, data security,
and decentralized applications.</p>
<p>What are your thoughts on our zkVM analysis? Do you agree with our findings, or do you know of other zkVM projects that should be on our radar? We would love to hear your insights, questions,
or suggestions! Feel free to join the <a href="https://forum.vac.dev/t/exploring-zkvms-which-projects-truly-qualify-as-zero-knowledge-virtual-machines/317" target="_blank" rel="noopener noreferrer">discussion</a> on our forum.</p>
<h1>References</h1>
<p>[1] Introducing SP1: A performant, 100% open-source, contributor-friendly zkVM. Retrieved from <a href="https://blog.succinct.xyz/introducing-sp1/" target="_blank" rel="noopener noreferrer">https://blog.succinct.xyz/introducing-sp1/</a></p>
<p>[2] The Nexus 2.0 zkVM. Retrieved from <a href="https://docs.nexus.xyz/" target="_blank" rel="noopener noreferrer">https://docs.nexus.xyz/</a></p>
<p>[3] The first general purpose zkVM. Retrieved from <a href="https://www.risczero.com/zkvm" target="_blank" rel="noopener noreferrer">https://www.risczero.com/zkvm</a></p>
<p>[4] Powdr. Retrieved from <a href="https://docs.powdr.org/" target="_blank" rel="noopener noreferrer">https://docs.powdr.org/</a></p>
<p>[5] ZKM Architecture. Retrieved from <a href="https://docs.zkm.io/zkm-architecture" target="_blank" rel="noopener noreferrer">https://docs.zkm.io/zkm-architecture</a></p>
<p>[6] Valida zkVM Design. Retrieved from <a href="https://delendum.xyz/writings/2023-05-10-zkvm-design.html" target="_blank" rel="noopener noreferrer">https://delendum.xyz/writings/2023-05-10-zkvm-design.html</a></p>
<p>[7] Building Jolt: A fast, easy-to-use zkVM. Retrieved from <a href="https://a16zcrypto.com/posts/article/building-jolt/" target="_blank" rel="noopener noreferrer">https://a16zcrypto.com/posts/article/building-jolt/</a></p>
<p>[8] ZK-WASM. Retrieved from <a href="https://delphinuslab.com/zk-wasm/" target="_blank" rel="noopener noreferrer">https://delphinuslab.com/zk-wasm/</a></p>
<p>[9] Aleo. Retrieved from <a href="https://aleo.org/blog/" target="_blank" rel="noopener noreferrer">https://aleo.org/blog/</a></p>
<p>[10] OlaVM Whitepaper V2. Retrieved from <a href="https://github.com/Sin7Y/olavm-whitepaper-v2/tree/master" target="_blank" rel="noopener noreferrer">https://github.com/Sin7Y/olavm-whitepaper-v2/tree/master</a></p>
<p>[11] Polygon Miden VM. Retrieved from <a href="https://0xpolygonmiden.github.io/miden-vm/intro/main.html" target="_blank" rel="noopener noreferrer">https://0xpolygonmiden.github.io/miden-vm/intro/main.html</a></p>
<p>[12] The Adventures of OS: Making a RISC-V Operating System using Rust. Retrieved from <a href="https://osblog.stephenmarz.com/index.html" target="_blank" rel="noopener noreferrer">https://osblog.stephenmarz.com/index.html</a></p>
<p>[13] Triton VM. Retrieved from <a href="https://triton-vm.org/spec/" target="_blank" rel="noopener noreferrer">https://triton-vm.org/spec/</a></p>
<p>[14] How does the original Cairo VM work?. Retrieved from <a href="https://github.com/lambdaclass/cairo-vm/blob/main/docs/python_vm/README.md" target="_blank" rel="noopener noreferrer">https://github.com/lambdaclass/cairo-vm/blob/main/docs/python_vm/README.md</a></p>
<p>[15] Aleo completes security audits of snarkOS &amp; snarkVM. Retrieved from <a href="https://aleo.org/post/aleo-completes-security-audits-of-snarkos-and-snarkvm/" target="_blank" rel="noopener noreferrer">https://aleo.org/post/aleo-completes-security-audits-of-snarkos-and-snarkvm/</a></p>
<p>[16] Lurk zkVM. Retrieved from <a href="https://github.com/lurk-lab" target="_blank" rel="noopener noreferrer">https://github.com/lurk-lab</a></p>
<p>[17] Piecrust VM. Retrieved from <a href="https://docs.rs/piecrust/latest/piecrust/" target="_blank" rel="noopener noreferrer">https://docs.rs/piecrust/latest/piecrust/</a></p>
<p>[18] Ceno: Non-uniform, Segment and Parallel Zero-knowledge Virtual Machine. Retrieved from <a href="https://eprint.iacr.org/2024/387" target="_blank" rel="noopener noreferrer">https://eprint.iacr.org/2024/387</a></p>
<p>[19] ZkVM: a new design for fast, confidential smart contracts. Retrieved from <a href="https://stellar.org/blog/developers/zkvm-a-new-design-for-fast-confidential-smart-contracts" target="_blank" rel="noopener noreferrer">https://stellar.org/blog/developers/zkvm-a-new-design-for-fast-confidential-smart-contracts</a></p>
<p>[20] Novanet. Retrieved from <a href="https://www.novanet.xyz/blog" target="_blank" rel="noopener noreferrer">https://www.novanet.xyz/blog</a></p>
<p>[21] ZKLLVM. Retrieved from <a href="https://github.com/NilFoundation/zkLLVM" target="_blank" rel="noopener noreferrer">https://github.com/NilFoundation/zkLLVM</a></p>
<p>[22] zkMove 0.2.0 - Achieving Full Bytecode Compatibility with Move. Retrieved from <a href="https://www.zkmove.net/2023-06-20-zkMove-0.2.0-Achieving-Full-Bytecode-Compatibility-with-Move/" target="_blank" rel="noopener noreferrer">https://www.zkmove.net/2023-06-20-zkMove-0.2.0-Achieving-Full-Bytecode-Compatibility-with-Move/</a></p>
<p>[23] O1VM. Retrieved from <a href="https://github.com/o1-labs/proof-systems/tree/master/o1vm" target="_blank" rel="noopener noreferrer">https://github.com/o1-labs/proof-systems/tree/master/o1vm</a></p>]]></content>
<author>
<name>Moudy</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Nescience: A User-Centric State-Separation Architecture]]></title>
<id>https://vac.dev/rlog/Nescience-state-separation-architecture</id>
<link href="https://vac.dev/rlog/Nescience-state-separation-architecture"/>
<updated>2024-08-23T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Nescience: A user-centric state-separation architecture.]]></summary>
<content type="html"><![CDATA[<p>Nescience: A user-centric state-separation architecture.</p>
<!-- -->
<p><em>Disclaimer: This content is a work in progress. Some components may be updated, changed, or expanded as new research findings become available.</em></p>
<p>In blockchain applications, privacy settings are typically predefined by developers, leaving users with limited control. This traditional,
one-size-fits-all approach often leads to inefficiencies and potential privacy concerns as it fails to cater to the diverse needs of individual users.
The Nescience state-separation architecture (NSSA) aims to address these issues by shifting privacy control from developers to users. NSSA introduces a flexible,
user-centric approach that allows for customized privacy settings to better meet individual needs. This blog post will delve into the details of NSSA,
including its different execution types, cryptographic foundations, and unique challenges.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introducing-nssa-a-user-centric-approach">Introducing NSSA: A user-centric approach<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#introducing-nssa-a-user-centric-approach" class="hash-link" aria-label="Direct link to Introducing NSSA: A user-centric approach" title="Direct link to Introducing NSSA: A user-centric approach">​</a></h2>
<p>NSSA gives users control over their privacy settings by introducing <em>shielded</em> (which creates a layer of privacy for the outputs, and only the necessary details are shared)
and <em>deshielded</em> (which reveal private details, making them publicly visible) execution types in addition to the traditional public and private modes. This flexibility allows
users to customize their privacy settings to match their unique needs, whether they require high levels of confidentiality or more transparency. In NSSA, the system is divided
into two states: public and private. The public state uses an account-based model while the private state employs a UTXO-based (unspent transaction output) model. Private executions within NSSA utilize
UTXO exchanges, ensuring that transaction details remain confidential. The sequencer verifies these exchanges without accessing specific details, enhancing privacy by unlinking
sender and receiver identities. Zero-knowledge proofs (ZKPs) allow users to prove transaction validity without revealing data, maintaining the integrity and confidentiality of
private transactions. UTXOs contain assets such as balances, NFTs, or private storage data, and are stored in plaintext within Sparse Merkle trees (SMTs) in the private state and
as hashes in the public state. This dual-storage approach keeps UTXO details confidential while allowing public verification through hashes, achieving a balance between privacy and transparency.</p>
<p>Implementing NSSA introduces unique challenges, particularly in cryptographic implementation and maintaining the integrity of private executions. These challenges are addressed
through various solutions such as ZKPs, which ensure transaction validity without compromising privacy, and the dual-storage approach, which maintains confidentiality while enabling
public verification. By allowing users to customize their privacy settings, NSSA enhances user experience and promotes wider adoption of private execution platforms. As we move towards
a future where user-empowered privacy control is crucial, NSSA provides a flexible and user-centric solution that meets the diverse needs of blockchain users.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="why-nssa-differs-from-other-hybrid-execution-platforms">Why NSSA differs from other hybrid execution platforms<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#why-nssa-differs-from-other-hybrid-execution-platforms" class="hash-link" aria-label="Direct link to Why NSSA differs from other hybrid execution platforms" title="Direct link to Why NSSA differs from other hybrid execution platforms">​</a></h2>
<p>In many existing hybrid execution platforms, privacy settings are predefined by developers, often applying a one-size-fits-all approach that does not accommodate the
diverse privacy needs of users. These platforms blend public and private states, but control over privacy remains with the application developers.
While this approach is straightforward for developers (who bear the responsibility for any potential privacy leaks), it leaves users with no control over their own privacy settings.
This rigidity becomes problematic as user needs evolve over time, or as new regulations necessitate changes to privacy configurations. In such cases,
updates to decentralized applications are required to adjust privacy settings, which can disrupt the user experience and create friction.</p>
<p>NSSA addresses these limitations by introducing a groundbreaking concept: <strong>selective privacy</strong>. Unlike traditional platforms where privacy
is static and determined by developers, selective privacy empowers users to dynamically choose their own privacy levels based on their unique needs and sensitivity.
This flexibility is critical in a decentralized ecosystem where the diversity of users and use cases demands a more adaptable privacy solution.</p>
<p>In the NSSA model, users have the autonomy to select how they interact with decentralized applications (dapps) by choosing from four types of transaction executions: <strong>public</strong>,
<strong>private</strong>, <strong>shielded</strong>, and <strong>deshielded</strong>. This model allows users to tailor their privacy settings on a per-transaction basis, selecting the most appropriate execution type for each
specific interaction. For instance, a user concerned about data confidentiality might opt for a fully private transaction while another user, wary of privacy but seeking transparency,
might choose a public execution.</p>
<p>While selective privacy may appear complex, especially for users who are not technically inclined, Nescience mitigates this by allowing the community or developers to
establish best practices and recommended approaches. These guidelines provide users with an informed starting point, and over time, users can adjust their privacy
settings as their preferences and trust in the platform evolve. Importantly, selective privacy gives users the right to alter their privacy level at any point in the future,
ensuring that their privacy settings remain aligned with their needs as they change.</p>
<p>This approach not only empowers users but also facilitates greater adoption of dapps. Users who are skeptical about privacy concerns can initially engage with transparent
transactions and gradually shift towards more private executions as they gain confidence in the system and vice versa for users who start with privacy but later find transparency
beneficial for certain transactions. In this way, selective privacy bridges the gap between privacy and transparency, allowing for an optimal balance to emerge from the community’s
collective preferences.</p>
<p>To liken this to open-source projects: in traditional systems, developers fix privacy rules much like immutable code—users must comply with these fixed settings.
In contrast, with selective privacy, the rules are malleable and shaped by the users’ preferences, enabling the community to find the ideal balance between privacy and efficiency over time.</p>
<p>NSSA is distinct from traditional zero-knowledge (ZK) rollups in several key ways. One of the unique features of NSSA is its <strong>public execution type</strong>, which does not
require ZKPs or a zero-knowledge virtual machine (zkVM). This provides a significant advantage in terms of scalability and efficiency as users can choose public executions for
transactions that do not require enhanced privacy, avoiding the overhead associated with ZKP generation and verification.</p>
<p>Moreover, NSSA introduces two additional execution types—<strong>shielded and deshielded</strong>—which further distinguish it from traditional privacy-preserving rollups.
These execution types allow for more nuanced control over privacy, giving users the ability to shield certain aspects of a transaction while deshielding others.
This flexibility sets NSSA apart as a more adaptable and user-centric platform, catering to a wide range of privacy needs without imposing a one-size-fits-all solution.</p>
<p>By combining selective privacy with a flexible execution model, NSSA offers a more robust and adaptable framework for decentralized applications,
ensuring that users maintain control over their privacy while benefiting from the security and efficiency of blockchain technology.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="how-nescience-state-separation-architecture-can-be-used">How Nescience state-separation architecture can be used<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#how-nescience-state-separation-architecture-can-be-used" class="hash-link" aria-label="Direct link to How Nescience state-separation architecture can be used" title="Direct link to How Nescience state-separation architecture can be used">​</a></h2>
<p>NSSA offers a flexible, privacy-preserving add-on that can be applied to existing dapps.
One of the emerging trends in the blockchain space is that each dapp is expected to have its own rollup for efficiency, and it is estimated that Ethereum could see
the deployment of different rollups in the near future. A key question arises: how many of these rollups will incorporate privacy? For dapp developers who want to offer flexible,
user-centric privacy features, NSSA provides a solution through selective privacy.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="use-case-adding-privacy-to-existing-dapps">Use case: Adding privacy to existing dapps<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#use-case-adding-privacy-to-existing-dapps" class="hash-link" aria-label="Direct link to Use case: Adding privacy to existing dapps" title="Direct link to Use case: Adding privacy to existing dapps">​</a></h3>
<p>Consider a dapp running on a transparent network that offers no inherent privacy to its users. Converting this dapp to a privacy-preserving architecture from scratch would
require significant effort, restructuring, and a deep understanding of cryptographic frameworks. However, with NSSA, the dapp does not need to undergo extensive changes.
Instead, the <strong>Nescience state-separation model</strong> can be deployed as an <strong>add-on</strong>, offering selective privacy as an option for the dapp’s users.</p>
<p>This allows the dapp to retain its existing functionality while providing users with a choice between the traditional, transparent version and a new version with selective privacy features.
With NSSA, the privacy settings are flexible, meaning users can tailor their level of privacy according to their individual needs while the dapp operates on its current infrastructure.
This contrasts sharply with the typical approach, where dapps are either entirely transparent or fully private, with no flexibility for users to select their own privacy preferences.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="key-advantage-decoupling-from-the-host-chain">Key advantage: Decoupling from the host chain<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#key-advantage-decoupling-from-the-host-chain" class="hash-link" aria-label="Direct link to Key advantage: Decoupling from the host chain" title="Direct link to Key advantage: Decoupling from the host chain">​</a></h3>
<p>A key feature of NSSA is that it operates independently of the privacy characteristics of the host blockchain. Whether the host chain is fully transparent or fully private,
the Nescience state-separation architecture can be deployed on top of it, offering users the ability to choose their own privacy settings.
This decoupling from the host chain’s inherent privacy model is critical as it allows users to benefit from selective privacy even in environments that were not originally designed to offer it.</p>
<p>In <strong>fully private chains</strong>, NSSA allows users to selectively reveal transaction details when compliance with regulations or other requirements is necessary.
In <strong>fully transparent chains</strong>, NSSA allows users to maintain privacy for specific transactions, offering flexibility that would not otherwise be possible.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h3>
<p>NSSA provides a powerful tool for dapp developers who want to offer <strong>selective privacy</strong> to their users without the need for a complete overhaul of their existing systems.
By deploying NSSA as an add-on, dapps can give users the ability to choose their own privacy settings whether they are operating on
transparent or private blockchains. This flexibility makes NSSA a valuable option for any dapp looking to provide enhanced privacy options while maintaining efficiency and ease of use.</p>
<h1>B. Design</h1>
<p>In this section, we will delve into the core design components of the Nescience state-separation architecture, covering its key structural elements and the mechanisms
that drive its functionality. We will explore the following topics:</p>
<ol>
<li>
<p><strong>Architecture's components</strong>: An in-depth look at the foundational building blocks of NSSA, including the public and private states, UTXO structures, zkVM, and smart contracts.
These components work together to facilitate secure, flexible, and scalable transactions within the architecture.</p>
</li>
<li>
<p><strong>General execution overview</strong>: We will outline the overall flow of transaction execution within NSSA, describing how users interact with the system and how the architecture
supports various types of executions—public, private, shielded, and deshielded—while preserving privacy and efficiency.</p>
</li>
<li>
<p><strong>Execution processes and UTXO management</strong>: This section will focus on the lifecycle of UTXOs within the architecture, from their generation to consumption.
We will also cover the processes involved in managing UTXOs, including proof generation, state transitions, and ensuring transaction validity.</p>
</li>
</ol>
<p>These topics will provide a comprehensive understanding of how NSSA enables flexible and secure interactions within dapps.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-architectures-components">1. Architecture's components<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#1-architectures-components" class="hash-link" aria-label="Direct link to 1. Architecture's components" title="Direct link to 1. Architecture's components">​</a></h2>
<hr>
<p>NSSA introduces an advanced prototype execution framework designed to enhance privacy and security in blockchain applications.
This framework integrates several essential components: the public state, private state, zkVM, various execution types, Nescience users, and smart contracts.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-public-state">a) Public state<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-public-state" class="hash-link" aria-label="Direct link to a) Public state" title="Direct link to a) Public state">​</a></h3>
<hr>
<p>The public state in the NSSA is a fundamental component designed to hold all publicly accessible information within
the blockchain network. This state is organized as a single Merkle tree structure, a sophisticated data structure that ensures efficient and secure data verification.
The public state includes critical information such as user balances and the public storage data of smart contracts.</p>
<p>In an account-based model, the public state operates by storing each account or smart contract's public data as individual leaf nodes within the Merkle tree.
When transactions occur, they directly modify the state by updating these leaf nodes. This direct modification ensures that the most current state of the network
is always reflected accurately.</p>
<p>The Merkle tree structure is essential for maintaining data integrity. Each leaf node contains a hash of a data block, and each non-leaf node contains the
hash of its child nodes. This hierarchical arrangement means that any change in the data will result in a change in the corresponding hash, making it easy to detect
any tampering. The root hash, or Merkle root, is stored on the blockchain, providing a cryptographic guarantee of the data's integrity. This root hash serves as a single,
concise representation of the entire state, enabling quick and reliable verification by any network participant.</p>
<p>Transparency is a key feature of the public state. All data stored within this state is openly accessible and verifiable by any participant in the network.
This openness ensures that all transactions and state changes are visible and auditable, fostering trust and accountability. For example, user balances are
publicly viewable, which helps ensure transparency and trust in the system. Similarly, public smart contract storage can be accessed and verified by anyone,
making it suitable for applications that require public scrutiny and auditability, such as public record updates and some financial transactions.</p>
<p>The workflow of managing the public state involves several steps to ensure data integrity and transparency. When a user initiates a transaction involving public data,
the relevant changes are proposed and applied to the public state tree. The transaction details, such as transferring funds between accounts or updating smart contract storage,
update the corresponding leaf nodes in the Merkle tree. Following this, the hashes of the affected nodes are recalculated up to the root, ensuring that the entire tree
accurately reflects the new state of the network. The updated Merkle root is then recorded on the blockchain, allowing all network participants to verify the integrity
of the public state. Any discrepancy in the data will result in a mismatched root hash, signaling potential tampering or errors.</p>
<p>In summary, the public state in NSSA leverages the robustness of the Merkle tree structure to provide a secure, transparent, and verifiable environment for publicly
accessible information. By operating on an account-based model and maintaining rigorous data integrity checks, the public state ensures that all transactions are
transparent and trustworthy, laying a strong foundation for a reliable blockchain network.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-private-state">b) Private state<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#b-private-state" class="hash-link" aria-label="Direct link to b) Private state" title="Direct link to b) Private state">​</a></h3>
<hr>
<p>The private state in the NSSA is a sophisticated system designed to maintain user privacy while ensuring transaction integrity.
Each user has their own individual Merkle tree, which holds their private information such as balances and storage data. This structure is distinct from the public state,
which uses an account-based model. Instead, the private state employs a UTXO-based model. In this model, each transaction output is a discrete
unit that can be independently spent in future transactions. This design provides users with granular control over their transaction outputs.</p>
<p>A key aspect of maintaining privacy within the private state is the use of ZKPs. ZKPs allow transactions to be validated without revealing any
underlying private data. This means that while the system can verify that a transaction is legitimate, the details of the transaction remain confidential. Only parties
with the appropriate viewing key can access and reconstruct the user’s list of UTXOs, ensuring that sensitive information is protected.</p>
<p>The private state also employs a dual-storage approach to balance privacy and transparency. UTXOs are stored in plaintext within SMTs in the private state,
providing detailed and accessible records for the user. In contrast, the public state only holds hashes of these UTXOs. This method ensures that while the public can verify
the existence and integrity of private transactions through these hashes, they cannot access the specific details.</p>
<p>The workflow for a transaction in the private state begins with the user initiating a transaction involving their private data, such as transferring a private balance or
updating private smart contract storage. The transaction involves spending existing UTXOs, represented as leaves in the Merkle tree, and creating new UTXOs,
which are then appended to the user’s private list. The zkVM generates a ZKP to validate the transaction without revealing
any private data, ensuring the transaction adheres to the system's rules.</p>
<p>Once the proof is generated, it is submitted to the sequencer, which verifies the transaction’s validity. Upon successful verification, the nullifier is added to the nullifier set,
preventing double spending of the same UTXO. The use of ZKPs and nullifiers ensures that the private state maintains both security and privacy.</p>
<p>In summary, the private state in NSSA is meticulously designed to provide users with control over their private information while ensuring the security and integrity of transactions.
By utilizing a UTXO-based model, individual Merkle trees, ZKPs, and a dual-storage system, NSSA achieves a balance between confidentiality and verifiability,
making it a robust solution for managing private blockchain transactions.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="c-zkvm-zero-knowledge-virtual-machine">c) ZkVM (zero-knowledge virtual machine)<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#c-zkvm-zero-knowledge-virtual-machine" class="hash-link" aria-label="Direct link to c) ZkVM (zero-knowledge virtual machine)" title="Direct link to c) ZkVM (zero-knowledge virtual machine)">​</a></h3>
<hr>
<p>The zkVM is a pivotal component in NSSA, designed to uphold the highest standards
of privacy and security in blockchain transactions. Its primary function is to generate and aggregate ZKPs, enabling users to validate the
correctness of their transactions without disclosing any underlying details. This capability is crucial for maintaining the confidentiality and integrity of sensitive
data within the blockchain network.</p>
<p>ZKPs are sophisticated cryptographic protocols that allow one party, the prover, to convince another party, the verifier, that a certain statement is true,
without revealing any information beyond the validity of the statement itself. In the context of the zkVM, this means users can prove their transactions are valid without
exposing transaction specifics, such as amounts or parties involved. This process is essential for transactions within the private state, where maintaining confidentiality is paramount.</p>
<p>The generation of ZKPs involves intricate cryptographic computations. When a user initiates a transaction, the zkVM processes the transaction inputs and produces a proof
that the transaction adheres to the protocol's rules. This proof must be robust enough to convince the verifier of the transaction's validity while preserving the privacy
of the transaction details.</p>
<p>Performance optimization is another critical function of the zkVM. In a typical blockchain scenario, verifying multiple individual proofs can be computationally intensive
and time consuming, potentially leading to network congestion and delays. To address this, the zkVM can aggregate multiple ZKPs into a single, consolidated proof.
This aggregation significantly reduces the verification overhead as the verifier needs to check only one comprehensive proof rather than multiple individual ones.
This efficiency is vital for maintaining high throughput and low latency in the blockchain network, ensuring that the system can handle a large volume of transactions swiftly and securely.</p>
<p>Furthermore, the zkVM's role extends beyond mere proof generation and aggregation. It also ensures that all transactions meet the required privacy and security standards
before they are recorded on the blockchain. By interacting seamlessly with other components such as the public and private states, the zkVM ensures that any transaction,
whether it involves public data, private data, or a mix of both, is thoroughly validated and secured.</p>
<p>In summary, the zkVM is essential for the NSSA, providing the cryptographic backbone necessary to support secure and private transactions. Its ability to generate and
aggregate ZKPs not only preserves the confidentiality of user data but also enhances the overall efficiency and scalability of the blockchain network.
By ensuring that all transactions are validated without revealing sensitive information, the zkVM upholds the integrity and trustworthiness of the Nescience blockchain system.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="d-execution-types-in-nssa">d) Execution types in NSSA<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#d-execution-types-in-nssa" class="hash-link" aria-label="Direct link to d) Execution types in NSSA" title="Direct link to d) Execution types in NSSA">​</a></h3>
<hr>
<p>NSSA incorporates multiple execution types to cater to varying levels of privacy and security requirements.
These execution types—public, private, shielded, and deshielded—are designed to provide users with flexible options for managing their transactions based on their specific privacy needs.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="1-public-executions">1. Public executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#1-public-executions" class="hash-link" aria-label="Direct link to 1. Public executions" title="Direct link to 1. Public executions">​</a></h4>
<p>Public executions are straightforward transactions that involve reading from and writing to the public state. In this model, data is openly accessible and verifiable
by all participants in the network. Public executions do not require ZKPs since transparency is the primary goal. This execution type is ideal
for non-sensitive transactions where public visibility is beneficial, such as updating public records, performing open financial transactions, or interacting with public smart contracts.</p>
<p>The workflow for a public execution starts with a user initiating a transaction that modifies public data. The transaction details are then used to update the relevant
leaf nodes in the Merkle tree. As changes are made, the hashes of affected nodes are recalculated up to the root, ensuring that the entire tree reflects the most recent state.
Finally, the updated Merkle root is recorded on the blockchain, making the new state publicly verifiable.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-private-executions">2. Private executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#2-private-executions" class="hash-link" aria-label="Direct link to 2. Private executions" title="Direct link to 2. Private executions">​</a></h4>
<p>Private executions are designed for confidential transactions, reading from and writing to the private state. These transactions require ZKPs to ensure that while the
transaction details are validated, the actual data remains private. This execution type is suitable for scenarios where privacy is crucial, such as private financial
transactions or sensitive data management within smart contracts.</p>
<p>In a private execution, the user initiates a transaction involving private data. The transaction spends existing UTXOs and creates new ones, all of which are represented as
leaves in the Merkle tree. The zkVM generates a ZKP to validate the transaction without revealing private data. This proof is submitted to the sequencer,
which verifies the proof to ensure the transaction's validity. Upon successful verification, the nullifier is added to the nullifier set, and the private state is updated
with the new Merkle root.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-shielded-executions">3. Shielded executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#3-shielded-executions" class="hash-link" aria-label="Direct link to 3. Shielded executions" title="Direct link to 3. Shielded executions">​</a></h4>
<p>Shielded executions create a layer of privacy for the outputs by allowing interactions between the public and private states. When a transaction occurs in a shielded execution,
details of the transaction are processed within the private state, ensuring that sensitive information remains confidential. Only the necessary details are shared with the public state,
often in a masked or encrypted form. This approach allows for the validation of the transaction without revealing critical data, thus preserving the privacy of the involved parties.</p>
<p>The workflow for shielded executions begins with the user initiating a transaction that reads from the public state and prepares to write to the private state. Public data is accessed,
and the private state is prepared to receive new data. The zkVM generates a ZKP to hide the receiver’s identity. This proof is submitted to the sequencer, which verifies
the proof to ensure the transaction's validity. If valid, the private state is updated with the new data while the public state reflects the change without revealing private details.
This type of execution is particularly useful for scenarios where the receiver’s identity needs to be hidden, such as in anonymous donation systems or confidential data storage.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="4-deshielded-executions">4. Deshielded executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#4-deshielded-executions" class="hash-link" aria-label="Direct link to 4. Deshielded executions" title="Direct link to 4. Deshielded executions">​</a></h4>
<p>Deshielded executions operate in the opposite manner of shielded executions, where data is read from the private state and written to the public state. This execution type is useful
in situations where the sender's identity needs to be kept confidential while making the transaction results publicly visible.</p>
<p>In a deshielded execution, the user initiates a transaction that reads from the private state and prepares to write to the public state. Private data is accessed,
and the transaction details are prepared. The zkVM generates a ZKP to hide the sender’s identity. This proof is then submitted to the sequencer,
which verifies the proof to ensure the transaction's validity. Once verified, the public state is updated with the new data, reflecting the change while keeping the sender’s
details confidential. This can be useful when transparency is needed, such as when auditing or proving certain aspects of a transaction to a wider audience.
By selectively deshielding certain transactions, users can control what information is shared publicly, thus maintaining a balance between privacy and transparency
as required by their specific use case.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="table-of-execution-types">Table of execution types<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#table-of-execution-types" class="hash-link" aria-label="Direct link to Table of execution types" title="Direct link to Table of execution types">​</a></h4>
<table><thead><tr><th>Type</th><th>Read from</th><th>Write to</th><th>ZKP required</th><th>Use case</th><th>Description</th></tr></thead><tbody><tr><td>Public</td><td>Public state</td><td>Public state</td><td>No</td><td>Non-sensitive transactions requiring transparency.</td><td>Ideal for transactions that do not require privacy, ensuring full transparency.</td></tr><tr><td>Private</td><td>Private state</td><td>Private state</td><td>Yes</td><td>Confidential transactions needing privacy.</td><td>Suitable for transactions that require confidentiality. Ensures that transaction details remain private through the use of ZKPs.</td></tr><tr><td>Shielded</td><td>Public state</td><td>Private state</td><td>Yes</td><td>Transactions where the receiver’s identity needs to be hidden.</td><td>Hides the identity of the receiver while keeping the transaction details private. Suitable for anonymous donations or confidential data storage.</td></tr><tr><td>Deshielded</td><td>Private state</td><td>Public state</td><td>Yes</td><td>Transactions where the sender’s identity needs to be hidden.</td><td>Ensures the sender’s identity remains confidential while making the transaction results public. Suitable for confidential disbursements or anonymized data publication.</td></tr></tbody></table>
<hr>
<p>By supporting a range of execution types, NSSA provides a flexible and robust framework for managing privacy and security in blockchain transactions.
Whether the need is for complete transparency, total privacy, or a balanced approach, NSSA's execution types allow users to select the level of confidentiality
that best fits their requirements. This flexibility enhances the overall utility of the blockchain, making it suitable for a wide array of applications and use cases.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="e-nescience-users">e) Nescience users<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#e-nescience-users" class="hash-link" aria-label="Direct link to e) Nescience users" title="Direct link to e) Nescience users">​</a></h3>
<hr>
<p>Nescience users are integral to the architecture, managing balances and assets within the blockchain network and invoking smart contracts with various privacy options.
They can choose the appropriate execution type—public, private, shielded, or deshielded—based on their specific privacy and security needs.</p>
<p>Users handle both public and private balances. Public balances are visible to all network participants and suitable for non-sensitive transactions,
while private balances are confidential and used for transactions requiring privacy. Digital wallets provide a user-friendly interface for managing
these balances, assets, and transactions, allowing users to select the desired execution type seamlessly.</p>
<p>Security is ensured through the use of cryptographic keys, which authenticate and verify transactions. ZKPs maintain privacy
by validating transaction correctness without revealing underlying data, ensuring sensitive information remains confidential even during verification.</p>
<p>The workflow for users involves initiating a transaction, preparing inputs, interacting with smart contracts, generating proofs if needed,
and submitting the transaction to the sequencer for verification and state update. This flexible approach supports various use cases,
from financial transactions and decentralized applications to data privacy management, allowing users to maintain control over their privacy settings.</p>
<p>By offering this high degree of flexibility and security, Nescience enables users to tailor their privacy settings to their specific needs,
ensuring sensitive transactions remain confidential while non-sensitive ones are transparent. This integration of cryptographic keys and ZKPs
provides a robust framework for a wide range of blockchain applications, enhancing both utility and trust within the network.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="f-smart-contracts-in-nssa">f) Smart contracts in NSSA<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#f-smart-contracts-in-nssa" class="hash-link" aria-label="Direct link to f) Smart contracts in NSSA" title="Direct link to f) Smart contracts in NSSA">​</a></h3>
<hr>
<p>Smart contracts are a core feature of NSSA, providing a way to automate and execute predefined actions based on coded rules.
Once deployed on the blockchain, these contracts become immutable, meaning their behavior cannot be altered. This ensures that they perform exactly as
intended without the risk of tampering. Because the state and data of the contract are stored permanently on the blockchain, all interactions are fully
transparent and auditable, creating a reliable and trustworthy environment.</p>
<p>One of the key strengths of smart contracts is their ability to automate processes. They are designed to automatically execute when specific conditions are met,
reducing the need for manual oversight or intermediaries. For example, a smart contract might transfer funds when a certain deadline is reached or update a record
once a task is completed. This self-executing nature makes them efficient and minimizes human error.</p>
<p>Smart contracts operate deterministically, meaning they will always produce the same result given the same inputs. This predictability is crucial for ensuring reliability,
especially in complex systems. Additionally, they run in isolated environments on the blockchain, which enhances security by preventing unintended interactions with other processes.</p>
<p>Security is another critical feature of smart contracts. They leverage the underlying cryptographic protections of the blockchain, ensuring that every interaction
is secure and authenticated. Before deployment, the contract code can be audited and verified to ensure it functions correctly. Once on the blockchain,
the immutable nature of the code prevents unauthorized modifications, further ensuring the integrity of the system.</p>
<p>Running smart contracts requires computational resources, which are compensated through gas fees. These fees vary depending on the complexity of the operations within the contract.
More resource-intensive contracts incur higher fees, which helps manage the computational load on the blockchain network.</p>
<p>The workflow of a smart contract begins with its development, where developers code the contract using languages like Rust. Once the code is compiled and deployed to the blockchain,
it becomes a permanent part of the network. Users can then interact with the contract by sending transactions that invoke specific functions. The contract checks whether the
required conditions are met, and if so, it automatically executes the specified actions, such as transferring tokens or updating data on the blockchain.</p>
<p>The benefits of smart contracts are numerous. They eliminate the need for intermediaries by providing a system where trust is built into the code itself.
This not only reduces costs but also increases efficiency by automating repetitive processes. The inherent security of smart contracts, combined with their
transparency—where every action is recorded and visible on the blockchain—makes them a powerful tool for ensuring accountability and trust in decentralized systems.
They can be ideal for managing decentralized autonomous organizations (DAOs), where governance decisions are automated through coded rules.</p>
<p>By integrating smart contracts, NSSA offers a highly versatile, secure, and transparent framework that can support a wide range of applications
across various industries, from finance to governance, supply chains, and more.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="2-general-execution-overview">2. General execution overview<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#2-general-execution-overview" class="hash-link" aria-label="Direct link to 2. General execution overview" title="Direct link to 2. General execution overview">​</a></h2>
<hr>
<p>This section explains the execution process within NSSA, providing an overview of how it works from start to finish.
It outlines the steps involved in each execution type, guiding the reader through the entire process from user interaction to completion.</p>
<p>The process begins when a user initiates a transaction by invoking a smart contract. This invocation involves selecting at least one of
the four execution types: public, private, shielded, or deshielded. The choice of execution type determines how data will be read from and written to the blockchain,
affecting the transaction's privacy and security levels. Each execution type caters to different privacy needs, allowing the user to tailor the transaction according
to their specific requirements, whether it be full transparency or complete confidentiality.</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="general" src="https://vac.dev/assets/images/general-5851c1b4d07c68f30307b25f8dbdea85.png" width="2548" height="984" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="user-actions">User actions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#user-actions" class="hash-link" aria-label="Direct link to User actions" title="Direct link to User actions">​</a></h3>
<hr>
<p><strong>Step 1</strong>: <strong>Smart contract selection and input creation</strong></p>
<ul>
<li><strong>Smart contract selection</strong>: The user selects a smart contract they wish to invoke.</li>
<li><strong>Input creation</strong>: The user creates a set of inputs required for the invocation by reading the necessary data from both the public and private states. This includes:<!-- -->
<ul>
<li>Public data such as current account balances, public keys, and smart contract states.</li>
<li>Private data such as private account balances and UTXOs.</li>
</ul>
</li>
</ul>
<p><strong>Step 2</strong>: <strong>Choosing execution type</strong></p>
<ul>
<li><strong>Execution type selection</strong>: The user selects the type of execution based on their privacy needs. The options include:<!-- -->
<ul>
<li><strong>Public execution</strong>: Suitable for transactions where transparency is desired.</li>
<li><strong>Private execution</strong>: Used when transaction details need to be confidential.</li>
<li><strong>Shielded execution</strong>: Hides the receiver's identity.</li>
<li><strong>Deshielded execution</strong>: Hides the sender's identity.</li>
</ul>
</li>
<li><strong>ZkVM requirement</strong>: If the execution involves private, shielded, or deshielded types, the user must call the zkVM to handle these confidential transactions.
For purely public executions, the zkVM is not needed, and the user can directly transmit the transaction code to the sequencer.</li>
</ul>
<p><strong>Step 3</strong>: <strong>Calling zkVM for proof generation</strong></p>
<ul>
<li><strong>ZkVM compilation</strong>: The user calls the zkVM to compile the smart contract with both public and private inputs.<!-- -->
<ul>
<li><strong>Kernel circuit proofs</strong>: The zkVM generates individual proofs for each execution type through kernel circuits.</li>
<li><strong>Proof aggregation</strong>: The zkVM aggregates these individual proofs into a single comprehensive proof, combining both private and public inputs.</li>
</ul>
</li>
</ul>
<p><strong>Step 4</strong>: <strong>Transmitting public inputs and retaining private inputs</strong></p>
<ul>
<li><strong>Retaining private inputs</strong>: The user keeps the private inputs secure and does not transmit them.</li>
<li><strong>Revealing public inputs</strong>: The user transmits the following public inputs to the sequencer:<!-- -->
<ul>
<li>Public inputs of the recursive proof</li>
<li>Hashes of UTXOs</li>
<li>Updates to the public state</li>
<li>Transaction signature</li>
<li>Nullifiers (to prevent double spending)</li>
</ul>
</li>
</ul>
<p>After completing these steps, the user's part of the execution is done, and the sequencer takes over the process.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="sequencer-actions">Sequencer actions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#sequencer-actions" class="hash-link" aria-label="Direct link to Sequencer actions" title="Direct link to Sequencer actions">​</a></h3>
<hr>
<p><strong>Step 5</strong>: <strong>Proof verification</strong></p>
<ul>
<li><strong>Proof and data reception</strong>: The sequencer receives the proof and public inputs from the user.</li>
<li><strong>Verification process</strong>:<!-- -->
<ul>
<li>For private, shielded, and deshielded executions, the sequencer verifies the proof using the provided public data.</li>
<li>For public executions, the sequencer reruns the smart contract code with the provided inputs to check the results.</li>
</ul>
</li>
<li><strong>Validation</strong>: If both the zkVM proofs and public execution results are verified successfully, the sequencer collects the proof and public data to proceed.
If verification fails, the process is aborted, and the transaction is rejected.</li>
</ul>
<p><strong>Step 6</strong>: <strong>Aggregating proofs and finalizing the block</strong></p>
<ul>
<li><strong>Proof aggregation</strong>: The sequencer calls the zkVM again to aggregate all received proofs into one comprehensive proof to finalize the block.</li>
<li><strong>Finalizing the block</strong>:<!-- -->
<ul>
<li><strong>Public state update</strong>: The sequencer updates the public state with the new transaction data.</li>
<li><strong>Nullifier tree update</strong>: Updates the nullifier tree to reflect the new state and prevent double spending.</li>
<li><strong>Synchronization mechanism</strong>: Runs synchronization mechanisms to ensure fairness and consistency across the network.</li>
<li><strong>UTXO validation</strong>: Validates the exchanged UTXOs to complete the transaction process.</li>
</ul>
</li>
</ul>
<p>This comprehensive process ensures that transactions are executed securely, with the appropriate level of privacy and state updates synchronized across the network.</p>
<p>Below, we outline the execution process of the four different execution types within NSSA:</p>
<ul>
<li><strong>Public execution</strong>:</li>
</ul>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="public" src="https://vac.dev/assets/images/public-a6c41ad7e95eba55ef1c25d074023685.png" width="2214" height="884" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<ul>
<li><strong>Private execution</strong>:</li>
</ul>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="private" src="https://vac.dev/assets/images/private-4cc6385c296c7363327a4ceea2f75646.png" width="2602" height="872" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<ul>
<li><strong>Shielded execution</strong>:</li>
</ul>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="shielded" src="https://vac.dev/assets/images/se-98cd7f97b42e3b54c8664e188853f587.png" width="2542" height="894" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<ul>
<li><strong>Deshielded execution</strong>:</li>
</ul>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="deshielded" src="https://vac.dev/assets/images/de-99876f700ddaa6df7ff25e213167562b.png" width="2540" height="842" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="3-execution-processes-and-utxo-management">3. Execution processes and UTXO management<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#3-execution-processes-and-utxo-management" class="hash-link" aria-label="Direct link to 3. Execution processes and UTXO management" title="Direct link to 3. Execution processes and UTXO management">​</a></h2>
<hr>
<p>In Nescience state-separation architecture, UTXOs are key components for managing private data and assets. They serve as private entities that hold both storage and assets,
facilitating secure and confidential transactions. UTXOs are utilized in three of the four execution types within NSSA: private, shielded,
and deshielded executions. This section explores the lifecycle of UTXOs, detailing their generation, transfer, encryption, and eventual consumption within the private execution framework.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-components-of-a-nescience-utxo">a) Components of a Nescience UTXO<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-components-of-a-nescience-utxo" class="hash-link" aria-label="Direct link to a) Components of a Nescience UTXO" title="Direct link to a) Components of a Nescience UTXO">​</a></h3>
<hr>
<p>A Nescience UTXO is a critical and versatile component of the private state in the Nescience state-separation architecture.
It carries essential information that ensures its proper functionality within private execution, such as the owner, value, private storage slot, non-fungibles,
and other cryptographic components. Below is a detailed breakdown of each component and its role in maintaining the integrity, security, and privacy of the system:</p>
<ul>
<li>
<p><strong>Owner:</strong>
The owner component represents the public key of the entity that controls the UTXO. Only the owner can spend this UTXO, ensuring its security and privacy through public key cryptography.
This means that the UTXO remains secure as only the rightful owner, using their private key, can generate valid signatures to authorize the transaction. For example,
if Alice owns a UTXO linked to her public key, she must sign any transaction to spend it using her private key. This cryptographic protection ensures that only Alice can authorize
spending the UTXO and transfer it to someone else, such as Bob.</p>
</li>
<li>
<p><strong>Value:</strong>
The value in a UTXO represents the balance or asset contained within it. This could be cryptocurrency, tokens, or other digital assets. The value ensures accurate accounting,
preventing double spending and maintaining the overall integrity of the system. For instance, if Alice's UTXO has a value of 10 tokens, this represents her ownership of that amount
within the network, and when spent, this value will be deducted from her UTXO and transferred accordingly.</p>
</li>
<li>
<p><strong>Private storage slot:</strong>
The private storage slot is an arbitrary and flexible storage space within the UTXO for Nescience applications. It allows users and smart contracts to store additional private data
that is only accessible by the owner. This could be used to hold metadata, smart contract states, or user-specific information. For example, if a smart contract is holding private user data,
this information is securely stored in the private storage slot and can only be accessed or modified by the owner, ensuring privacy and security.</p>
</li>
<li>
<p><strong>Non-fungibles:</strong>
Non-fungibles within the UTXO represent unique assets, such as NFTs (Non-Fungible Tokens). Each non-fungible asset is assigned a unique serial number or identifier within the UTXO,
ensuring its distinctiveness and traceability. For example, if Alice owns a digital artwork represented as an NFT, the non-fungible component of the UTXO will store the unique identifier
for this NFT, preventing duplication or forgery of the digital asset.</p>
</li>
<li>
<p><strong>Random commitment key:</strong>
The random commitment key (RCK) is a randomly generated number used to create a cryptographic commitment to the contents of the UTXO. This commitment ensures the integrity of the data
without revealing any private information. By generating a random key for the commitment, the system ensures that even if someone observes the commitment, they cannot infer any details
about the underlying UTXO. For example, RCK helps maintain confidentiality in the system while still allowing the verification of transactions.</p>
</li>
<li>
<p><strong>Nullifier key:</strong>
The Nullifier key is another randomly generated number, used to ensure that a UTXO is only spent once. When a UTXO is spent, its nullifier key is recorded in a nullifier set to prevent
double spending. This key guarantees that once a UTXO is spent, it cannot be reused in another transaction, effectively nullifying it from future use. This mechanism is crucial for
maintaining the security and integrity of the system, as it ensures that no UTXO can be spent more than once.</p>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-utxo-lifecycle-from-generation-to-consumption">b) UTXO lifecycle: From generation to consumption<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#b-utxo-lifecycle-from-generation-to-consumption" class="hash-link" aria-label="Direct link to b) UTXO lifecycle: From generation to consumption" title="Direct link to b) UTXO lifecycle: From generation to consumption">​</a></h3>
<hr>
<p>UTXOs in NSSA are created when a transaction outputs a specific value, asset, or data intended for future use. Once generated, these UTXOs become private entities
owned by specific users, containing sensitive information such as balances, private data, or unique assets like NFTs.</p>
<p>To maintain the required level of confidentiality, UTXOs are encrypted and transferred anonymously across the network. This encryption process ensures that the data within each UTXO
remains hidden from network participants, including the sequencer, while still allowing for verification and validation through ZKPs. These proofs enable the network
to ensure that UTXOs are valid, prevent double spending, and maintain security, all without revealing any sensitive information.</p>
<p>When a user wishes to spend or transfer a UTXO, the lifecycle progresses towards its consumption. The user must prove ownership and validity of the UTXO through a ZKP,
which is then verified by the sequencer. This process occurs in private, shielded, and deshielded executions, where confidentiality is a priority. Once the proof is validated,
the UTXO is consumed, meaning it is marked as spent and cannot be reused, ensuring the integrity of the transaction and preventing double spending.</p>
<p>UTXOs are central to the private, shielded, and deshielded execution types in Nescience. In private executions, UTXOs are transferred securely between parties without revealing any
details to the public state. In shielded executions, UTXOs are used to receive assets from the public state while keeping the recipient's identity confidential. Finally,
in deshielded executions, UTXOs are used to send assets from the private state to the public state, while preserving the sender's anonymity.</p>
<p>Since UTXOs are not exchanged in public executions, this lifecycle analysis is focused solely on private, shielded, and deshielded executions, where privacy and confidentiality are essential.
In these contexts, the careful management and transfer of UTXOs ensure that the users' private data and assets remain secure, while still allowing for seamless and confidential transactions
within the network.</p>
<p>At this point, it's crucial to introduce two key components that will play a significant role in the next section: the ephemeral key and the nillifier.</p>
<ul>
<li>
<p><strong>Ephemeral key:</strong> The ephemeral key is embedded in the transaction message and plays a crucial role in maintaining privacy. It is used by the sender, alongside the receiver's public key,
in a key agreement protocol to derive a shared secret. This shared secret is then employed to encrypt the transaction details, ensuring that only those with the receiver's viewing key can
decrypt the transaction. By using the ephemeral key, the receiver can regenerate the shared secret, granting access to the transaction's contents. The sender generates the ephemeral key
using their spending key and the UTXO's nullifier, reinforcing the security of the transaction. (more details in <a href="https://vac.dev/rlog/Nescience-state-separation-architecture#key">key management and addresses section</a>)</p>
</li>
<li>
<p><strong>Nullifier:</strong> A nullifier is a unique value tied to a specific UTXO, ensuring that it has not been spent before. Its uniqueness is essential, as a nullifier must never correspond to more
than one UTXO—otherwise, even if both UTXOs are valid, only one could be spent. This would undermine the integrity of the system. To spend a UTXO, a proof must be provided showing that
the nullifier does not already exist in the Nullifier Tree. Once the transaction is confirmed and included in the blockchain, the nullifier is added to the Nullifier Tree, preventing any
future reuse of the same UTXO. A UTXO's nullifier is generated by combining the receiver's nullifier key with the transaction note's commitment, further ensuring its distinctiveness
and security. (More details in <a href="https://vac.dev/rlog/Nescience-state-separation-architecture#nul">nullifier tree section</a>.)</p>
</li>
</ul>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-i-utxos-in-private-executions"><a id="pe"></a> I) UTXOs in private executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#-i-utxos-in-private-executions" class="hash-link" aria-label="Direct link to -i-utxos-in-private-executions" title="Direct link to -i-utxos-in-private-executions">​</a></h4>
<hr>
<p>In private executions within NSSA, transactions are handled ensuring maximum privacy by concealing all transaction details from the public state.
This approach is particularly useful for confidential payments, where the identities of the sender and receiver, as well as the transaction amounts, must remain hidden.
The process is powered by ZKPs, ensuring that only the involved parties have access to the transaction details while maintaining the integrity of the network.</p>
<ul>
<li>
<p><strong>Stages of private execution</strong>: Private executions operate in two key stages: UTXO consumption and UTXO creation. In the first stage, UTXOs from the private state are used
as inputs for the transaction. In the second stage, new UTXOs are generated as outputs and stored back in the private state. Throughout this process, the details of the
transaction are kept confidential and only shared between the sender and receiver.</p>
</li>
<li>
<p><strong>Private transaction workflow (transaction initialization)</strong>: The user initiates a private transaction by selecting the input UTXOs that will be spent and determining the
output UTXOs to be created. This involves specifying the amounts to be transferred and the recipient’s private address (a divestified address that hides the recipient's public
address from the network). The nullifier key and random number for commitments (RCK) are also generated at this stage to define how these UTXOs can be spent or nullified in the
future by the receiver.</p>
</li>
<li>
<p><strong>Proof generation and verification</strong>: Next, the zkVM generates a ZKP to validate the transaction. This proof includes both a membership proof for the input UTXOs,
confirming their presence in the hashed UTXO tree, and a non-membership proof to ensure that the input UTXOs have not already been spent (i.e., they are not in the nullifier tree).
The proof also confirms that the total input value matches the total output value, ensuring no discrepancies. The user then submits the proof, along with the necessary metadata, to the sequencer.</p>
</li>
<li>
<p><strong>Shared secret and encryption</strong>: To maintain confidentiality, the sender uses the receiver’s divestified address to generate an ephemeral public key.
This allows the creation of a shared secret between the sender and receiver. Using a key derivation function, a symmetric encryption key is generated from the shared secret.
The input and output UTXOs are then encrypted using this symmetric key, ensuring that only the intended recipient can decrypt the data.</p>
</li>
<li>
<p><strong>Broadcasting the transaction</strong>: The user broadcasts the encrypted UTXOs to the network, along with a commitment to the output UTXOs using Pedersen hashes.
These committed UTXOs are sent to the sequencer, which updates the hashed UTXO tree without knowing the transaction details.</p>
</li>
<li>
<p><strong>Decryption by the receiver</strong>: After the broadcast, the receiver attempts to decrypt the broadcast UTXOs using their symmetric key, derived from the ephemeral public key.
If the receiver successfully decrypts a UTXO, it confirms ownership of that UTXO. The receiver then computes the nullifier for the UTXO and verifies its presence in the hashed
UTXO tree and its absence from the nullifier tree, ensuring it has not been spent. Finally, the new UTXO is added to the receiver’s locally stored UTXO tree for future transactions.</p>
</li>
</ul>
<p>Throughout the private execution process, the identities of both the sender and receiver, as well as all transaction details, remain hidden from the public.
The use of ZKPs ensures that the integrity of the transaction is verified without revealing any sensitive information. At the end of the process,
the network guarantees that no participant, aside from the sender and receiver, can deduce any details about the transaction or the involved parties.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="ii-utxos-in-shielded-executions">II) UTXOs in shielded executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#ii-utxos-in-shielded-executions" class="hash-link" aria-label="Direct link to II) UTXOs in shielded executions" title="Direct link to II) UTXOs in shielded executions">​</a></h4>
<hr>
<p>In shielded executions, the interaction between public and private states provides a hybrid privacy model that balances transparency and confidentiality.
This model is suitable for scenarios where the initial step, such as a public transaction, requires visibility, while subsequent actions, such as private asset management,
need to remain confidential. One common use case is asset conversion—where a public token is converted into a private token. The conversion is visible on the public ledger,
but subsequent transactions remain private.</p>
<h5 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-how-shielded-executions-work">a) How shielded executions work<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-how-shielded-executions-work" class="hash-link" aria-label="Direct link to a) How shielded executions work" title="Direct link to a) How shielded executions work">​</a></h5>
<p>Shielded executions operate in two distinct stages: first, there is a modification of the public state, and then new UTXOs are created and stored in the private state.
Importantly, shielded executions do not consume UTXOs but instead mint them, as new UTXOs are created to reflect the changes in the private state. This structure demands
ZKPs to ensure that the newly minted UTXOs are consistent with the modifications in the public state. Here’s a step-by-step breakdown of how the shielded
execution process unfolds:</p>
<ol>
<li>
<p><strong>Transaction initiation:</strong> The user initiates a transaction that modifies the public state, such as converting a public token to a private token.
The transaction alters the public state (e.g., balances or smart contract storage) while simultaneously preparing to mint new UTXOs in the private state.</p>
</li>
<li>
<p><strong>Generating UTXOs:</strong> After modifying the public state, the system mints new UTXOs in the private state. These UTXOs must be securely created, ensuring their integrity
and consistency with the initial public state modification. A ZKP is generated by the user to prove that these new UTXOs align with the changes made in the public state.</p>
</li>
<li>
<p><strong>Key setup for privacy</strong>: The sender retrieves the receiver's address and uses it to create a shared secret through an ephemeral public key. This shared secret is then used
to derive a symmetric key, which encrypts the output UTXOs. This encryption ensures that only the intended receiver can decrypt and access the UTXOs.</p>
</li>
<li>
<p><strong>Broadcasting and verifying UTXOs</strong>: After encrypting the UTXOs, the sender broadcasts them to the network. The new hashed UTXOs are sent to the sequencer,
which verifies the validity of the UTXOs and attaches them to the hashed UTXO tree within the private state. The public inputs for the ZKP circuits consist of the
Pedersen-hashed UTXOs and the modifications in the public state.</p>
</li>
<li>
<p><strong>Receiver's role</strong>: Once the UTXOs are broadcast, the receiver attempts to decrypt each UTXO using the symmetric key derived from the shared secret. If the decryption is successful,
the UTXO belongs to the receiver. The receiver then verifies the UTXO’s validity by checking its inclusion in the hashed UTXO tree and ensuring that its nullifier has not yet been used.</p>
</li>
<li>
<p><strong>Nullifier check and integration</strong>: To prevent double spending, the receiver computes the nullifier for the received UTXO and verifies that it is not already present in the nullifier tree.
Once verified, the receiver adds the UTXO to their locally stored UTXO tree for future use in private transactions.</p>
</li>
</ol>
<p>While shielded executions offer privacy, certain information is still exposed to the public state, such as the sender's identity. To further enhance privacy,
the sender can create empty UTXOs—UTXOs that don’t belong to anyone but are included in the transaction to obfuscate the true details of the transaction.
Though this approach increases the size of the data, it adds a layer of privacy by complicating the identification of meaningful transactions.</p>
<h5 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-summary-of-shielded-execution-flow">b) Summary of shielded execution flow<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#b-summary-of-shielded-execution-flow" class="hash-link" aria-label="Direct link to b) Summary of shielded execution flow" title="Direct link to b) Summary of shielded execution flow">​</a></h5>
<ul>
<li><strong>Stage 1 (public modification):</strong> The user modifies public state data, such as converting tokens from public to private. This stage is visible to the public.</li>
<li><strong>Stage 2 (UTXO minting and privacy):</strong> New UTXOs are minted in the private state, encrypted, and broadcast to the network. The transaction remains private from this point forward,
secured by ZKPs and cryptographic keys.</li>
<li><strong>Receiver’s role:</strong> The receiver decrypts the UTXOs and verifies their validity, ensuring the UTXOs are not double spent and are ready for future transactions.</li>
</ul>
<p>In summary, shielded executions enable a hybrid privacy model in Nescience, balancing public transparency and private confidentiality. They are well-suited for
transactions requiring initial public visibility, such as asset conversions, while ensuring that subsequent actions remain secure and private within the network.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iii-utxos-in-deshielded-executions">III) UTXOs in deshielded executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#iii-utxos-in-deshielded-executions" class="hash-link" aria-label="Direct link to III) UTXOs in deshielded executions" title="Direct link to III) UTXOs in deshielded executions">​</a></h4>
<hr>
<p>In NSSA, deshielded executions offer a unique way to move data and assets from the private state to the public state, revealing previously private
information in a controlled and verifiable manner. This type of execution allows for selective disclosure, ensuring transparency when needed while still maintaining
the security and privacy of critical details through cryptographic techniques like ZKPs. Deshielded executions are particularly valuable for use cases
such as regulatory compliance reporting, where specific transaction details must be revealed to meet legal requirements, while other sensitive transactions remain private.</p>
<h5 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-stages-of-deshielded-executions">a) Stages of deshielded executions<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-stages-of-deshielded-executions" class="hash-link" aria-label="Direct link to a) Stages of deshielded executions" title="Direct link to a) Stages of deshielded executions">​</a></h5>
<ul>
<li>
<p><strong>Stage 1 (UTXO consumption):</strong> The process begins in the private state, where UTXOs are consumed as inputs for the transaction. This involves gathering all necessary
UTXOs that contain the assets or balances to be made public, as well as any associated private data stored in memory slots.</p>
</li>
<li>
<p><strong>Stage 2 (public state modification):</strong> After the UTXOs are consumed, the transaction details are made public by modifying the public state. This update includes changes
to the public balances, storage data, and any necessary public records. While the public state is updated, the sender’s identity and other sensitive information remain hidden,
thanks to the privacy-preserving properties of ZKPs.</p>
</li>
</ul>
<p>This model ensures that private data can be selectively revealed when needed, offering both flexibility and transparency. It is particularly useful for scenarios requiring
auditing or compliance reporting, where specific details must be made publicly verifiable without exposing the entire history or contents of private transactions.</p>
<h5 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-how-deshielded-executions-work">b) How deshielded executions work<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#b-how-deshielded-executions-work" class="hash-link" aria-label="Direct link to b) How deshielded executions work" title="Direct link to b) How deshielded executions work">​</a></h5>
<p>The deshielded execution process starts when a user initiates a transaction using private UTXOs. The Nescience zkVM is called to generate a ZKP,
which validates the transaction without revealing sensitive details such as the sender's identity or the specifics of the Nescience application being executed.</p>
<p>During the transaction, the UTXOs from the private state are consumed, meaning they are used up as inputs and will no longer be available for future transactions.
Instead of generating new UTXOs, the transaction modifies the public state, updating the necessary balances or memory slots related to the transaction.
Here’s a step-by-step breakdown of how the deshielded execution process unfolds:</p>
<ol>
<li>
<p><strong>Get receiver's public address:</strong> The sender first identifies the public address of the receiver, to which the information or assets will be made public.</p>
</li>
<li>
<p><strong>Determine input UTXOs and public state modifications:</strong> The sender gathers all the input UTXOs needed for the transaction and determines the public state modifications
necessary for the Nescience applications and token transfers involved.</p>
</li>
<li>
<p><strong>Calculate nullifiers:</strong> Nullifiers are generated for each input UTXO, ensuring that these UTXOs cannot be reused or double spent. The nullifiers are derived from the
corresponding UTXO commitments.</p>
</li>
<li>
<p><strong>Call zkVM with deshielded circuits:</strong> The sender invokes the zkVM with deshielded kernel circuits, which generates the proof. The proof ensures that all input UTXOs
are valid by verifying their membership in the UTXO tree and their non-membership in the nullifier tree, ensuring they haven’t been spent.</p>
</li>
<li>
<p><strong>Generate and submit proof:</strong> The zkVM generates a ZKP that verifies the correctness of the transaction without revealing private details.
The proof includes the nullifiers and the planned modifications to the public state.</p>
</li>
<li>
<p><strong>Send proof to sequencer:</strong> The sender then sends the proof and any relevant public information to the sequencer. The sequencer is responsible for verifying the proof,
updating the public state accordingly, and adding the nullifiers to the nullifier tree.</p>
</li>
</ol>
<p>Once the proof and public information have been broadcast to the network, the receiver does not need to take any further action.
The sequencer manages the public state updates and ensures that the transaction is properly executed. By the end of the deshielded execution,
specific transaction details become publicly visible, such as the identity of the receiver and the outcome of the transaction.
This allows participants in the public state to extract information about the transaction, including the receiver's identity and some details about the execution.
While the receiver's identity is revealed, the sender's identity and sensitive transaction details remain hidden, thanks to the use of ZKPs.
This makes deshielded executions ideal for cases where transparency is needed, but complete privacy is still a priority for certain elements of the transaction.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary-of-utxo-consumption-in-nssa">Summary of UTXO consumption in NSSA<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#summary-of-utxo-consumption-in-nssa" class="hash-link" aria-label="Direct link to Summary of UTXO consumption in NSSA" title="Direct link to Summary of UTXO consumption in NSSA">​</a></h3>
<hr>
<p>In NSSA, consuming UTXOs is a critical step in maintaining the security and integrity of the blockchain by preventing double spending.
When a UTXO is consumed, it is used as an input in a transaction, effectively marking it as spent. This ensures that the UTXO cannot be reused, preserving the integrity of the blockchain.</p>
<ol>
<li><strong>The process of consuming UTXOs:</strong> The process of consuming a UTXO begins when a user selects a UTXO from their private state. The user verifies the UTXO’s existence and
ownership using their viewing key, ensuring that they are the legitimate owner of the UTXO. Once verified, the user generates two key cryptographic proofs:<!-- -->
<ul>
<li><strong>Membership proof:</strong> This proof confirms that the UTXO exists within the hashed UTXO tree, ensuring its validity within the system.</li>
<li><strong>Non-membership proof:</strong> This proof ensures that the UTXO has not been previously consumed by checking its absence in the nullifier tree, which tracks spent UTXOs.</li>
</ul>
</li>
</ol>
<p>To mark the UTXO as spent, a <strong>nullifier</strong> is generated. This nullifier is a unique cryptographic hash derived from the UTXO, which is then added to the nullifier tree in the public state.
Adding the nullifier to the tree prevents the UTXO from being reused in future transactions, thus preventing double spending.</p>
<p>After generating the membership and non-membership proofs, the user compiles the transaction using the zkVM. The zkVM is responsible for generating the necessary ZKPs,
which validate the transaction without revealing sensitive details. The compiled transaction, along with the proofs, is then submitted to the sequencer for verification.</p>
<ol start="2">
<li><strong>The role of the sequencer:</strong> Once the transaction is submitted, the sequencer verifies the ZKPs to confirm that the transaction is valid. If the proofs are verified
successfully, the sequencer updates both the private and public states to reflect the transaction. This includes updating the nullifier tree with the newly generated nullifier,
ensuring that the UTXO is marked as spent and cannot be reused.</li>
</ol>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="example-alice-sending-tokens-to-bob">Example: Alice sending tokens to Bob<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#example-alice-sending-tokens-to-bob" class="hash-link" aria-label="Direct link to Example: Alice sending tokens to Bob" title="Direct link to Example: Alice sending tokens to Bob">​</a></h4>
<hr>
<p>Consider an example where Alice wants to send 5 Nescience tokens to Bob using a private execution. Alice selects a UTXO from her private state that contains 5 Nescience tokens.
She generates the necessary membership and non-membership proofs, ensuring that her UTXO exists in the system and has not been previously spent. Alice then creates a nullifier by
hashing the UTXO and compiles the transaction with the zkVM.</p>
<p>Once Alice submits the transaction, the sequencer verifies the proofs and updates the blockchain by adding the nullifier to the nullifier tree and recording the transaction details.
This ensures that Alice’s UTXO is marked as spent and cannot be used again, while Bob receives the 5 tokens.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-importance-of-nullifiers">The importance of nullifiers<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#the-importance-of-nullifiers" class="hash-link" aria-label="Direct link to The importance of nullifiers" title="Direct link to The importance of nullifiers">​</a></h4>
<hr>
<p>Nullifiers are a key mechanism in preventing double spending. By marking consumed UTXOs as spent and tracking them in the nullifier tree, NSSA ensures that
once a UTXO is used in a transaction, it cannot be reused in any future transactions. This process is fundamental to maintaining the integrity and security of the blockchain,
as it guarantees that assets are only spent once and prevents potential attacks on the system.</p>
<p>In conclusion, the process of consuming UTXOs in NSSA combines cryptographic proofs, nullifiers, and ZKPs to ensure that transactions
are secure, confidential, and free from the risks of double spending.</p>
<h1>C. Cryptographic primitives in NSSA</h1>
<p>In the NSSA, cryptographic primitives are the foundational elements that ensure the security, privacy, and efficiency of the state separation model.
These cryptographic tools enable private transactions, secure data management, and robust verification processes across both public and private states.
The architecture leverages a wide range of cryptographic mechanisms, including advanced hash functions, key management systems, tree structures, and ZKPs,
to safeguard user data and maintain the integrity of transactions.</p>
<p>Cryptographic hash functions play a pivotal role in concealing UTXO details, generating nullifiers, and constructing sparse Merkle trees, which organize and verify
data efficiently within the network. Key management and address generation further enhance the security of user assets and identity, ensuring that only authorized
users can access and control their holdings.</p>
<p>The architecture also relies on specialized tree structures for organizing data, verifying the existence of UTXOs, and tracking nullifiers, which prevent double spending.
Additionally, Nescience features a privacy-preserving zero-knowledge virtual machine (zk-zkVM), which allows users to prove the correctness of an execution without
disclosing sensitive information. This enables private transactions and maintains confidentiality across the network.</p>
<p>As Nescience evolves, optional cryptographic mechanisms such as multi-party computation (MPC) may be integrated to enhance synchronization across privacy levels.
This MPC-based synchronization mechanism is still under development and under review for potential inclusion in the system. Together, these cryptographic primitives
form the backbone of Nescience’s security architecture, ensuring that users can transact and interact privately, securely, and efficiently.</p>
<p>In the following sections, we will explore each of these cryptographic components in detail, beginning with the role of hash functions.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-hash-functions-in-nescience">a) Hash functions in Nescience<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-hash-functions-in-nescience" class="hash-link" aria-label="Direct link to a) Hash functions in Nescience" title="Direct link to a) Hash functions in Nescience">​</a></h2>
<hr>
<p>Hash functions are a foundational element of Nescience’s cryptographic framework, serving multiple critical roles that ensure the security, privacy, and efficiency of the system.
One of the primary uses of hash functions in Nescience is to conceal sensitive details of UTXOs by converting them into fixed-size hashes. This process allows UTXO details
to remain private, ensuring that sensitive information is not directly exposed on the blockchain, while still enabling their existence and integrity to be verified. Hashing
the UTXO details allows the actual data to remain confidential, with the hashes stored in a global tree structure for efficient management and retrieval.</p>
<p>Additionally, hash functions are essential for generating <strong>nullifiers</strong>, which play a crucial role in preventing double spending. Nullifiers are created by hashing UTXOs
and are used to mark them as spent, ensuring that they cannot be reused in subsequent transactions. These nullifiers are stored in a nullifier tree, and each transaction
must prove that its UTXO’s nullifier is not already present in the tree before it can be processed. This ensures that the UTXO has not been spent before, maintaining the
integrity of the transaction process.</p>
<p>Hash functions are also vital in the construction of <strong>sparse Merkle trees</strong>, which provide an efficient and secure method for verifying data within the blockchain.
Sparse Merkle trees enable quick and reliable proofs of membership and non-membership, making them essential for verifying both UTXOs and nullifiers. By using hash functions
to build these trees, Nescience can ensure the integrity of the data, as any tampering with the data would result in a change in the hash, making the manipulation detectable.</p>
<p>Another critical consideration in Nescience is the compatibility of hash functions with <strong>ZKPs</strong>. ZK-friendly hash functions are optimized for efficient
computation within the constraints of ZK circuits, ensuring that they do not become a bottleneck in the proof generation or verification process. These hash functions
maintain strong cryptographic security properties while enabling efficient computations in ZKP systems, which is essential for maintaining privacy and
integrity within the ZK framework.</p>
<p>The primary advantage of using hash functions in Nescience is their ability to ensure that transaction details remain private while still allowing for verification
of their validity. Furthermore, by integrating hash functions into Merkle trees, the blockchain data becomes tamper-proof, enabling quick and efficient verification
processes that uphold the system’s security and privacy standards.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="use-case-how-to-use-the-pedersen-hash-to-create-the-utxo-commitment">Use case: How to use the Pedersen hash to create the UTXO commitment<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#use-case-how-to-use-the-pedersen-hash-to-create-the-utxo-commitment" class="hash-link" aria-label="Direct link to Use case: How to use the Pedersen hash to create the UTXO commitment" title="Direct link to Use case: How to use the Pedersen hash to create the UTXO commitment">​</a></h3>
<hr>
<p>As mentioned in the <a href="https://vac.dev/rlog/Nescience-state-separation-architecture#pe">UTXOs in private executions section</a>, the user broadcasts the encrypted UTXOs to the network, along with a commitment to the output UTXOs
using <strong>Pedersen hashes</strong>. The Pedersen hash is used to create the UTXO commitment. The Pedersen hash is a homomorphic commitment scheme that allows secure commitments
while maintaining privacy and enabling proofs of correctness in transactions. The commitment formula is as follows:</p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>=</mo><mi>C</mi><mo stretchy="false">(</mo><mi>U</mi><mi>T</mi><mi>X</mi><mi>O</mi><mo separator="true">,</mo><mi>R</mi><mi>C</mi><mi>K</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mi>g</mi><mrow><mi>U</mi><mi>T</mi><mi>X</mi><mi>O</mi></mrow></msup><mo>⋅</mo><msup><mi>h</mi><mrow><mi>R</mi><mi>C</mi><mi>K</mi></mrow></msup></mrow><annotation encoding="application/x-tex">Commitment = C(UTXO,RCK) =g^{UTXO}⋅h^{RCK}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="mord mathnormal">o</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mord mathnormal" style="margin-right:0.02778em">TXO</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.07153em">RC</span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0358em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.10903em">U</span><span class="mord mathnormal mtight" style="margin-right:0.02778em">TXO</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.8413em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.07153em">RC</span><span class="mord mathnormal mtight" style="margin-right:0.07153em">K</span></span></span></span></span></span></span></span></span></span></span></span></p>
<p>In this formula, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi></mrow><annotation encoding="application/x-tex">g</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">h</span></span></span></span> are two generators of a cryptographic group where no known relationship exists between them. This ensures that the commitment is secure
and computationally infeasible to reverse or manipulate without knowing the original UTXO components. The random number <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>C</mi><mi>K</mi></mrow><annotation encoding="application/x-tex">RCK</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">RC</span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span> adds an additional layer of security
by blinding the UTXO's contents, ensuring that the commitment doesn't leak any information about the underlying data.</p>
<p><strong>Importance of homomorphic commitments</strong></p>
<p>It is essential to use a homomorphic commitment like the Pedersen commitment for UTXOs because it allows for the verification of important properties in transactions,
such as ensuring that the total input value of a transaction equals the total output value. This balance is crucial for preventing the unauthorized creation of funds or d
discrepancies in transactions. A homomorphic commitment enables these proofs because of its additive properties. Specifically, the exponents in the commitment formula are additive,
meaning that commitments can be combined and verified without revealing the individual components. For instance, if you have two UTXOs with commitments <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mo stretchy="false">(</mo><mi>U</mi><mi>T</mi><mi>X</mi><msub><mi>O</mi><mn>1</mn></msub><mo separator="true">,</mo><mi>R</mi><mi>C</mi><msub><mi>K</mi><mn>1</mn></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">C(UTXO_1,RCK_1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mord mathnormal" style="margin-right:0.07847em">TX</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">O</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.07153em">RC</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>
and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mo stretchy="false">(</mo><mi>U</mi><mi>T</mi><mi>X</mi><msub><mi>O</mi><mn>2</mn></msub><mo separator="true">,</mo><mi>R</mi><mi>C</mi><msub><mi>K</mi><mn>2</mn></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">C(UTXO_2,RCK_2)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">U</span><span class="mord mathnormal" style="margin-right:0.07847em">TX</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">O</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.07153em">RC</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0715em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mclose">)</span></span></span></span>, you can combine them and verify that the resulting commitment is valid without exposing the actual amounts.</p>
<p>This capability is leveraged through a modified version of the Schnorr protocol, which is used in conjunction with the Pedersen hash to verify the correctness of transactions.
The Schnorr protocol allows users to prove, without revealing the actual values, that the sum of inputs equals the sum of outputs, ensuring that no funds are created or lost in the transaction.</p>
<p><strong>Limitations of standard cryptographic hashes</strong></p>
<p>Standard cryptographic hash functions, such as SHA-256, are not suitable for this purpose because they lack the algebraic structure needed for homomorphic properties.
In particular, while SHA-256 provides strong security for general hashing purposes, it does not allow the additive properties that are required to perform the type of
ZKPs used in Nescience for UTXO commitments. This is why the Pedersen hash is preferred, as it enables the secure and private execution of transactions
while allowing for balance verification and other critical proofs.</p>
<p><strong>Conclusion</strong></p>
<p>By using homomorphic commitments like the Pedersen hash, NSSA ensures that UTXOs can be securely committed and validated without exposing sensitive information.
The random component (RCK) adds an additional layer of security, and the additive properties of the Pedersen commitment enable powerful ZKPs that maintain the
integrity of the system.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-b-key-management-and-addresses-in-nescience"><a id="key"></a> b) Key management and addresses in Nescience<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#-b-key-management-and-addresses-in-nescience" class="hash-link" aria-label="Direct link to -b-key-management-and-addresses-in-nescience" title="Direct link to -b-key-management-and-addresses-in-nescience">​</a></h2>
<hr>
<p>NSSA utilizes different cryptographic schemes, such as public key encryption and digital signatures, to ensure secure private executions through
the exchange of UTXOs. These schemes rely on a structured set of cryptographic keys, each serving a specific purpose in maintaining privacy, security, and control over assets.
Here's a breakdown of the keys used in Nescience:</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="i-spending-key">I. Spending key<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#i-spending-key" class="hash-link" aria-label="Direct link to I. Spending key" title="Direct link to I. Spending key">​</a></h3>
<p>The spending key is the fundamental secret key in NSSA, acting as the primary control mechanism for a user’s UTXOs and other digital assets.
It plays a critical role in the cryptographic security of the system, ensuring that only the rightful owner can authorize and spend their assets.</p>
<ul>
<li>
<p><strong>Role of the spending key</strong>: The spending key is responsible for generating the user’s private keys, which are used in various cryptographic operations such as
signing transactions and creating commitments. This hierarchical relationship means that the spending key sits at the root of a user’s key structure, safeguarding
access to all associated private keys and, consequently, to the user’s assets. In Nescience’s privacy-focused model, the spending key is never exposed or shared outside
the user’s control. Unlike other keys, it does not interact with the public state, kernel circuits, or even the ZKP system. This isolation ensures that
the spending key remains completely private and inaccessible to external entities. By keeping the spending key separate from the operational aspects of the network,
Nescience minimizes the risk of key leakage or compromise.</p>
</li>
<li>
<p><strong>Generation and security of the spending key</strong>: The spending key is generated randomly from the scalar field, a large mathematical space that ensures uniqueness
and cryptographic strength. This randomness is crucial because it prevents attackers from predicting or replicating the key, thereby safeguarding the user’s assets
from unauthorized access: it is computationally infeasible for an attacker to guess or brute-force the key. Once the spending key is generated, it is securely stored
by the user, typically in a hardware wallet or another secure storage mechanism that prevents unauthorized access.</p>
</li>
<li>
<p><strong>Spending UTXOs with the spending key</strong>: The spending key’s primary function is to authorize the spending of UTXOs in private transactions. When a user initiates
a transaction, the spending key is used to generate the necessary cryptographic proofs and signatures, ensuring that the transaction is valid and originates from
the rightful owner. However, even though the spending key generates these proofs, it is never directly exposed during the transaction process. Instead, derived
private keys handle the operational aspects while the spending key remains secure in the background. For example, when Alice decides to spend a UTXO in a
private execution, her spending key generates the required private keys that will sign the transaction and ensure its validity. However, the spending key itself
never appears in any public state or transaction data, preserving its confidentiality.</p>
</li>
<li>
<p><strong>Ensuring security through isolation</strong>: One of the key security principles of the spending key is its isolation from the network. Since it never interacts with
public-facing elements, such as the public state or kernel circuits, the risk of exposure is significantly reduced. This isolation ensures that even if other parts
of the cryptographic infrastructure are compromised, the spending key remains protected, preventing unauthorized spending of UTXOs.</p>
</li>
</ul>
<p>In summary, the spending key in Nescience is a powerful and carefully guarded element of the cryptographic system. It is the root key from which other private keys
are derived, allowing users to spend their UTXOs securely and privately. Its isolation from the public state and its random generation from a secure scalar field ensures
that the spending key remains protected, making it a cornerstone of security in NSSA.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="ii-private-keys">II. Private keys<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#ii-private-keys" class="hash-link" aria-label="Direct link to II. Private keys" title="Direct link to II. Private keys">​</a></h3>
<p>In Nescience, the private key is an essential cryptographic element responsible for facilitating various secure operations, such as generating commitments and signing
transactions. While the spending key plays a foundational role in safeguarding access to UTXOs and assets, the private keys handle the operational aspects of transactions
and cryptographic proofs. The private key consists of three critical components: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>s</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rsd</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rs</span><span class="mord mathnormal">d</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>, each serving a
distinct purpose within the Nescience cryptographic framework.</p>
<ol>
<li>
<p><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>s</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rsd</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rs</span><span class="mord mathnormal">d</span></span></span></span> (random seed)</strong>: The random seed (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>s</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rsd</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rs</span><span class="mord mathnormal">d</span></span></span></span>) is the first and foundational component of the private key. It is a value randomly chosen from the scalar field, which ensures
its cryptographic security and unpredictability. This seed is generated using a random number generator, making it virtually impossible to predict or replicate.
The random seed is essential because it is used to derive the other two components of the private key. By leveraging a secure random seed, Nescience ensures that
the entire private key structure is rooted in randomness, preventing external entities from guessing or deriving the key through brute-force attacks.
The strength of the random seed ensures the overall security of the private key and, consequently, the integrity of the user's transactions and commitments.</p>
</li>
<li>
<p><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> (random commitment)</strong>: The random commitment component (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span>) is a crucial part of the private key used specifically in the commitment scheme. It acts as a blinding factor,
adding a layer of security to commitments made by the user. The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> value is also drawn from the scalar field and is used to ensure that the commitment
to any UTXO or other sensitive data remains confidential. The commitment scheme in Nescience requires the use of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> to create cryptographic commitments
that bind the user to specific data (such as UTXO details) without revealing the actual data. The role of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> is to ensure that these commitments are
non-malleable and secure, preventing anyone from modifying the committed data without detection. For instance, when Alice commits to a UTXO, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> is used
to generate a Pedersen commitment that ensures the UTXO details are hidden but can still be verified cryptographically. This means that even though the actual UTXO details
are concealed, their existence and integrity can be proven.</p>
</li>
<li>
<p><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> (signing key for transactions)</strong>: The signing key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>) is the third and final component of the private key, used primarily for signing transactions. One possible approach is that
Nescience employs Schnorr signatures, a cryptographic protocol known for its efficiency and security. In this case, the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> component would generate
Schnorr signatures that are used to authenticate transactions, ensuring that only the rightful owner of the private key can authorize the spending of UTXOs. Schnorr
signatures are important as they provide a secure and non-repudiable method of verifying that a transaction was initiated by the legitimate owner of the assets.
When Alice signs a transaction using her <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>, the corresponding public key allows others to verify that the transaction was indeed signed by Alice,
without revealing her private key. This verification process ensures that all transactions are legitimate and prevents unauthorized entities from forging transactions
or spending assets they do not control. Even if an attacker gains access to the signed transaction, they cannot reverse engineer the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>, ensuring
the security of Alice's future transactions.</p>
</li>
</ol>
<p><strong>Robustness of private keys in Nescience</strong></p>
<p>Despite the critical role of the private key in the operation of NSSA, the system is designed to maintain security even in the event that the
private key is compromised. This resilience is achieved through the integrity of the spending key, which is never exposed in the process of signing or committing.
The spending key acts as the ultimate safeguard, ensuring that even if a private key component is compromised, the attacker cannot access or spend the user's assets
without control over the spending key.</p>
<p>The architecture’s design, where private keys handle operational tasks but rely on the spending key for ultimate control, ensures a layered approach to security.
This way, the system can mitigate the damage of a compromised private key by maintaining the inviolability of the user's assets.</p>
<p><strong>Conclusion</strong></p>
<p>In summary, the private key in Nescience consists of three interrelated components that together ensure secure transaction signing, commitment creation, and the
protection of user data. The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>s</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rsd</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rs</span><span class="mord mathnormal">d</span></span></span></span> serves as the root from which the other key components are derived, ensuring randomness and security.
The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> plays a crucial role in generating commitments, while <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> provides the signing capability needed for transaction authentication.
Together, these components enable users to engage in private, secure transactions while preserving the integrity of their assets, even in the face of potential key compromise.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iii-public-keys">III. Public keys<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#iii-public-keys" class="hash-link" aria-label="Direct link to III. Public keys" title="Direct link to III. Public keys">​</a></h3>
<p>Public keys in Nescience serve as the user's interface with the network, allowing for secure interaction and verification without exposing the user's private keys.
Derived directly from the user's private keys, public keys play a crucial role in enabling cryptographic operations such as transaction verification, commitment schemes,
and deterministic computations. The public key components correspond to their private key counterparts and ensure that transactions and commitments are securely processed
and validated across the network.</p>
<ol>
<li><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> (verifying Schnorr signatures)</strong>:</li>
</ol>
<p>The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> is derived from the signing component of the private key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>) and is used for verifying <strong>Schnorr signatures</strong>.
Schnorr signatures are used to authenticate transactions, ensuring that they have been signed by the legitimate owner of the private key. This public key is
essentially a verification key, allowing others in the network to confirm that a specific transaction was indeed authorized by the user. When a transaction is
broadcast to the network, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span> enables any participant to verify that the transaction’s signature matches the user’s private key without
needing access to the private key itself. This mechanism prevents forgeries as only the legitimate owner with access to the private key can generate a valid Schnorr signature.
For example, if Alice sends a transaction, she signs it with her private key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>). Bob, or any other network participant, can use Alice’s <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>
to verify the signature. If the signature is valid, Bob can be confident that the transaction was authorized by Alice and not by an imposter.</p>
<ol start="2">
<li><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> (commitment schemes)</strong></li>
</ol>
<p>The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> is derived from the commitment component of the private key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>a</mi><mi>t</mi><mi>e</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{private}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0398em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span>). It is used in the <strong>commitment schemes</strong>
that underpin Nescience’s privacy-preserving architecture. Commitments are a crucial cryptographic technique that allows users to commit to a piece of data (such as a UTXO)
without revealing the actual data, while still enabling proof of its integrity and existence. In Nescience, the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> is used as part of the Pedersen commitment scheme,
where it functions as a public commitment to certain transaction details. Even though the actual values are hidden (thanks to the private key component), the commitment can
still be verified by other network participants using <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span>. This enables secure and private transactions while maintaining the ability to verify that commitments
are consistent with the original data. For instance, when Alice commits to a UTXO, she uses her private key to generate the commitment, and the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span> is available
to others to verify the commitment’s validity without revealing the underlying details.</p>
<ol start="3">
<li><strong><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span> (pseudorandom function)</strong></li>
</ol>
<p>The <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span> is derived from a random field element within the private key and is used to generate the <strong>pseudorandom function (PRF)</strong> associated with the user's account.
This PRF is essential for producing deterministic outputs based on the user’s keys and transaction data while ensuring that these outputs are unique to the user and cannot be
predicted or replicated by others. The PRF is crucial in scenarios where the user needs to derive unique identifiers or values that are tied to their specific account,
ensuring that these values remain consistent across different transactions or interactions without revealing sensitive information. For example, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span> may be
used in generating deterministic yet secure addresses or transaction references, which can be linked to the user’s activity in a controlled manner. By using <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span>,
Nescience ensures that certain operations, like generating addresses or computing deterministic transaction outcomes, remain both private and cryptographically secure. The public key’s
role in this process is to maintain consistency in these outputs while preventing unauthorized parties from reverse engineering the associated private keys or transaction data.</p>
<p><strong>Summary</strong></p>
<p>Public keys in Nescience are essential for secure interactions within the network. "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>i</mi><mi>g</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.sig</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span></span></span></span>" allows others to verify that transactions were signed by the legitimate owner,
ensuring the authenticity of every operation. "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>r</mi><mi>c</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">{public}_{key}.rcm</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0747em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">rc</span><span class="mord mathnormal">m</span></span></span></span>" enables secure and private commitment schemes, allowing participants to commit to transaction details without
revealing sensitive information. Finally, "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span>" powers deterministic outputs through a pseudorandom function, ensuring that user-specific data remains consistent
and secure throughout various transactions. Together, these public key components facilitate privacy, security, and trust within NSSA, enabling seamless interactions while safeguarding user data.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="iv-viewing-key">IV. Viewing key<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#iv-viewing-key" class="hash-link" aria-label="Direct link to IV. Viewing key" title="Direct link to IV. Viewing key">​</a></h3>
<p>The <strong>viewing key</strong> in NSSA is a specialized cryptographic key that allows a user to decrypt both incoming and outgoing transactions associated with their account.
This key is designed to offer a degree of transparency to the user, enabling them to view the details of their transactions without compromising the security of their assets or granting
control over those assets.</p>
<ul>
<li>
<p><strong>Role of the viewing key</strong>: The primary function of the viewing key is to provide visibility into transaction details while maintaining the integrity of private, shielded,
or deshielded transactions. It enables the user to see the specifics of the transactions they are involved in—such as amounts transferred, asset types, and metadata—without
exposing the sensitive transaction data to the broader network. For instance, if Alice has executed a private transaction with Bob, her viewing key allows her to decrypt and
review the details of the transaction, ensuring that everything was processed correctly. This ability to audit her own transactions helps Alice maintain confidence in the integrity
of her private interactions on the blockchain.</p>
</li>
<li>
<p><strong>Security considerations</strong>: Despite its utility, the viewing key must be handled with care as its exposure could potentially compromise the user’s privacy.
Although possessing the viewing key does <strong>not</strong> provide the ability to spend or sign transactions (that authority remains strictly with the spending key and private keys),
it does allow anyone with access to the viewing key to decrypt the details of the user’s private transactions. This means that if the viewing key is leaked or stolen,
the privacy guarantees of Nescience’s private, shielded, and deshielded executions could be undermined. Specifically, the viewing key could be used to link various transactions,
breaking the unlinkability of private transactions. For example, an attacker with access to the viewing key could decrypt past and future transactions, exposing the relationships
between different parties and transaction flows. To mitigate this risk, Nescience recommends that users treat their viewing key with the same level of protection as their private keys.
It should be stored securely in encrypted hardware wallets or other secure storage solutions to prevent unauthorized access.</p>
</li>
<li>
<p><strong>Balancing privacy and transparency</strong>: The viewing key provides an essential balance between privacy and transparency in NSSA. While it ensures that users
can monitor their transaction history and verify the details of their private transactions, it does so without compromising the control of their funds. This allows users to maintain
a transparent view of their interactions while keeping their assets secure. For example, if Alice is using shielded execution to transfer assets, her viewing key enables her to
audit the transaction without allowing anyone else, including Bob or external observers, to see the specific details unless they also have access to the viewing key. Moreover,
since the viewing key does not grant signing or spending authority, even if it were exposed, an attacker would still not be able to manipulate the user’s assets. However,
to maintain the unlinkability and confidentiality of private transactions, the viewing key must be kept secure at all times.</p>
</li>
<li>
<p><strong>Protecting transaction unlinkability</strong>: In private transactions, unlinkability is one of the core privacy guarantees. This property ensures that individual
transactions cannot be correlated with each other or linked to the same user unless that user chooses to reveal the connection. The viewing key must be carefully
protected to preserve this unlinkability, as its compromise could allow someone to map out a user’s private transaction history. For instance, in deshielded transactions,
the viewing key allows the user to see which private UTXOs were consumed and how the public state was modified. If the viewing key is compromised, an attacker could potentially
link private UTXOs across multiple transactions, unraveling the user’s privacy.</p>
</li>
</ul>
<p><strong>Conclusion</strong></p>
<p>The viewing key in Nescience is a powerful tool for providing insight into both incoming and outgoing transactions without granting control over assets. It allows users
to decrypt and verify their transaction details, maintaining transparency in their interactions. However, due to its potential to compromise privacy if exposed, the viewing
key must be handled with great care. Proper security measures are necessary to protect the viewing key, ensuring that the unlinkability of private, shielded, and deshielded
transactions remains intact. In this way, the viewing key offers a crucial balance between privacy and transparency within the Nescience ecosystem.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-v-ephemeral-key"><a id="key"></a> V. Ephemeral key<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#-v-ephemeral-key" class="hash-link" aria-label="Direct link to -v-ephemeral-key" title="Direct link to -v-ephemeral-key">​</a></h3>
<p>The ephemeral key is generated using a combination of the sender’s spending key and the UTXO's nullifier, ensuring that the key is unique to each transaction.
The process can be informally described as follows:</p>
<ol>
<li><strong>Ephemeral key generation</strong><br>
<!-- -->Let <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ρ</mi></mrow><annotation encoding="application/x-tex">\rho</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">ρ</span></span></span></span> denote the nullifier of the UTXO being consumed in the transaction. The sender uses the receiver’s public key component <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span>,
which is derived from the receiver’s private key, to compute an <strong>ephemeral secret key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>s</mi><mi>k</mi></mrow><annotation encoding="application/x-tex">esk</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">es</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span>)</strong>. The computation is based on the nullifier <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ρ</mi></mrow><annotation encoding="application/x-tex">\rho</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">ρ</span></span></span></span> and a base value:</li>
</ol>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>s</mi><mi>k</mi><mo>=</mo><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">(</mo><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>0</mn><mo separator="true">,</mo><mn>0</mn><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>ρ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">esk = {public}_{key}.sk(prf((0,0,0,0) || \rho)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">es</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mopen">((</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mclose">)</span><span class="mord">∣∣</span><span class="mord mathnormal">ρ</span><span class="mclose">)</span></span></span></span>
This formula binds the secret key to the specific transaction, leveraging the receiver’s cryptographic identity and the unique properties of the UTXO being spent.</p>
<ol start="2">
<li><strong>Deriving the ephemeral public key</strong><br>
<!-- -->After computing the ephemeral secret key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>s</mi><mi>k</mi></mrow><annotation encoding="application/x-tex">esk</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">es</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span>), the next step is to derive the corresponding <strong>ephemeral public key (epk)</strong>. This is done using the Key Agreement
Protocol's <strong>DerivePublic algorithm</strong>, which generates the public key associated with the shared secret key. The ephemeral public key is computed as:</li>
</ol>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>p</mi><mi>k</mi><mo>=</mo><mi>K</mi><mi>A</mi><mi mathvariant="normal">.</mi><mi>D</mi><mi>e</mi><mi>r</mi><mi>i</mi><mi>v</mi><mi>e</mi><mi>P</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi><mo stretchy="false">(</mo><mi>e</mi><mi>s</mi><mi>k</mi><mo separator="true">,</mo><mi>g</mi><mi>d</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">epk = KA.DerivePublic(esk, gd)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mord mathnormal">A</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.02778em">Der</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.13889em">P</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span><span class="mopen">(</span><span class="mord mathnormal">es</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">d</span><span class="mclose">)</span></span></span></span></p>
<p>Here, (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">gd</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">d</span></span></span></span>) is the <strong>diversifier address</strong> associated with the receiver’s account. The diversifier address is computed from the receiver’s
account using the <strong>DiversifierHash</strong> function:</p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>g</mi><mi>d</mi><mo>=</mo><mi>r</mi><mi>e</mi><mi>c</mi><mi>e</mi><mi>i</mi><mi>v</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">.</mi><mi>D</mi><mi>i</mi><mi>v</mi><mi>e</mi><mi>r</mi><mi>s</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mi>H</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>d</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">gd = receiver.DiversifierHash(d)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">d</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">rece</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord">.</span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">ers</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.08125em">erH</span><span class="mord mathnormal">a</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal">d</span><span class="mclose">)</span></span></span></span></p>
<p>The diversifier (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">d</span></span></span></span>) is a random value selected by the sender to add randomness to the process. This diversifier ensures that even if a single receiver is involved
in multiple transactions, the derived keys remain distinct for each transaction. The value (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">d</span></span></span></span>) is included in the transaction note for transparency and reproducibility.</p>
<ol start="3">
<li><strong>Establishing the shared secret</strong><br>
<!-- -->The shared secret, used to encrypt the transaction details, is derived from the key agreement between the sender’s ephemeral key and the receiver’s viewing key.
Any party possessing the receiver’s viewing key can use it in conjunction with the ephemeral key to compute the shared secret, which is then used to decrypt the transaction.
This ensures that only the intended recipient (or anyone with their viewing key) can access the transaction details.</li>
</ol>
<p><strong>Key components and protocol</strong></p>
<p>The formal protocol for generating ephemeral keys closely follows this informal description but involves additional intermediate steps for converting values to
binary sequences to fit implementation requirements. These steps are essential for ensuring compatibility with cryptographic algorithms used in NSSA.
The protocol uses the following key components:</p>
<ul>
<li><strong>Nullifier (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ρ</mi></mrow><annotation encoding="application/x-tex">\rho</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal">ρ</span></span></span></span>):</strong> Ensures that the ephemeral key is tied to the specific UTXO being consumed, preventing reuse of the key in future transactions.</li>
<li><strong>Receiver’s public key (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>p</mi><mi>u</mi><mi>b</mi><mi>l</mi><mi>i</mi><mi>c</mi></mrow><mrow><mi>k</mi><mi>e</mi><mi>y</mi></mrow></msub><mi mathvariant="normal">.</mi><mi>s</mi><mi>k</mi><mo stretchy="false">(</mo><mi>p</mi><mi>r</mi><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{public}_{key}.sk(prf)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1302em;vertical-align:-0.3802em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">p</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">i</span><span class="mord mathnormal">c</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.242em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">ey</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.3802em"><span></span></span></span></span></span></span><span class="mord">.</span><span class="mord mathnormal">s</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mopen">(</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mclose">)</span></span></span></span>:</strong> Establishes the receiver's identity in the key generation process, ensuring that the shared secret can
only be derived by the intended party.</li>
<li><strong>Diversifier (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">d</span></span></span></span>):</strong> Adds randomness to the transaction, ensuring that keys remain unique across different transactions involving the same receiver.</li>
</ul>
<p>The end result is an ephemeral key system that provides strong cryptographic guarantees for transaction privacy, leveraging key agreement protocols and secure
cryptographic primitives to prevent unauthorized access to sensitive transaction data.</p>
<p><strong>Conclusion</strong></p>
<p>The ephemeral key in Nescience is a critical element for maintaining transaction confidentiality. It facilitates a secure key agreement between the sender and the receiver,
allowing for the encryption of transaction details with a shared secret that can only be derived by the intended recipient. By incorporating the nullifier, receiver's public key,
and diversifier address, the ephemeral key ensures that transaction privacy is preserved while preventing unauthorized access to transaction information, even in a complex,
multi-party blockchain environment.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="vi-nescience-addresses">VI. Nescience addresses<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#vi-nescience-addresses" class="hash-link" aria-label="Direct link to VI. Nescience addresses" title="Direct link to VI. Nescience addresses">​</a></h3>
<p>Nescience’s dual address system is a core component of its privacy-focused architecture, designed to balance transparency and confidentiality across different types of transactions.
The architecture provides each user or smart contract with both public addresses and private addresses, allowing them to participate in both open and confidential activities on the blockchain.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="a-public-addresses">a) Public addresses<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#a-public-addresses" class="hash-link" aria-label="Direct link to a) Public addresses" title="Direct link to a) Public addresses">​</a></h4>
<p>Public addresses in Nescience are visible to all participants on the network and reside within the public state. These addresses are essential for engaging in
transparent and verifiable interactions, such as sending tokens or invoking smart contracts that are meant to be publicly auditable. Public addresses serve as
the interface for users who need to engage with the transparent elements of the system, including public transactions or smart contracts that require public access.</p>
<p>They are analogous to traditional blockchain addresses seen in systems like Ethereum or Bitcoin, where every participant can see the address and the transactions associated with it.
For example, when Alice wants to receive tokens from Bob in a public transaction, she can provide her public address, allowing Bob to send the tokens transparently.
Anyone on the network can verify the transaction, providing accountability and trust in the public state.</p>
<p>Because public addresses are visible and auditable, they are typically used for interactions where privacy is not a concern or where transparency is desirable.
This could include simple token transfers, public contract calls, or interactions with dapps that require public accountability,
such as voting or governance systems.</p>
<h4 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="b-private-addresses">b) Private addresses<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#b-private-addresses" class="hash-link" aria-label="Direct link to b) Private addresses" title="Direct link to b) Private addresses">​</a></h4>
<p>In contrast, private addresses are designed for confidentiality and are not visible onchain. These addresses are used exclusively for private transactions and executions,
ensuring that sensitive details—such as the sender, receiver, or amount transferred—remain hidden from the public state. Private addresses are a key feature of
Nescience’s private, shielded, and deshielded execution models, where preserving the confidentiality of participants is crucial.</p>
<p>Users can generate an unlimited number of private addresses using their private keys. This flexibility allows users to compartmentalize their interactions,
giving them the ability to provide different private addresses to different parties. For instance, Alice could create a unique private address for each entity
she interacts with, thereby ensuring that her transactions remain isolated and difficult to trace. This feature enhances privacy by preventing any direct linkage
between different transactions or activities associated with a single user.</p>
<p>Private addresses are not tied to the public state and are only accessible through the user’s private key infrastructure. Transactions involving private addresses
are conducted within the confines of the private state and are only decrypted by the intended participants. For example, when Alice sends tokens to Bob using
a private address, the details of that transaction remain confidential, accessible only to Alice and Bob, unless they choose to reveal it.</p>
<p><strong>Role of the viewing key in private addresses</strong>: A key feature of Nescience’s private address system is the viewing key, which allows users to decrypt any transaction
involving their private addresses. This capability provides oversight and transparency into the user’s private transactions, ensuring that they can monitor their own
activity without exposing the details to the public. The viewing key does not compromise the security of the user's assets as it does not grant spending or signing authority.
However, it does allow the user to audit and verify the accuracy of their private transactions, ensuring that everything proceeds as expected. For instance, Alice can use her
viewing key to review the details of a private transaction she conducted with Bob, ensuring that the correct amount was transferred and that the transaction was properly processed.
This functionality is critical for users who want to maintain control over their private interactions while still benefiting from transparency into their transaction history.
The ability to generate multiple private addresses and decrypt them with the viewing key ensures that users can maintain compartmentalized privacy without sacrificing oversight.</p>
<p><strong>Summary</strong></p>
<p>Nescience’s dual address system—comprising public and private addresses—provides users with the flexibility to engage in both transparent and confidential transactions.
Public addresses are visible onchain and are used for open, public interactions that require accountability and auditability. In contrast, private addresses are
invisible onchain and are used for confidential transactions, enhancing privacy and security.</p>
<p>By allowing users to generate multiple private addresses, Nescience gives individuals control over the visibility of their transactions. Combined with the viewing
key’s ability to decrypt transactions involving private addresses, the system ensures that users can maintain transparency over their private transactions without
exposing sensitive information to the public state. This dual-address approach enables users to seamlessly switch between public and private interactions depending on their needs,
providing a robust framework for both privacy and transparency in NSSA.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="vii-conclusion">VII. Conclusion<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#vii-conclusion" class="hash-link" aria-label="Direct link to VII. Conclusion" title="Direct link to VII. Conclusion">​</a></h3>
<p>Key management in NSSA is a carefully designed system that strikes an optimal balance between security, privacy, and flexibility.
The architecture’s hierarchical structure, with distinct roles for the spending key, private keys, and public keys, ensures that users retain full control
over their assets while maintaining the integrity of their transactions. The spending key, as the root of security, provides unassailable control over the
user's UTXOs and assets, ensuring that only the rightful owner can authorize spending. Private keys, derived from the spending key, enable users to engage
in cryptographic operations such as signing transactions and generating commitments without exposing sensitive information to the network.</p>
<p>The viewing key adds another layer of transparency, allowing users to decrypt and review their transactions without compromising their authority over their assets.
While it provides a window into transaction history, the viewing key does not grant spending power, preserving the critical separation between visibility and control.</p>
<p>The dual system of public and private addresses gives users the flexibility to navigate between open, transparent transactions and confidential, privacy-protected activities.
Public addresses allow users to engage in verifiable, public interactions while private addresses enable compartmentalized, secure transactions that remain hidden
from the public eye. This dual-address framework ensures that users can seamlessly adapt to different privacy requirements, whether they are participating in public
dapps or conducting sensitive financial operations.</p>
<p>Overall, Nescience’s cryptographic infrastructure is designed to empower users to engage confidently in both transparent and confidential activities.
By providing flexible, secure key management and address systems, Nescience ensures that users can fully participate in the blockchain ecosystem without
compromising their privacy or control. The architecture supports the nuanced needs of modern blockchain users, who require both the transparency of public
interactions and the security of private transactions, all while maintaining the integrity and confidentiality of their assets.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="c-trees-in-nssa">c) Trees in NSSA<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#c-trees-in-nssa" class="hash-link" aria-label="Direct link to c) Trees in NSSA" title="Direct link to c) Trees in NSSA">​</a></h2>
<p>Trees in NSSA serve as verifiable databases, essential for maintaining privacy and security. Different types of trees are used for various purposes:</p>
<ol>
<li>
<p><strong>Global state tree:</strong> The global state tree is a single, public tree that holds all public assets and storage information. It acts as a central repository for all
publicly accessible data on the blockchain. By organizing this data in a Merkle tree structure, the Global State Tree allows for efficient and secure verification of public information.</p>
</li>
<li>
<p><strong>Hashed UTXO tree:</strong> The hashed UTXO tree is a public tree that contains hashes of all created UTXOs. When users wish to consume a UTXO, they provide a membership
proof to demonstrate that the UTXO exists within this tree. This process ensures that only valid and existing UTXOs can be spent, maintaining the integrity of transactions.
In fact, users generate membership proofs that verify the presence of specific UTXOs in the tree without revealing their actual data. The benefit here is that the Merkle
tree structure allows for quick and efficient verification of UTXO existence.</p>
</li>
<li>
<p><strong>UTXO trees (private states):</strong> Each user or smart contract has its private state stored in UTXO trees. These trees are kept as plaintext on the client’s
local system (off-chain), ensuring privacy as sensitive information remains confidential. The private state includes all UTXOs owned by the user or the smart contract, and these
are not directly exposed to the public blockchain. For instance, users have full control over their private state, which is not visible to other participants in the network.</p>
</li>
</ol>
<p>In conclusion, the tree structures enable efficient verification of transaction validity without compromising privacy. By using Merkle trees,
Nescience ensures that any tampering with the data can be easily detected. The efficient structure of these trees supports the scalability of the architecture,
allowing it to handle a large number of transactions and data entries. By leveraging different types of trees, Nescience ensures efficient and secure management
of both public and private states.</p>
<p>##<a id="nul"></a> d) Nullifier tree in Nescience</p>
<p>The <strong>nullifier tree</strong> is a fundamental component of NSSA, designed to prevent double spending by securely tracking all consumed UTXOs.
This tree acts as a public ledger of spent UTXOs, ensuring that once a UTXO is consumed in a transaction, it cannot be reused in future transactions.</p>
<p>The primary function of the nullifier Tree is to store the <strong>nullifiers</strong> of all consumed UTXOs. By recording the nullifiers in a public tree,
the system ensures that each UTXO is spent only once, thereby safeguarding the integrity of the entire network.</p>
<ul>
<li>
<p><strong>Ensuring non-membership and preventing double spending</strong>
Before a user can consume a UTXO in a transaction, they must provide a <strong>non-membership proof</strong>. This proof demonstrates that the UTXO’s nullifier
does not already exist in the Nullifier Tree, proving that the UTXO has not been spent before. If the UTXO’s nullifier is found in the tree,
the system will reject the transaction, preventing double spending. The non-membership proof ensures that users cannot attempt to spend the
same UTXO in multiple transactions. This mechanism is critical for maintaining the security and reliability of NSSA.
The tree structure, which is typically built using a cryptographic tree like a Merkle tree, allows for efficient verification of nullifiers.
Verifiers can quickly check whether a nullifier is present or absent in the tree, ensuring that each UTXO is only spent once.</p>
</li>
<li>
<p><strong>Nullifier tree structure and operation</strong>
The nullifier tree is likely structured as a <strong>Merkle tree</strong>, which is a cryptographic binary tree where each node represents the hash of its child nodes.
This structure allows for efficient storage and verification of large sets of nullifiers as only the root hash of the tree needs to be stored on the blockchain.
When a new nullifier is added to the tree, the tree is recalculated, and the root hash is updated. This process ensures that all consumed UTXOs are securely recorded.
Each time a transaction consumes a UTXO, the nullifier is added to the Nullifier Tree, and the tree is updated to reflect this new entry. To verify that a
UTXO has not been double spent, verifiers can use the tree’s root hash and a proof of inclusion or exclusion (membership or non-membership proof) to check whether the
nullifier is present in the tree. For example, if Alice wants to spend a UTXO, she must prove that the nullifier associated with that UTXO is not already in the Nullifier Tree.
She generates a non-membership proof that shows her nullifier is not recorded in the tree, and the transaction is allowed to proceed. Once the transaction is completed,
the nullifier is added to the tree, ensuring that the UTXO cannot be used again.</p>
</li>
</ul>
<p><strong>Conclusion</strong>
The Nullifier Tree is a crucial element of Nescience's security. By recording all consumed UTXOs and ensuring that nullifiers are unique, the tree prevents double spending
and maintains the integrity of the blockchain. The non-membership proof mechanism guarantees that every transaction is validated against the tree. This structure supports
the scalability and security of NSSA, providing a reliable method for verifying the validity of transactions while preventing malicious behavior.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="e-recursive-friendly-privacy-preserving-zk-zkvm">e) Recursive-friendly privacy-preserving zk-zkVM<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#e-recursive-friendly-privacy-preserving-zk-zkvm" class="hash-link" aria-label="Direct link to e) Recursive-friendly privacy-preserving zk-zkVM" title="Direct link to e) Recursive-friendly privacy-preserving zk-zkVM">​</a></h2>
<p>The development of the zk-zkVM in Nescience is a work in progress, as the architecture continues to evolve to support privacy-preserving transactions
and efficient ZKP generation. The goal of the zk-zkVM is to seamlessly integrate with the Nescience state-separation architecture,
ensuring that private transactions remain confidential while allowing the network to verify their validity without compromising privacy.</p>
<p>Currently, we are exploring and testing several existing zkVMs to identify the most suitable platform for our needs. Our focus is on finding a zkVM
that not only supports the core features of Nescience, such as state separation and privacy, but also provides the efficiency and scalability required
for a decentralized system. Once a suitable zkVM is chosen, we will begin implementing advanced privacy features on top of it, including support for
confidential transactions, selective disclosure, and recursive proof aggregation.</p>
<p>The integration of these privacy-preserving features with an existing zkVM will enable Nescience to fully employ its state-separation architecture,
ensuring that users can conduct private transactions with robust security and scalability. This approach will allow us to leverage the strengths of
proven zkVM technologies while enhancing them with the unique privacy and state-separation capabilities that Nescience requires.</p>
<ul>
<li>
<p><strong>Privacy-preserving features</strong>: At its core, the zk-zkVM is designed with privacy in mind. One of the zk-zkVM’s standout privacy features is <strong>selective disclosure</strong>,
which allows users to reveal only specific details of a transaction as needed. For example, a user could disclose the transaction amount while concealing the identities
of the participants. The zk-zkVM employs advanced encryption techniques to protect this sensitive data. All transaction data is encrypted before being stored on the blockchain,
so even if the data is intercepted, it cannot be deciphered without the appropriate decryption keys. Another of the crucial privacy-preserving features is the support
for <strong>confidential transactions</strong>. Only the parties involved in the transaction can access the encrypted data. Furthermore, the zk-zkVM supports <strong>verifiable encryption</strong>,
a powerful capability that allows encrypted data to be included in ZKPs without needing to decrypt it. This ensures that transaction details remain private
while their correctness can still be proven.</p>
</li>
<li>
<p><strong>Lightweight design for accessibility</strong>: The zk-zkVM is being designed to be lightweight and efficient, enabling it to run on standard consumer-grade hardware.
This makes it accessible to a wide range of users without requiring specialized equipment or significant computational resources.</p>
</li>
<li>
<p><strong>Faster proving time</strong>: To maintain a seamless user experience, especially during high transaction volumes, the zk-zkVM is being optimized for <strong>fast proving times</strong>.
Fast proof generation is particularly important for ensuring that the system remains usable during periods of peak activity, preventing bottlenecks and maintaining the fluidity of the network.</p>
</li>
<li>
<p><strong>Recursive-friendly operations</strong>: One of the most advanced features of the zk-zkVM will be its support for <strong>recursive operations</strong>. Recursion enables the aggregation
of multiple proofs into a single proof, improving efficiency on both the client and sequencer sides of the network.</p>
</li>
<li>
<p><strong>Client-side recursion (batch processing):</strong> When a single transaction involves multiple executions, each requiring its own ZKP, these individual
proofs can be recursively aggregated before being sent to the sequencer. This reduces the overall data transmitted, enhancing the efficiency of the transaction process
by compressing multiple proofs into a single package.</p>
</li>
<li>
<p><strong>Sequencer-side recursion (reduced redundancy):</strong> The sequencer, which is responsible for processing transactions and creating verifiable blocks, collects transactions
containing aggregated proofs. These proofs are further merged into a single comprehensive proof, ensuring that all transactions within a block are validated collectively.
This process reduces redundancy and optimizes the blockchain’s efficiency by minimizing the size and complexity of the proofs required for verification.</p>
</li>
<li>
<p><strong>Developer-friendly language</strong>: To foster widespread adoption and innovation within the Nescience ecosystem, the zk-zkVM would include a <strong>developer-friendly language</strong>.
This high-level language simplifies the process of building applications that leverage state separation and privacy-preserving transactions. The language should offer extensive
support for modular design, APIs, and SDKs, enabling developers to integrate their applications with the zk-zkVM more easily. By lowering the barrier to entry, Nescience encourages
innovation and helps expand the range of privacy-preserving applications that can be built on its platform.</p>
</li>
</ul>
<p><strong>Conclusion</strong></p>
<p>The zk-zkVM in Nescience is a powerful and versatile virtual machine that embodies the principles of privacy, efficiency, and scalability. By supporting ZKPs
and integrating with advanced privacy technologies like homomorphic encryption. Its lightweight design allows it to run efficiently on standard hardware, promoting decentralization,
and its recursive operations further enhance the system's scalability. With its developer-friendly language and fast proving times, the zk-zkVM is positioned as a key component in
fostering the growth and adoption of privacy-preserving blockchain applications.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="f-mpc-based-synchronization-mechanism-under-review">f) MPC-based synchronization mechanism (under review)<a href="https://vac.dev/rlog/Nescience-state-separation-architecture#f-mpc-based-synchronization-mechanism-under-review" class="hash-link" aria-label="Direct link to f) MPC-based synchronization mechanism (under review)" title="Direct link to f) MPC-based synchronization mechanism (under review)">​</a></h2>
<p>Nescience is developing an <strong>MPC-based</strong> synchronization mechanism to balance privacy and fairness between public and private execution types.
This mechanism extracts common information from encrypted UTXOs without revealing private details, ensuring privacy and preventing UTXO linkage to users or specific transactions.
It guarantees that public and private executions remain equitable, with the total input equaling the public output.</p>
<p>The mechanism employs <strong>MPC protocols</strong> to perform computations privately, <strong>ZKPs</strong> to verify correctness, and <strong>cryptographic protocols</strong>
to secure data during synchronization. This ensures a consistent and fair environment for all users, regardless of their chosen privacy level. Currently,
this feature is under development and review for potential inclusion depending on the research output and compatibility.</p>
<h1>D. Future plans for Nescience</h1>
<p>Nescience is committed to continuously evolving its architecture to ensure scalability, privacy, and security in a growing blockchain landscape.
One of the primary goals is to integrate the <strong>zk-zkVM</strong> and the <strong>Nescience state-separation architecture</strong> into a fully functioning node,
enabling efficient private transactions while maintaining network integrity.</p>
<ul>
<li><strong>Addressing scalability challenges</strong>: A key challenge facing Nescience is the increasing size of nullifier and hashed UTXO trees, which could impact
network performance and scalability over time. To mitigate this, Nescience plans to adopt state-of-the-art scalable privacy techniques such as:<!-- -->
<ul>
<li><strong>Mutator sets:</strong> Dynamically adjusting data structures to manage the growth of the nullifier set efficiently.</li>
<li><strong>SNARK-based accumulators:</strong> Compressing data in a verifiable way to ensure that only relevant information is stored while maintaining cryptographic security.</li>
<li><strong>Pruning techniques:</strong> Periodically trimming unnecessary data from trees to maintain optimal size and performance, ensuring that the network scales logarithmically
rather than exponentially as more transactions occur.</li>
</ul>
</li>
</ul>
<p>By implementing these approaches, Nescience aims to keep the size of its data structures manageable, ensuring that scalability does not come at the cost of performance or privacy.</p>
<ul>
<li>
<p><strong>Enhanced key management</strong>: Another critical focus for Nescience is improving key management to streamline operations and enhance security.
The plan is to integrate the different keys used for signatures, addresses, UTXO encryption, and SNARK verification into a unified system.
This integration will simplify key management for users while reducing the risk of security breaches caused by complex, disparate key systems.
Nescience also plans to implement <strong>Hierarchical Deterministic (HD) keys</strong>, which allow users to derive multiple keys from a single seed,
enhancing both security and usability. This approach reduces the complexity of managing multiple keys across various functions and provides an additional
layer of protection for private transactions. Additionally, <strong>multi-signature schemes</strong> will be introduced, requiring multiple parties to authorize transactions.
This feature increases security by reducing the likelihood of unauthorized access, ensuring that a single compromised key cannot lead to malicious transactions.</p>
</li>
<li>
<p><strong>Integrating advanced cryptographic techniques</strong>: Nescience will integrate advanced cryptographic techniques, enhancing both privacy and scalability. Among these are:</p>
<ul>
<li><strong>Homomorphic encryption:</strong> Allowing computations to be performed on encrypted data without the need to decrypt it, preserving privacy while enabling secure, complex data processing.</li>
<li><strong>Zero-knowledge rollups:</strong> Bundling multiple transactions into a single proof to reduce the amount of data processed and stored on the blockchain,
significantly improving scalability without sacrificing security.</li>
</ul>
</li>
</ul>
<p>These cryptographic enhancements will ensure that Nescience can support a growing network while continuing to protect user privacy and maintaining high transaction throughput.</p>
<ul>
<li><strong>Long-term vision</strong></li>
</ul>
<p>The ultimate goal for Nescience is to deploy a fully operational <strong>node powered by zk-zkVM</strong> and the <strong>Nescience state-separation architecture</strong>.
This node will handle complex, private transactions at scale while integrating all of the advanced cryptographic techniques outlined in the roadmap.
Nescience aims to provide users with an infrastructure that balances privacy, security, and efficiency, ensuring the network remains resilient and capable of handling future demands.</p>
<p>By pursuing these future plans, Nescience is poised to not only address current challenges around scalability and key management but also lead the way in
applying advanced cryptography to decentralized systems. This vision will help secure the long-term integrity and performance of the Nescience state-separation
model as the blockchain grows and evolves.</p>
<h1>References</h1>
<p>[1] Nakamoto, S. (2008). Bitcoin: A Peer-to-Peer Electronic Cash System. Retrieved from <a href="https://bitcoin.org/bitcoin.pdf" target="_blank" rel="noopener noreferrer">https://bitcoin.org/bitcoin.pdf</a></p>
<p>[2] Sanchez, F. (2021). Cardano’s Extended UTXO accounting model. Retrieved from <a href="https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/" target="_blank" rel="noopener noreferrer">https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/</a></p>
<p>[3] Morgan, D. (2020). HD Wallets Explained: From High Level to Nuts and Bolts. Retrieved from <a href="https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14" target="_blank" rel="noopener noreferrer">https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14</a></p>
<p>[4] Wuille, P. (2012). Bitcoin Improvement Proposal (BIP) 32. Retrieved from <a href="https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki" target="_blank" rel="noopener noreferrer">https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki</a></p>
<p>[5] Sin7y Tech Review (29): Design Principles of Private Transactions in Aleo &amp; Zcash. Retrieved from <a href="https://hackmd.io/@sin7y/rkxFXLkgs" target="_blank" rel="noopener noreferrer">https://hackmd.io/@sin7y/rkxFXLkgs</a></p>
<p>[6] Sin7y Tech Review (33): Principles of private transactions and regulatory compliance issues. Retrieved from <a href="https://hackmd.io/@sin7y/S16RyFzZn" target="_blank" rel="noopener noreferrer">https://hackmd.io/@sin7y/S16RyFzZn</a></p>
<p>[7] Zcash Protocol Specification. Retrieved from <a href="https://zips.z.cash/protocol/protocol.pdf" target="_blank" rel="noopener noreferrer">https://zips.z.cash/protocol/protocol.pdf</a></p>
<p>[8] Anatomy of a Zcash Transaction. Retrieved from <a href="https://electriccoin.co/blog/anatomy-of-zcash" target="_blank" rel="noopener noreferrer">https://electriccoin.co/blog/anatomy-of-zcash</a></p>
<p>[9] The Penumbra Protocol: Notes, Nullifiers, and Trees. Retrieved from <a href="https://protocol.penumbra.zone/main/concepts/notes_nullifiers_trees.html" target="_blank" rel="noopener noreferrer">https://protocol.penumbra.zone/main/concepts/notes_nullifiers_trees.html</a></p>
<p>[10] Zero-knowledge Virtual Machine (ZKVM). Retrieved from <a href="https://medium.com/@abhilashkrish/zero-knowledge-virtual-machine-zkvm-95adc2082cfd" target="_blank" rel="noopener noreferrer">https://medium.com/@abhilashkrish/zero-knowledge-virtual-machine-zkvm-95adc2082cfd</a></p>
<p>[11] What's a Sparse Merkle tree?. Retrieved from <a href="https://medium.com/@kelvinfichter/whats-a-sparse-merkle-tree-acda70aeb837" target="_blank" rel="noopener noreferrer">https://medium.com/@kelvinfichter/whats-a-sparse-merkle-tree-acda70aeb837</a></p>
<p>[12] Lecture 10: Accounts Model and Merkle Trees. Retrieved from <a href="https://web.stanford.edu/class/ee374/lec_notes/lec10.pdf" target="_blank" rel="noopener noreferrer">https://web.stanford.edu/class/ee374/lec_notes/lec10.pdf</a></p>
<p>[13] The UTXO vs Account Model. Retrieved from <a href="https://www.horizen.io/academy/utxo-vs-account-model/" target="_blank" rel="noopener noreferrer">https://www.horizen.io/academy/utxo-vs-account-model/</a></p>
<p>[14] Addresses and Value Pools in Zcash. Retrieved from <a href="https://zcash.readthedocs.io/en/latest/rtd_pages/addresses.html" target="_blank" rel="noopener noreferrer">https://zcash.readthedocs.io/en/latest/rtd_pages/addresses.html</a></p>]]></content>
<author>
<name>Moudy</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Vac 101: Membership with Bloom Filters and Cuckoo Filters]]></title>
<id>https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters</id>
<link href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters"/>
<updated>2024-07-19T12:00:00.000Z</updated>
<summary type="html"><![CDATA[We examine two data structures: Bloom filters and Cuckoo filters.]]></summary>
<content type="html"><![CDATA[<p>We examine two data structures: Bloom filters and Cuckoo filters.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="membership-with-bloom-filters-and-cuckoo-filters">Membership with Bloom Filters and Cuckoo Filters<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#membership-with-bloom-filters-and-cuckoo-filters" class="hash-link" aria-label="Direct link to Membership with Bloom Filters and Cuckoo Filters" title="Direct link to Membership with Bloom Filters and Cuckoo Filters">​</a></h2>
<p>The ability to efficiently query the membership of an element in a given data set is crucial.
In certain applications, it is more important to output a result quickly than to have a 'perfect' result.
In particular, false positives may be an acceptable tradeoff for speed.
In this blog, we examine <a href="https://dl.acm.org/doi/10.1145/362686.362692" target="_blank" rel="noopener noreferrer">Bloom</a> and <a href="https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf" target="_blank" rel="noopener noreferrer">Cuckoo</a> data filters.
Both of these filters are data structures that can be used for membership proofs.</p>
<p>Everyone is familiar with the process of creating a new account for various websites, whether it is an e-mail account or a social media account.
Consider when you enter your desired username.
Many sites provide real-time feedback, as you type, on the availability of a given string.
In this scenario, it is necessary that the result is seemingly instant, regardless of the number of existing accounts.
However, it is not important that the usernames that are flagged as unavailable are, in fact, in use.
That is, it is sufficient to have a probabilistic check for membership.</p>
<p><strong>Bloom filters</strong> and <strong>Cuckoo filters</strong> are data structures that can be used to accumulate data with a fixed amount of space.
The associated filter <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>F</mi></mrow><annotation encoding="application/x-tex">F</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.13889em">F</span></span></span></span> for a digest of data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> can be queried to determine whether an element is (possibly) a member of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>:</p>
<ul>
<li><strong>0:</strong> The queried element is definitely not a member of digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.</li>
<li><strong>1:</strong> The entry is possibly a member of the digest <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>.</li>
</ul>
<p>The algorithms associated with Bloom filters and Cuckoo filters, which we will discuss shortly, are deterministic.
The possibility of false positives arises from the query algorithm.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="bloom-filters">Bloom filters<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#bloom-filters" class="hash-link" aria-label="Direct link to Bloom filters" title="Direct link to Bloom filters">​</a></h2>
<p>A <strong>Bloom filter</strong> is a data structure that can be used to accumulate an arbitrary amount of data with a fixed amount of space.
Bloom filters have been a popular data structure for proof of non-membership due to their small storage size.
Specifically, a Bloom filter consists of a binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mi>n</mi></msup></mrow><annotation encoding="application/x-tex">{\bf{v}} \in \{0,1\}^n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> hash functions <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">{</mo><msub><mi>h</mi><mi>i</mi></msub><mo>:</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mo>∗</mo></msup><mo>→</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>n</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo><msubsup><mo stretchy="false">}</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow></msubsup></mrow><annotation encoding="application/x-tex">\{h_i: \{0,1\}^* \rightarrow \{0,\dots,n-1\}\}_{i=0}^{k-1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6887em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mbin mtight">∗</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">→</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.1661em;vertical-align:-0.2769em"></span><span class="mord">1</span><span class="mclose">}</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8892em"><span style="top:-2.4231em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">i</span><span class="mrel mtight">=</span><span class="mord mtight">0</span></span></span></span><span style="top:-3.1031em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mbin mtight">−</span><span class="mord mtight">1</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2769em"><span></span></span></span></span></span></span></span></span></span>.
We note that each hash function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>h</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">h_i</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> is used to determine an index of our binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi></mrow><annotation encoding="application/x-tex">{\bf{v}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4444em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span></span></span></span> to flip the associated bit to 1.
The binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi></mrow><annotation encoding="application/x-tex">{\bf{v}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4444em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span></span></span></span> is initialized with every entry as 0.
The hash functions do not need to be cryptographic hash functions.</p>
<ul>
<li>
<p><strong>Append:</strong> Suppose that we wish to add the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> to the Bloom filter.</p>
<ul>
<li>Define the vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">b</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>n</mi><mo>−</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">{\bf{b}} \in \{0,\dots,n-1\}^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7335em;vertical-align:-0.0391em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">b</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo><mo>:</mo><mo>=</mo><msub><mi>h</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{\bf{b}}[i] := h_i(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mopen">[</span><span class="mord mathnormal">i</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>.</li>
<li>Update the binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mi mathvariant="bold">b</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo><mo stretchy="false">]</mo><mo>←</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[{\bf{b}}[i]] \leftarrow 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mopen">[</span><span class="mord mathnormal">i</span><span class="mclose">]]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>.</li>
</ul>
</li>
<li>
<p><strong>Query:</strong> Suppose that we wish to query the Bloom filter for element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span>.</p>
<ul>
<li>Return 1 provided <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><msub><mi>h</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[h_i(y)] = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> for every <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>. Otherwise, return 0.</li>
</ul>
</li>
</ul>
<p>The algorithm <strong>Query</strong> will output 1 for every element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> that has been added to the Bloom filter.
This is a consequence of the <strong>Append</strong> algorithm.
However, due to potential collisions over a set of hash functions, it is possible for false positives to occur.
Moreover, the possibility of collisions makes it impossible to remove elements from the Bloom filter.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="complexity">Complexity<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#complexity" class="hash-link" aria-label="Direct link to Complexity" title="Direct link to Complexity">​</a></h3>
<p>The storage of a Bloom filter requires constant space.
Specifically, the Bloom filter uses <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> bits regardless of the size of the digest.
So, regardless of the number of elements that we append, the Bloom filter will use <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> bits.
Further, if we assume that each of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> hash functions runs in constant time, then we can append/query an entry in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(k)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.02778em">O</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mclose">)</span></span></span></span>.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="example">Example<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#example" class="hash-link" aria-label="Direct link to Example" title="Direct link to Example">​</a></h3>
<p>Suppose that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">k = 3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">3</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>=</mo><mn>10</mn></mrow><annotation encoding="application/x-tex">n = 10</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">10</span></span></span></span>.
Our Bloom filter is initialized as <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo>=</mo><mrow><mo fence="true">(</mo><mtable rowspacing="0.16em" columnalign="center center center center center center center center center center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd></mtr></mtable><mo fence="true">)</mo></mrow><mi mathvariant="bold">.</mi></mrow><annotation encoding="application/x-tex">\bf{v} = \begin{pmatrix}0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\end{pmatrix}.</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2em;vertical-align:-0.35em"></span><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="minner"><span class="mopen delimcenter" style="top:0em"><span class="delimsizing size1">(</span></span><span class="mord"><span class="mtable"><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span></span></span><span class="mclose delimcenter" style="top:0em"><span class="delimsizing size1">)</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathbf">.</span></span></span></span></span>
Now, we will append the words <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span>.
Suppose that</p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtable rowspacing="0.16em" columnalign="center center center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mi>a</mi><mi>d</mi><mi>d</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>1</mn></msub><mo stretchy="false">(</mo><mi>a</mi><mi>d</mi><mi>d</mi><mo stretchy="false">)</mo><mo>=</mo><mn>4</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>2</mn></msub><mo stretchy="false">(</mo><mi>a</mi><mi>d</mi><mi>d</mi><mo stretchy="false">)</mo><mo>=</mo><mn>7</mn></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>m</mi><mo stretchy="false">)</mo><mo>=</mo><mn>9</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>1</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>m</mi><mo stretchy="false">)</mo><mo>=</mo><mn>2</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>2</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>m</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi><mo stretchy="false">)</mo><mo>=</mo><mn>5</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>1</mn></msub><mo stretchy="false">(</mo><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi><mo stretchy="false">)</mo><mo>=</mo><mn>8</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>2</mn></msub><mo stretchy="false">(</mo><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0.</mn></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{matrix}
h_0(add) = 1 &amp; h_1(add) = 4 &amp; h_2(add) = 7\\
h_0(sum) = 9 &amp; h_1(sum) = 2 &amp; h_2(sum) = 1\\
h_0(equal) = 5 &amp; h_1(equal) = 8 &amp; h_2(equal) = 0.
\end{matrix}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:3.6em;vertical-align:-1.55em"></span><span class="mord"><span class="mtable"><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:2.05em"><span style="top:-4.21em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">1</span></span></span><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">9</span></span></span><span style="top:-1.81em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">5</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.55em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:2.05em"><span style="top:-4.21em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">4</span></span></span><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">2</span></span></span><span style="top:-1.81em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">8</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.55em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:2.05em"><span style="top:-4.21em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">7</span></span></span><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">1</span></span></span><span style="top:-1.81em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">0.</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.55em"><span></span></span></span></span></span></span></span></span></span></span></p>
<p>After appending these words, the Bloom filter is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo>=</mo><mrow><mo fence="true">(</mo><mtable rowspacing="0.16em" columnalign="center center center center center center center center center center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">0</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mn mathvariant="bold">1</mn></mstyle></mtd></mtr></mtable><mo fence="true">)</mo></mrow><mi mathvariant="bold">.</mi></mrow><annotation encoding="application/x-tex">\bf{v} = \begin{pmatrix}1&amp;1&amp;1&amp;0&amp;1&amp;1&amp;0&amp;1&amp;1&amp;1\end{pmatrix}.</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.2em;vertical-align:-0.35em"></span><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="minner"><span class="mopen delimcenter" style="top:0em"><span class="delimsizing size1">(</span></span><span class="mord"><span class="mtable"><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.85em"><span style="top:-3.01em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.35em"><span></span></span></span></span></span></span></span><span class="mclose delimcenter" style="top:0em"><span class="delimsizing size1">)</span></span></span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathbf">.</span></span></span></span></span></p>
<p>Now, suppose that we query the words <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>a</mi><mi>c</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">subtract</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">c</span><span class="mord mathnormal">t</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi></mrow><annotation encoding="application/x-tex">multiple</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span></span></span></span> so that</p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtable rowspacing="0.16em" columnalign="center center center" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>a</mi><mi>c</mi><mi>t</mi><mo stretchy="false">)</mo><mo>=</mo><mn>3</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>1</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>a</mi><mi>c</mi><mi>t</mi><mo stretchy="false">)</mo><mo>=</mo><mn>5</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>2</mn></msub><mo stretchy="false">(</mo><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>a</mi><mi>c</mi><mi>t</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>0</mn></msub><mo stretchy="false">(</mo><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi><mo stretchy="false">)</mo><mo>=</mo><mn>7</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>1</mn></msub><mo stretchy="false">(</mo><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>h</mi><mn>2</mn></msub><mo stretchy="false">(</mo><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi><mo stretchy="false">)</mo><mo>=</mo><mn>4</mn></mrow></mstyle></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{matrix} h_0(subtract) = 3 &amp; h_1(subtract) = 5 &amp; h_2(subtract) = 1\\ h_0(multiple) = 7 &amp; h_1(multiple) = 1 &amp; h_2(multiple) = 4\\
\end{matrix}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:2.4em;vertical-align:-0.95em"></span><span class="mord"><span class="mtable"><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.45em"><span style="top:-3.61em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">c</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">3</span></span></span><span style="top:-2.41em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">0</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">7</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.95em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.45em"><span style="top:-3.61em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">c</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">5</span></span></span><span style="top:-2.41em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.95em"><span></span></span></span></span></span><span class="arraycolsep" style="width:0.5em"></span><span class="arraycolsep" style="width:0.5em"></span><span class="col-align-c"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.45em"><span style="top:-3.61em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">c</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">1</span></span></span><span style="top:-2.41em"><span class="pstrut" style="height:3em"></span><span class="mord"><span class="mord"><span class="mord mathnormal">h</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mord">4</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.95em"><span></span></span></span></span></span></span></span></span></span></span>.</p>
<p>The query for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>a</mi><mi>c</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">subtract</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">c</span><span class="mord mathnormal">t</span></span></span></span> returns 0 since <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mn>3</mn><mo stretchy="false">]</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[3]=0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord">3</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">0</span></span></span></span>.
On the other hand, the query for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi></mrow><annotation encoding="application/x-tex">multiple</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span></span></span></span> returns 1 since <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mn>1</mn><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mn>4</mn><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[1]=1, {\bf{v}}[4] = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord">1</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord">4</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mn>7</mn><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[7]=1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord">7</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span>.
Even though <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi><mi>u</mi><mi>l</mi><mi>t</mi><mi>i</mi><mi>p</mi><mi>l</mi><mi>e</mi></mrow><annotation encoding="application/x-tex">multiple</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">m</span><span class="mord mathnormal">u</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">pl</span><span class="mord mathnormal">e</span></span></span></span> was not used to generate the Bloom filter <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi></mrow><annotation encoding="application/x-tex">{\bf{v}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4444em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span></span></span></span>, our query returns the false positive.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="probability-of-false-positives">Probability of false positives<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#probability-of-false-positives" class="hash-link" aria-label="Direct link to Probability of false positives" title="Direct link to Probability of false positives">​</a></h3>
<p>For our analysis, we will assume that the probabilities that arise in our analysis are independent.
However, this assumption can be removed to gain the same approximation.</p>
<p>We note that for a single hash function, the probability that a specific bit is flipped to 1 is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>n</mi></mrow><annotation encoding="application/x-tex">1/n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1/</span><span class="mord mathnormal">n</span></span></span></span>.
So, the probability that the specific bit is not flipped by the hash function is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>−</mo><mn>1</mn><mi mathvariant="normal">/</mi><mi>n</mi></mrow><annotation encoding="application/x-tex">1-1/n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1/</span><span class="mord mathnormal">n</span></span></span></span>.
Applying our assumption that the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> hash functions are 'independent,'
the probability that the specific bit is not flipped by any of the hash functions is
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mn>1</mn><mi mathvariant="normal">/</mi><mi>n</mi><msup><mo stretchy="false">)</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">(1-1/n)^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord">1/</span><span class="mord mathnormal">n</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span>.</p>
<p>Recall the calculus fact <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>lim</mi><mo>⁡</mo></mrow><mi mathvariant="normal">∞</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mn>1</mn><mi mathvariant="normal">/</mi><mi>n</mi><msup><mo stretchy="false">)</mo><mi>n</mi></msup><mo>=</mo><msup><mi>e</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow><annotation encoding="application/x-tex">\lim_{\infty} (1-1/n)^n = e^{-1}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop"><span class="mop">lim</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">∞</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1/</span><span class="mord mathnormal">n</span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">1</span></span></span></span></span></span></span></span></span></span></span></span>.
That is, as we increase the number of bits that our Bloom filter uses, the approximate probability that a given bit is not flipped by any of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> hash functions is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mrow><mo>−</mo><mi>k</mi><mi mathvariant="normal">/</mi><mi>n</mi></mrow></msup></mrow><annotation encoding="application/x-tex">e^{-k/n}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.888em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mtight">/</span><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span></span>.</p>
<p>Suppose that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> entries have been added to the Bloom filter.
The probability that a specific bit is still 0 after the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span> entries have been added is approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mrow><mo>−</mo><mi mathvariant="normal">ℓ</mi><mi>k</mi><mi mathvariant="normal">/</mi><mi>n</mi></mrow></msup></mrow><annotation encoding="application/x-tex">e^{-\ell k/n}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.888em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">ℓ</span><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mtight">/</span><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span></span></span></span>.
The probability that a queried element is erroneously claimed as a member of the digest is approximately
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msup><mi>e</mi><mrow><mo>−</mo><mi mathvariant="normal">ℓ</mi><mi>k</mi><mi mathvariant="normal">/</mi><mi>n</mi></mrow></msup><msup><mo stretchy="false">)</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">(1-e^{-\ell k/n})^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">ℓ</span><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mtight">/</span><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span>.</p>
<p>The following table provides concrete values for these approximations.</p>
<table><thead><tr><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span></th><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span></th><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span></th><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msup><mi>e</mi><mrow><mo>−</mo><mi mathvariant="normal">ℓ</mi><mi>k</mi><mi mathvariant="normal">/</mi><mi>n</mi></mrow></msup><msup><mo stretchy="false">)</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">(1-e^{-\ell k/n})^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.138em;vertical-align:-0.25em"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">−</span><span class="mord mtight">ℓ</span><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span><span class="mord mtight">/</span><span class="mord mathnormal mtight">n</span></span></span></span></span></span></span></span></span><span class="mclose"><span class="mclose">)</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span></th></tr></thead><tbody><tr><td>32</td><td>3</td><td>3</td><td>0.01474</td></tr><tr><td>32</td><td>3</td><td>7</td><td>0.11143</td></tr><tr><td>32</td><td>3</td><td>12</td><td>0.30802</td></tr><tr><td>32</td><td>3</td><td>17</td><td>0.50595</td></tr><tr><td>32</td><td>3</td><td>28</td><td>0.79804</td></tr></tbody></table>
<p>Notice that the probability of false positives increases as the number of elements (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">ℓ</mi></mrow><annotation encoding="application/x-tex">\ell</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">ℓ</span></span></span></span>) that have been added to the digest increases.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="sliding-window-bloom-filter">Sliding-Window Bloom filter<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#sliding-window-bloom-filter" class="hash-link" aria-label="Direct link to Sliding-Window Bloom filter" title="Direct link to Sliding-Window Bloom filter">​</a></h3>
<p>Our toy example and table illustrated an issue concerning Bloom filters.
The number of entries that can be added to a Bloom filter is restricted by our choice of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span>.
Not only does the probability that false positives will occur increase,
but it is possible that our vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi></mrow><annotation encoding="application/x-tex">{\bf{v}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4444em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span></span></span></span> can be a string of all 1s.
<a href="https://eprint.iacr.org/2023/1208.pdf" target="_blank" rel="noopener noreferrer">Szepieniec and Værge</a> proposed a modification to Bloom filters to handle this.</p>
<p>Instead of having a fixed number of bits for our Bloom filter, we dynamically allot memory based on the number of entries that have been added to the filter.
Given a predetermined threshold (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">b</span></span></span></span>) for the number of entries, we shift our 'window' of flipping bits by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span></span></span></span> bits.
Note that this means that it is necessary to keep track of when a given entry is added to the digest.
This means that querying the Sliding-Window Bloom filter will yield different results when different timestamps are used.</p>
<p>This can be done with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> hash functions as we used earlier.
Alternatively, Szepieniec and Værge proposed using the same hash function but to produce <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span></span></span></span> entries in the current window.
Specifically, we obtain the bits we wish to flip to 1s by computing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mo stretchy="false">(</mo><mi>X</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>i</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h(X || i)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mord">∣∣</span><span class="mord mathnormal">i</span><span class="mclose">)</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots, k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> as we will define next.
For Sliding-Window Bloom filters, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> is more than just the element we wish to append to the filter.
Instead, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07847em">X</span></span></span></span> consists of the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> and a timestamp <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span>.
The timestamp <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span> is used to locate the correct window for bits, as we see below:</p>
<ul>
<li>
<p><strong>Append:</strong> Suppose that we wish to add the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> with timestamp <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span> to the Sliding-Window Bloom filter.</p>
<ul>
<li>Define the vector <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">b</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>n</mi><mo>−</mo><mn>1</mn><msup><mo stretchy="false">}</mo><mi>k</mi></msup></mrow><annotation encoding="application/x-tex">{\bf{b}} \in \{0,\dots,n-1\}^k</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7335em;vertical-align:-0.0391em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">n</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0991em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose"><span class="mclose">}</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8491em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em">k</span></span></span></span></span></span></span></span></span></span></span> so that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">b</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo><mo>:</mo><mo>=</mo><mi>h</mi><mo stretchy="false">(</mo><mi>x</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>t</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>i</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">{\bf{b}}[i] := h(x||t||i)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mopen">[</span><span class="mord mathnormal">i</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mord">∣∣</span><span class="mord mathnormal">t</span><span class="mord">∣∣</span><span class="mord mathnormal">i</span><span class="mclose">)</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>.</li>
<li>Update the binary string <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mi mathvariant="bold">b</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo><mo>+</mo><mo stretchy="false">⌊</mo><mi>t</mi><mi mathvariant="normal">/</mi><mi>b</mi><mo stretchy="false">⌋</mo><mi>s</mi><mo stretchy="false">]</mo><mo>←</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[{\bf{b}}[i]+\lfloor t/b \rfloor s] \leftarrow 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf">b</span></span></span></span><span class="mopen">[</span><span class="mord mathnormal">i</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">⌊</span><span class="mord mathnormal">t</span><span class="mord">/</span><span class="mord mathnormal">b</span><span class="mclose">⌋</span><span class="mord mathnormal">s</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> for each <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>.</li>
</ul>
</li>
<li>
<p><strong>Query:</strong> Suppose that we wish to query the Bloom filter for element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> with timestamp <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em"></span><span class="mord mathnormal">t</span></span></span></span>.</p>
<ul>
<li>Return 1 provided <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="bold">v</mi><mo stretchy="false">[</mo><mi>h</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>t</mi><mi mathvariant="normal">∣</mi><mi mathvariant="normal">∣</mi><mi>i</mi><mo stretchy="false">)</mo><mo>+</mo><mo stretchy="false">⌊</mo><mi>t</mi><mi mathvariant="normal">/</mi><mi>b</mi><mo stretchy="false">⌋</mo><mi>s</mi><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">{\bf{v}}[h(y||t||i) + \lfloor t/b \rfloor s] = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord"><span class="mord"><span class="mord"><span class="mord mathbf" style="margin-right:0.01597em">v</span></span></span></span><span class="mopen">[</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord">∣∣</span><span class="mord mathnormal">t</span><span class="mord">∣∣</span><span class="mord mathnormal">i</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">⌊</span><span class="mord mathnormal">t</span><span class="mord">/</span><span class="mord mathnormal">b</span><span class="mclose">⌋</span><span class="mord mathnormal">s</span><span class="mclose">]</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span> for every <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mi>k</mi><mo>−</mo><mn>1</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0,\dots,k-1\}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6986em;vertical-align:-0.0391em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">∈</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">{</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="minner">…</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.03148em">k</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">1</span><span class="mclose">}</span></span></span></span>. Otherwise, return 0.</li>
</ul>
</li>
</ul>
<p>By incorporating a shifting window, we maintain efficient querying and appending at the cost of constant space.
However, by losing constant space, we gain 'infinite' scalability.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="cuckoo-filters">Cuckoo filters<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#cuckoo-filters" class="hash-link" aria-label="Direct link to Cuckoo filters" title="Direct link to Cuckoo filters">​</a></h2>
<p>A Cuckoo filter is a data structure for probabilistic membership proofs based on Cuckoo hash tables.
The specific design goal for Cuckoo filters is to address the inability to remove elements from a Bloom Filter.
This is done by replacing a list of bits with a list of 'fingerprints.'
A fingerprint can be thought of as the hash value for an entry in the digest.
A Cuckoo filter is a fixed-length list of 'fingerprints.'
If the maximum number of entries that a Cuckoo filter can hold is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> and a fingerprint occupies <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span></span></span></span> bits,
then the Cuckoo filter occupies <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mi>f</mi></mrow><annotation encoding="application/x-tex">nf</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span></span></span></span> bits.</p>
<p>Now, we describe the algorithms associated with the Cuckoo filter <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span> with hash function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">hash(X)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mclose">)</span></span></span></span> and fingerprint function <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(X)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.07847em">X</span><span class="mclose">)</span></span></span></span>.</p>
<ul>
<li>
<p><strong>Append:</strong> Suppose that we wish to add the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span> to the Cuckoo filter.</p>
<ul>
<li>If either position <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>x</mi></msub><mo>:</mo><mo>=</mo><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">i_x := hash(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8095em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>x</mi></msub><mo>:</mo><mo>=</mo><mi>i</mi><mo>⊗</mo><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">j_x := i \otimes hash(fingerprint(x))</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7429em;vertical-align:-0.0833em"></span><span class="mord mathnormal">i</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">))</span></span></span></span> of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span> is empty,
then <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span> is inserted into an empty position.</li>
<li>If both <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">i_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8095em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">j_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> are occupied with a fingerprint that is distinct from <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span>,
then we select either <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">i_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8095em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">j_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to insert <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(x)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span></span></span></span>.
The fingerprint that had previously occupied this position cannot be discarried.
Instead, we insert this fingerprint into its alternate location.
This reshuffling process either ends with fingerprints all having their own bucket or one that cannot be inserted.
In the case that we have a fingerprint that cannot be inserted, then the Cuckoo filter is overfilled.</li>
</ul>
</li>
<li>
<p><strong>Query:</strong> Suppose that we wish to query the Cuckoo filter for element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span>.</p>
<ul>
<li>Return 1 provided <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is either in position <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">i_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9456em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">j_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9456em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.</li>
</ul>
</li>
<li>
<p><strong>Delete:</strong> Suppose that we wish to delete the element <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> from the Cuckoo filter.</p>
<ul>
<li>If <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi></mrow><annotation encoding="application/x-tex">y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.03588em">y</span></span></span></span> has been added to the Cuckoo filter, then <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> is either in position <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">i_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9456em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> or <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>y</mi></msub></mrow><annotation encoding="application/x-tex">j_y</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9456em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.
We remove <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">fingerprint(y)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mclose">)</span></span></span></span> from the appropriate position.</li>
</ul>
</li>
</ul>
<p>We note that false positives in Cuckoo filters only occur when an element shares a fingerprint and hash with a value that has already been added to the Cuckoo filter.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="example-1">Example<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#example-1" class="hash-link" aria-label="Direct link to Example" title="Direct link to Example">​</a></h3>
<p>In this example, we will append the words <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span>, and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> to a Cuckoo filter with 8 slots.</p>
<p>For each word <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span>, we compute two indices:
<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>x</mi></msub><mo>:</mo><mo>=</mo><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mtext>&nbsp;and&nbsp;</mtext><msub><mi>j</mi><mi>x</mi></msub><mo>:</mo><mo>=</mo><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>⊗</mo><mi>h</mi><mi>a</mi><mi>s</mi><mi>h</mi><mo stretchy="false">(</mo><mi>f</mi><mi>i</mi><mi>n</mi><mi>g</mi><mi>e</mi><mi>r</mi><mi>p</mi><mi>r</mi><mi>i</mi><mi>n</mi><mi>t</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mi mathvariant="normal">.</mi></mrow><annotation encoding="application/x-tex">i_x := hash(x) \text{ and } j_x := hash(x) \otimes hash(fingerprint(x)).</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8095em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span><span class="mord text"><span class="mord">&nbsp;and&nbsp;</span></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">:=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">ha</span><span class="mord mathnormal">s</span><span class="mord mathnormal">h</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">in</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">p</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">in</span><span class="mord mathnormal">t</span><span class="mopen">(</span><span class="mord mathnormal">x</span><span class="mclose">))</span><span class="mord">.</span></span></span></span>
Suppose that we have the following values for
our words:</p>
<table><thead><tr><th>word</th><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>i</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">i_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8095em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">i</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></th><th><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>j</mi><mi>x</mi></msub></mrow><annotation encoding="application/x-tex">j_x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.854em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05724em">j</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0572em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">x</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span></th></tr></thead><tbody><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(0,1,0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mclose">)</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(1,0,0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mclose">)</span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(1,0,1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose">)</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo separator="true">,</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(1,1,0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mclose">)</span></span></span></span></td></tr><tr><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(0,1,0)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mclose">)</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo separator="true">,</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(1,0,1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">(</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">0</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">1</span><span class="mclose">)</span></span></span></span></td></tr></tbody></table>
<p>For clarity of the example, we append the words directly to the buckets instead of fingerprints of our data.</p>
<table><thead><tr><th></th><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><td>append <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span></td><td></td><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>append <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span></td><td></td><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span></td><td></td><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span></td><td></td><td></td></tr></tbody></table>
<p>Notice that both of the buckets (2 and 5) that <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> can map to are occupied.
So, we select one of these buckets (say 2) to insert <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span> into.
Then, we have to insert <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span> to its possible bucket (1).
This leaves us with the Cuckoo filter:</p>
<table><thead><tr><th>0</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mi>d</mi><mi>d</mi></mrow><annotation encoding="application/x-tex">add</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord mathnormal">a</span><span class="mord mathnormal">dd</span></span></span></span></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><mi>q</mi><mi>u</mi><mi>a</mi><mi>l</mi></mrow><annotation encoding="application/x-tex">equal</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span></td><td></td><td></td><td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>u</mi><mi>m</mi></mrow><annotation encoding="application/x-tex">sum</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">m</span></span></span></span></td><td></td><td></td></tr></tbody></table>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="complexity-1">Complexity<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#complexity-1" class="hash-link" aria-label="Direct link to Complexity" title="Direct link to Complexity">​</a></h3>
<p>Notice that deletions and queries to Cuckoo filters are done in constant time.
Specifically, only two locations need to be checked for any data <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span>.
Appends may require shuffling previously added elements to their alternate locations.
As such, the append does not run in constant time.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="bloom-filters-vs-cuckoo-filters">Bloom filters vs Cuckoo filters<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#bloom-filters-vs-cuckoo-filters" class="hash-link" aria-label="Direct link to Bloom filters vs Cuckoo filters" title="Direct link to Bloom filters vs Cuckoo filters">​</a></h2>
<p>The design of Bloom filters is focused on space efficiency and quick query time.
Even though they occupy constant space,
Cuckoo filters require significantly more space for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">n</span></span></span></span> items than Bloom filters.
The worst-case append in a Cuckoo filter is slower than the append in a Bloom filter.
However, an append that does not require any shuffling in a Cuckoo filter can be quicker than appends in Bloom filters.
Cuckoo filters make up for these disadvantages with quicker query time and the ability to delete entries.
Further, the probability of false positives in Cuckoo filters is lower than the probability of false positives in Bloom filters.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="combining-filters-with-rln">Combining Filters with RLN<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#combining-filters-with-rln" class="hash-link" aria-label="Direct link to Combining Filters with RLN" title="Direct link to Combining Filters with RLN">​</a></h2>
<p>In a series of posts (<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">1</a>,<a href="https://vac.dev/rlog/rln-v3/" target="_blank" rel="noopener noreferrer">2</a>,<a href="https://vac.dev/rlog/rln-light-verifiers" target="_blank" rel="noopener noreferrer">3</a>),
various versons of rate limiting nullifiers (RLN) that are used by Waku has been discussed.
RLN uses a sparse Merkle tree for the membership set.
The computational power required to construct the Merkle tree prevent light clients from participating in verifying membership proofs.
In <a href="https://vac.dev/rlog/rln-light-verifiers" target="_blank" rel="noopener noreferrer">Verifying RLN Proofs in Light Clients with Subtrees</a>,
it was proposed to move the membership set on-chain so that it would not be necessary for a light client to construct the entire Merkle tree locally.
Unfortunately, the naive approach is not practical as the gas limit for a single call is too restrictive for an appropriately sized tree.
Instead, it was proposed to make utilize of subtrees.
In this section, we provide a discussion of an alternate solution for light clients by using filters for the membership set.
The two <a href="https://rate-limiting-nullifier.github.io/rln-docs/rln_in_details.html" target="_blank" rel="noopener noreferrer">parts of RLN</a> that we will focus on are user registration and deletion.</p>
<p>Both Bloom and Cuckoo filters support user registration as this is can be done as an append.
The fixed size of these filters would restrict the total number of users that can register.
This can be migitated by using Sliding-Window Bloom filter as this supports system growth.
The Sliding-Window can be adapted to Cuckoo filters as well.
In the case of a Sliding-Window filter, an user would maintain the epoch of when they registered.
The registration of new users to Bloom filters can be done in constant time which is a significant improvement over appending to subtrees.
Unfortunately, the complexity of registration to Cuckoo filters cannot be as easily computed.</p>
<p>A user could be slashed from the RLN by sending too many messages in a given epoch.
Unfortunately, Bloom filters do not support the deletion of members.
Luckily, Cuckoo filters allow for deletions that can performed in constant time.</p>
<p>Cuckoo filter that use Sliding-Window could be used so that light clients are able to verify proofs of membership in the RLN.
These proofs are not a substitute to the usual proofs that a heavy client can verify due to the allowance of false positives.
However, with the allowance of false positives, a light client can participate in verification RLN proofs in an efficient manner.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/vac101-membership-with-bloom-filters-and-cuckoo-filters#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h3>
<ul>
<li><a href="https://dl.acm.org/doi/10.1145/362686.362692" target="_blank" rel="noopener noreferrer">Space/Time Trade-offs in Hash Coding with Allowable Errors</a></li>
<li><a href="https://people.eecs.berkeley.edu/~daw/teaching/cs170-s03/Notes/lecture10.pdf" target="_blank" rel="noopener noreferrer">David Wagner's Lecture Notes on Bloom filters</a></li>
<li><a href="https://eprint.iacr.org/2023/1208" target="_blank" rel="noopener noreferrer">Mutator Sets and their Application to Scalable Privacy</a></li>
<li><a href="https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf" target="_blank" rel="noopener noreferrer">Cuckoo Filter: Practically Better than Bloom</a></li>
<li><a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a></li>
<li><a href="https://vac.dev/rlog/rln-v3/" target="_blank" rel="noopener noreferrer">RLN-v3: Towards a Flexible and Cost-Efficient Implementation</a></li>
<li><a href="https://vac.dev/rlog/rln-light-verifiers" target="_blank" rel="noopener noreferrer">Verifying RLN Proofs in Light Clients with Subtrees</a></li>
<li><a href="https://rate-limiting-nullifier.github.io/rln-docs/rln_in_details.html" target="_blank" rel="noopener noreferrer">RLN in details</a></li>
</ul>]]></content>
<author>
<name>Marvin</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[RLN-v3: Towards a Flexible and Cost-Efficient Implementation]]></title>
<id>https://vac.dev/rlog/rln-v3</id>
<link href="https://vac.dev/rlog/rln-v3"/>
<updated>2024-05-13T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Improving on the previous version of RLN by allowing dynamic epoch sizes.]]></summary>
<content type="html"><![CDATA[<p>Improving on the previous version of RLN by allowing dynamic epoch sizes.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/rln-v3#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Recommended previous reading: <a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a>.</p>
<p>The premise of RLN-v3 is to have a variable message rate per variable epoch,
which can be explained in the following way:</p>
<ul>
<li>
<p><strong>RLN-v1:</strong> “Alice can send 1 message per global epoch”</p>
<p>Practically, this is <code>1 msg/second</code></p>
</li>
<li>
<p><strong>RLN-v2:</strong> “Alice can send <code>x</code> messages per global epoch”</p>
<p>Practically, this is <code>x msg/second</code></p>
</li>
<li>
<p><strong>RLN-v3:</strong> “Alice can send <code>x</code> messages within a time interval <code>y</code> chosen by herself.
The funds she has to pay are affected by both the number of messages and the chosen time interval.
Other participants can choose different time intervals fitting their specific needs.</p>
<p>Practically, this is <code>x msg/y seconds</code></p>
</li>
</ul>
<p>RLN-v3 allows higher flexibility and ease of payment/stake for users who have more predictable usage patterns and therefore,
more predictable bandwidth usage on a p2p network (Waku, etc.).</p>
<p>For example:</p>
<ul>
<li>An AMM that broadcasts bids, asks, and fills over Waku may require a lot of throughput in the smallest epoch possible and hence may register an RLN-v3 membership of <code>10000 msg/1 second</code>.
They could do this with RLN-v2, too.</li>
<li>Alice, a casual user of a messaging app built on Waku, who messages maybe 3-4 people infrequently during the day, may register an RLN-v3 membership of <code>100 msg/hour</code>,
which would not be possible in RLN-v2 considering the <code>global epoch</code> was set to <code>1 second</code>.
With RLN-v2, Alice would have to register with a membership of <code>1 msg/sec</code>,
which would translate to <code>3600 msg/hour</code>. This is much higher than her usage and would
result in her overpaying to stake into the membership set.</li>
<li>A sync service built over Waku,
whose spec defines that it MUST broadcast a set of public keys every hour,
may register an RLN-v3 membership of <code>1 msg/hour</code>,
cutting down the costs to enter the membership set earlier.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="theory">Theory<a href="https://vac.dev/rlog/rln-v3#theory" class="hash-link" aria-label="Direct link to Theory" title="Direct link to Theory">​</a></h2>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modification-to-leaves-set-in-the-membership-merkle-tree">Modification to leaves set in the membership Merkle tree<a href="https://vac.dev/rlog/rln-v3#modification-to-leaves-set-in-the-membership-merkle-tree" class="hash-link" aria-label="Direct link to Modification to leaves set in the membership Merkle tree" title="Direct link to Modification to leaves set in the membership Merkle tree">​</a></h3>
<p>To ensure that a user’s epoch size (<code>user_epoch_limit</code>) is included within their membership we must modify the user’s commitment/leaf in the tree to contain it.
A user’s commitment/leaf in the tree is referred to as a <code>rate_commitment</code>,
which was previously derived from their public key (<code>identity_commitment</code>)
and their variable message rate (<code>user_message_limit</code>).</p>
<p>In <strong>RLN-v2:</strong></p>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>r</mi><mi>a</mi><mi>t</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">rate\_commitment = poseidon([identity\_commitment, user\_message\_limit])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">])</span></span></span></span></span>
<p>In <strong>RLN-v3:</strong></p>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>r</mi><mi>a</mi><mi>t</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo separator="true">,</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">rate\_commitment = poseidon([identity\_commitment, user\_message\_limit, user\_epoch\_limit])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">])</span></span></span></span></span>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modification-to-circuit-inputs">Modification to circuit inputs<a href="https://vac.dev/rlog/rln-v3#modification-to-circuit-inputs" class="hash-link" aria-label="Direct link to Modification to circuit inputs" title="Direct link to Modification to circuit inputs">​</a></h3>
<p>To detect double signaling,
we make use of a circuit output <code>nullifier</code>,
which remains the same if a user generates a proof with the same <code>message_id</code> and <code>external_nullifier</code>,
where the <code>external_nullifier</code> and <code>nullifier</code> are defined as:</p>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>e</mi><mi>x</mi><mi>t</mi><mi>e</mi><mi>r</mi><mi>n</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo separator="true">,</mo><mi>r</mi><mi>l</mi><mi>n</mi><mi mathvariant="normal">_</mi><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo><mspace linebreak="newline"></mspace><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo>=</mo><mi>p</mi><mi>o</mi><mi>s</mi><mi>e</mi><mi>i</mi><mi>d</mi><mi>o</mi><mi>n</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>e</mi><mi>c</mi><mi>r</mi><mi>e</mi><mi>t</mi><mo separator="true">,</mo><mi>e</mi><mi>x</mi><mi>t</mi><mi>e</mi><mi>r</mi><mi>n</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>n</mi><mi>u</mi><mi>l</mi><mi>l</mi><mi>i</mi><mi>f</mi><mi>i</mi><mi>e</mi><mi>r</mi><mo separator="true">,</mo><mi>m</mi><mi>e</mi><mi>s</mi><mi>s</mi><mi>a</mi><mi>g</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>i</mi><mi>d</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">external\_nullifier = poseidon([epoch, rln\_identifier]) \\
nullifier = poseidon([identity\_secret, external\_nullifier, message\_id])</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">x</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">na</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">n</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mclose">])</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">p</span><span class="mord mathnormal">ose</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span><span class="mopen">([</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">secre</span><span class="mord mathnormal">t</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">x</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord mathnormal">na</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">m</span><span class="mord mathnormal">ess</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">e</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mclose">])</span></span></span></span></span>
<p>Where:</p>
<ul>
<li><code>epoch</code> is defined as the Unix epoch timestamp with seconds precision.</li>
<li><code>rln_identifier</code> uniquely identifies an application for which a user submits a proof.</li>
<li><code>identity_secret</code> is the private key of the user.</li>
<li><code>message_id</code> is the sequence number of the user’s message within <code>user_message_limit</code> in an epoch.</li>
</ul>
<p>In RLN-v2, the global epoch was 1 second,
hence we did not need to perform any assertions to the epoch’s value inside the circuit,
and the validation of the epoch was handled off-circuit (i.e., too old, too large, bad values, etc.).</p>
<p>In RLN-v3, we propose that the <code>epoch</code> that is passed into the circuit
must be a valid multiple of <code>user_epoch_limit</code>
since the user may pass in values of the <code>epoch</code> which do not directly correlate with the <code>user_epoch_limit</code>.</p>
<p>For example:</p>
<ul>
<li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>237</code>
generates <code>user_message_limit</code> proofs with it,
can increment the epoch by <code>1</code>,
and generate <code>user_message_limit</code> proofs with it,
thereby allowing them to bypass the message per epoch restriction.</li>
</ul>
<p>One could say that we could perform this validation outside of the circuit,
but we maintain the <code>user_epoch_limit</code> as a private input to the circuit so that the user is not deanonymized by the anonymity set connected to that <code>user_epoch_limit</code>.
Since <code>user_epoch_limit</code> is kept private,
the verifier does not have access to that value and cannot perform validation on it.</p>
<p>If we ensure that the <code>epoch</code> is a multiple of <code>user_epoch_limit</code>,
we have the following scenarios:</p>
<ul>
<li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>237</code>.
Proof generation fails since the epoch is not a multiple of <code>user_epoch_limit</code>.</li>
<li>A user with <code>user_epoch_limit</code> of 120
passes in an epoch of <code>240</code> and
can generate <code>user_message_limit</code> proofs without being slashed.</li>
</ul>
<p>Since we perform operations on the <code>epoch</code>, we must include it as a circuit input (previously, it was removed from the circuit inputs to RLN-v2).</p>
<p>Therefore, the new circuit inputs are as follows:</p>
<div class="language-c codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-c codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token comment" style="color:rgb(98, 114, 164)">// unchanged</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private identity_secret</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_message_limit</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private message_id</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private pathElements</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private pathIndices</span><span class="token punctuation" style="color:rgb(248, 248, 242)">[</span><span class="token punctuation" style="color:rgb(248, 248, 242)">]</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public x </span><span class="token comment" style="color:rgb(98, 114, 164)">// messageHash</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain" style="display:inline-block"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain"></span><span class="token comment" style="color:rgb(98, 114, 164)">// new/changed</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_epoch_limit</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">private user_epoch_quotient </span><span class="token comment" style="color:rgb(98, 114, 164)">// epoch/user_epoch_limit to assert within circuit</span><span class="token plain"></span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public epoch</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">public rln_identifier</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>The circuit outputs remain the same.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="additional-circuit-constraints">Additional circuit constraints<a href="https://vac.dev/rlog/rln-v3#additional-circuit-constraints" class="hash-link" aria-label="Direct link to Additional circuit constraints" title="Direct link to Additional circuit constraints">​</a></h3>
<ol>
<li>
<p>Since we accept the <code>epoch</code>, <code>user_epoch_quotient</code>, and <code>user_epoch_limit</code>,
we must ensure that the relation between these 3 values is preserved. I.e.:</p>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo>=</mo><mo>=</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>∗</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>q</mi><mi>u</mi><mi>o</mi><mi>t</mi><mi>i</mi><mi>e</mi><mi>n</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">epoch == user\_epoch\_limit * user\_epoch\_quotient</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">==</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span></span></span></span></span>
</li>
<li>
<p>To ensure no overflows/underflows occur in the above multiplication,
we must constrain the inputs of <code>epoch</code>, <code>user_epoch_quotient</code>, and <code>user_epoch_limit</code>.
We have assumed <code>3600</code> to be the maximum valid size of the <code>user_epoch_quotient</code>.</p>
</li>
</ol>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo stretchy="false">(</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mo stretchy="false">)</mo><mo>≤</mo><mn>64</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>i</mi><mi>t</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo stretchy="false">(</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo stretchy="false">)</mo><mo>≤</mo><mn>12</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>i</mi><mi>t</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>≤</mo><mn>3600</mn><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi><mo>≤</mo><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mspace linebreak="newline"></mspace><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>q</mi><mi>u</mi><mi>o</mi><mi>t</mi><mi>i</mi><mi>e</mi><mi>n</mi><mi>t</mi><mo>&lt;</mo><mi>u</mi><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>e</mi><mi>p</mi><mi>o</mi><mi>c</mi><mi>h</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>i</mi><mi>m</mi><mi>i</mi><mi>t</mi></mrow><annotation encoding="application/x-tex">size(epoch) \leq 64\ bits \\
size(user\_epoch\_limit) \leq 12\ bits \\
user\_epoch\_limit \leq 3600 \\
user\_epoch\_limit \leq epoch \\
user\_epoch\_quotient &lt; user\_epoch\_limit</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mopen">(</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">64</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">bi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.06em;vertical-align:-0.31em"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mopen">(</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6944em"></span><span class="mord">12</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">bi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">s</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">3600</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.03588em">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">&lt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.02778em">ser</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">e</span><span class="mord mathnormal">p</span><span class="mord mathnormal">oc</span><span class="mord mathnormal">h</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">imi</span><span class="mord mathnormal">t</span></span></span></span></span>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modifications-to-external-epoch-validation-waku-etc">Modifications to external epoch validation (Waku, etc.)<a href="https://vac.dev/rlog/rln-v3#modifications-to-external-epoch-validation-waku-etc" class="hash-link" aria-label="Direct link to Modifications to external epoch validation (Waku, etc.)" title="Direct link to Modifications to external epoch validation (Waku, etc.)">​</a></h3>
<p>For receivers of an RLN-v3 proof
to detect if a message is too old, we must use the higher bound of the <code>user_epoch_limit</code>, which has been set to <code>3600</code>.
The <strong>trade-off</strong> here is that we allow hour-old messages to propagate within the network.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="modifications-to-double-signaling-detection-scheme-waku-etc">Modifications to double signaling detection scheme (Waku, etc.)<a href="https://vac.dev/rlog/rln-v3#modifications-to-double-signaling-detection-scheme-waku-etc" class="hash-link" aria-label="Direct link to Modifications to double signaling detection scheme (Waku, etc.)" title="Direct link to Modifications to double signaling detection scheme (Waku, etc.)">​</a></h3>
<p>For verifiers of RLN-v1/v2 proofs,
a log of nullifiers seen in the last epoch is maintained,
and if there is a match with a pre-existing nullifier,
double signaling has been detected and the verifier MAY proceed to slash the spamming user.</p>
<p>With the RLN-v3 scheme,
we need to increase the size of the nullifier log used,
which previously cleared itself every second to the higher bound of the <code>user_epoch_limit</code>, which is <code>3600</code>.
Now, the RLN proof verifier must clear the nullifier log every <code>3600</code> seconds to satisfactorily detect double signaling.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-implementation">The implementation<a href="https://vac.dev/rlog/rln-v3#the-implementation" class="hash-link" aria-label="Direct link to The implementation" title="Direct link to The implementation">​</a></h2>
<p>An implementation of the RLN-v3 scheme in <a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer">gnark</a> can be found <a href="https://github.com/vacp2p/gnark-rln/blob/9b05eddc89901a06d8f41b093ce8ce12fd0bb4e0/rln/rln.go" target="_blank" rel="noopener noreferrer">here</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="comments-on-performance">Comments on performance<a href="https://vac.dev/rlog/rln-v3#comments-on-performance" class="hash-link" aria-label="Direct link to Comments on performance" title="Direct link to Comments on performance">​</a></h2>
<ul>
<li>Hardware: Macbook Air M2, 16GB RAM</li>
<li>Circuit: <a href="https://github.com/vacp2p/gnark-rln/blob/9b05eddc89901a06d8f41b093ce8ce12fd0bb4e0/rln/rln.go" target="_blank" rel="noopener noreferrer">RLN-v3</a></li>
<li>Proving system: <a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer"><code>Groth16</code></a></li>
<li>Framework: <a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer"><code>gnark</code></a></li>
<li>Elliptic curve: <a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer"><code>bn254</code></a> (aka bn128) (not to be confused with the 254-bit Weierstrass curve)</li>
<li>Finite field: Prime-order subgroup of the group of points on the <code>bn254</code> curve</li>
<li>Default Merkle tree height: <code>20</code></li>
<li>Hashing algorithm: <a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer"><code>Poseidon</code></a></li>
<li>Merkle tree: <a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer"><code>Sparse Indexed Merkle Tree</code></a></li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proving">Proving<a href="https://vac.dev/rlog/rln-v3#proving" class="hash-link" aria-label="Direct link to Proving" title="Direct link to Proving">​</a></h3>
<p>The proving time for the RLN-v3 circuit is <code>90ms</code> for a single proof.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="verification">Verification<a href="https://vac.dev/rlog/rln-v3#verification" class="hash-link" aria-label="Direct link to Verification" title="Direct link to Verification">​</a></h3>
<p>The verification time for the RLN-v3 circuit is <code>1.7ms</code> for a single proof.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/rln-v3#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>The RLN-v3 scheme introduces a new epoch-based message rate-limiting scheme to the RLN protocol.
It enhances the user's flexibility in setting their message limits and cost-optimizes their stake.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="https://vac.dev/rlog/rln-v3#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2>
<ul>
<li>Implementing the RLN-v3 scheme in <a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li>
<li>Implementing the RLN-v3 scheme in <a href="https://github.com/waku-org/nwaku" target="_blank" rel="noopener noreferrer">Waku</a></li>
<li>Formal security analysis of the RLN-v3 scheme</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/rln-v3#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li><a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a></li>
<li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li>
<li><a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer">Groth16</a></li>
<li><a href="https://docs.gnark.consensys.io/" target="_blank" rel="noopener noreferrer">Gnark</a></li>
<li><a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer">Poseidon Hash</a></li>
<li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li>
<li><a href="https://rfc.vac.dev/vac/32/rln-v1" target="_blank" rel="noopener noreferrer">RLN-v1 RFC</a></li>
<li><a href="https://rfc.vac.dev/vac/raw/rln-v2" target="_blank" rel="noopener noreferrer">RLN-v2 RFC</a></li>
<li><a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a></li>
</ul>]]></content>
<author>
<name>Aaryamann</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Verifying RLN Proofs in Light Clients with Subtrees]]></title>
<id>https://vac.dev/rlog/rln-light-verifiers</id>
<link href="https://vac.dev/rlog/rln-light-verifiers"/>
<updated>2024-05-03T12:00:00.000Z</updated>
<summary type="html"><![CDATA[How resource-restricted devices can verify RLN proofs fast and efficiently.]]></summary>
<content type="html"><![CDATA[<p>How resource-restricted devices can verify RLN proofs fast and efficiently.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/rln-light-verifiers#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Recommended previous reading: <a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a>.</p>
<p>This post expands upon ideas described in the previous post,
focusing on how resource-restricted devices can verify RLN proofs fast and efficiently.</p>
<p>Previously, it was required to fetch all the memberships from the smart contract,
construct the merkle tree locally,
and derive the merkle root,
which is subsequently used to verify RLN proofs.</p>
<p>This process is not feasible for resource-restricted devices since it involves a lot of RPC calls, computation and fault tolerance.
One cannot expect a mobile phone to fetch all the memberships from the smart contract and construct the merkle tree locally.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="constraints-and-requirements">Constraints and requirements<a href="https://vac.dev/rlog/rln-light-verifiers#constraints-and-requirements" class="hash-link" aria-label="Direct link to Constraints and requirements" title="Direct link to Constraints and requirements">​</a></h2>
<p>An alternative solution to the one proposed in this post is to construct the merkle tree on-chain,
and have the root accessible with a single RPC call.
However, this approach increases gas costs for inserting new memberships and <em>may</em> not be feasible until it is optimized further with batching mechanisms, etc.</p>
<p>The other methods have been explored in more depth <a href="https://hackmd.io/@rymnc/rln-tree-storages" target="_blank" rel="noopener noreferrer">here</a>.</p>
<p>Following are the requirements and constraints for the solution proposed in this post:</p>
<ol>
<li>Cheap membership insertions.</li>
<li>As few RPC calls as possible to reduce startup time.</li>
<li>Merkle root of the tree is available on-chain.</li>
<li>No centralized services to sequence membership insertions.</li>
<li>Map inserted commitments to the block in which they were inserted.</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="metrics-on-sync-time-for-a-tree-with-2653-leaves">Metrics on sync time for a tree with 2,653 leaves<a href="https://vac.dev/rlog/rln-light-verifiers#metrics-on-sync-time-for-a-tree-with-2653-leaves" class="hash-link" aria-label="Direct link to Metrics on sync time for a tree with 2,653 leaves" title="Direct link to Metrics on sync time for a tree with 2,653 leaves">​</a></h2>
<p>The following metrics are based on the current implementation of RLN in the Waku gen0 network.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="test-bench">Test bench<a href="https://vac.dev/rlog/rln-light-verifiers#test-bench" class="hash-link" aria-label="Direct link to Test bench" title="Direct link to Test bench">​</a></h3>
<ul>
<li>Hardware: Macbook Air M2, 16GB RAM</li>
<li>Network: 120 Megabits/sec</li>
<li>Nwaku commit: <a href="https://github.com/waku-org/nwaku/tree/e61e4ff90a235657a7dc4248f5be41b6e031e98c" target="_blank" rel="noopener noreferrer">e61e4ff</a></li>
<li>RLN membership set contract: <a href="https://sepolia.etherscan.io/address/0xF471d71E9b1455bBF4b85d475afb9BB0954A29c4#code" target="_blank" rel="noopener noreferrer">0xF471d71E9b1455bBF4b85d475afb9BB0954A29c4</a></li>
<li>Deployed block number: 4,230,716</li>
<li>RLN Membership set depth: 20</li>
<li>Hash function: PoseidonT3 (which is a gas guzzler)</li>
<li>Max size of the membership set: 2^20 = 1,048,576 leaves</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="metrics">Metrics<a href="https://vac.dev/rlog/rln-light-verifiers#metrics" class="hash-link" aria-label="Direct link to Metrics" title="Direct link to Metrics">​</a></h3>
<ul>
<li>Time to sync the whole tree: 4 minutes</li>
<li>RPC calls: 702</li>
<li>Number of leaves: 2,653</li>
</ul>
<p>One can argue that the time to sync the tree at the current state is not <em>that</em> bad.
However, the number of RPC calls is a concern,
which scales linearly with the number of blocks since the contract was deployed
This is because the implementation fetches all events from the contract,
chunking 2,000 blocks at a time.
This is done to avoid hitting the block limit of 10,000 events per call,
which is a limitation of popular RPC providers.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proposed-solution">Proposed solution<a href="https://vac.dev/rlog/rln-light-verifiers#proposed-solution" class="hash-link" aria-label="Direct link to Proposed solution" title="Direct link to Proposed solution">​</a></h2>
<p>From a theoretical perspective,
one could construct the merkle tree on-chain,
in a view call, in-memory.
However, this is not feasible due to the gas costs associated with it.</p>
<p>To compute the root of a Merkle tree with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>20</mn></msup></mrow><annotation encoding="application/x-tex">2^{20}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span></span></span> leaves it costs approximately 2 billion gas.
With Infura and Alchemy capping the gas limit to 350M and 550M gas respectively,
it is not possible to compute the root of the tree in a single call.</p>
<p>Acknowledging that <a href="https://polygon.technology/blog/polygon-miden-state-model" target="_blank" rel="noopener noreferrer">Polygon Miden</a> and <a href="https://penumbra.zone/blog/tiered-commitment-tree/" target="_blank" rel="noopener noreferrer">Penumbra</a> both make use of a tiered commitment tree,
we propose a similar approach for RLN.</p>
<p>A tiered commitment tree is a tree which is sharded into multiple smaller subtrees,
each of which is a tree in itself.
This allows scaling in terms of the number of leaves,
as well as reducing state bloat by just storing the root of a subtree when it is full instead of all its leaves.</p>
<p>Here, the question arises:
What is the maximum number of leaves in a subtree with which the root can be computed in a single call?</p>
<p>It costs approximately 217M gas to compute the root of a Merkle tree with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span> leaves.</p>
<p>This is a feasible number for a single call,
and hence we propose a tiered commitment tree with a maximum of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span> leaves in a subtree and the number of subtrees is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">10</span></span></span></span></span></span></span></span></span></span></span></span>.
Therefore, the maximum number of leaves in the tree is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>20</mn></msup></mrow><annotation encoding="application/x-tex">2^{20}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em"><span style="top:-3.063em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span></span></span> (the same as the current implementation).</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="img" src="https://vac.dev/assets/images/light-rln-verifiers-f801999160884be6a1223ee7d76cebcf.png" width="631" height="381" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="insertion">Insertion<a href="https://vac.dev/rlog/rln-light-verifiers#insertion" class="hash-link" aria-label="Direct link to Insertion" title="Direct link to Insertion">​</a></h3>
<p>When a commitment is inserted into the tree it is first inserted into the first subtree.
When the first subtree is full the next insertions go into the second subtree and so on.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="syncing">Syncing<a href="https://vac.dev/rlog/rln-light-verifiers#syncing" class="hash-link" aria-label="Direct link to Syncing" title="Direct link to Syncing">​</a></h3>
<p>When syncing the tree,
one only needs to fetch the roots of the subtrees.
The root of the full tree can be computed in-memory or on-chain.</p>
<p>This allows us to derive the following relation:</p>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>n</mi><mi>u</mi><mi>m</mi><mi>b</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>o</mi><mi>f</mi><mi mathvariant="normal">_</mi><mi>r</mi><mi>p</mi><mi>c</mi><mi mathvariant="normal">_</mi><mi>c</mi><mi>a</mi><mi>l</mi><mi>l</mi><mi>s</mi><mo>=</mo><mi>n</mi><mi>u</mi><mi>m</mi><mi>b</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">_</mi><mi>o</mi><mi>f</mi><mi mathvariant="normal">_</mi><mi>f</mi><mi>i</mi><mi>l</mi><mi>l</mi><mi>e</mi><mi>d</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>u</mi><mi>b</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi>s</mi><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">number\_of\_rpc\_calls = number\_of\_filled\_subtrees + 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal">mb</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.02778em">r</span><span class="mord mathnormal">p</span><span class="mord mathnormal">c</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">c</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">n</span><span class="mord mathnormal">u</span><span class="mord mathnormal">mb</span><span class="mord mathnormal" style="margin-right:0.02778em">er</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">o</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.10764em">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.01968em">ll</span><span class="mord mathnormal">e</span><span class="mord mathnormal">d</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">u</span><span class="mord mathnormal">b</span><span class="mord mathnormal">t</span><span class="mord mathnormal">rees</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">1</span></span></span></span></span>
<p>This is a significant improvement over the current implementation,
which requires fetching all the memberships from the smart contract.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="gas-costs">Gas costs<a href="https://vac.dev/rlog/rln-light-verifiers#gas-costs" class="hash-link" aria-label="Direct link to Gas costs" title="Direct link to Gas costs">​</a></h3>
<p>The gas costs for inserting a commitment into the tree are the same as the current implementation except it consists of an extra SSTORE operation to store the <code>shardIndex</code> of the commitment.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="events">Events<a href="https://vac.dev/rlog/rln-light-verifiers#events" class="hash-link" aria-label="Direct link to Events" title="Direct link to Events">​</a></h3>
<p>The events emitted by the contract are the same as the current implementation,
appending the <code>shardIndex</code> of the commitment.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="proof-of-concept">Proof of concept<a href="https://vac.dev/rlog/rln-light-verifiers#proof-of-concept" class="hash-link" aria-label="Direct link to Proof of concept" title="Direct link to Proof of concept">​</a></h3>
<p>A proof of concept implementation of the tiered commitment tree is available <a href="https://github.com/vacp2p/rln-contract/pull/37" target="_blank" rel="noopener noreferrer">here</a>,
and is deployed on Sepolia at <a href="https://sepolia.etherscan.io/address/0xE7987c70B54Ff32f0D5CBbAA8c8Fc1cAf632b9A5" target="_blank" rel="noopener noreferrer">0xE7987c70B54Ff32f0D5CBbAA8c8Fc1cAf632b9A5</a>.</p>
<p>It is compatible with the current implementation of the RLN verifier.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="https://vac.dev/rlog/rln-light-verifiers#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2>
<ol>
<li>Optimize the gas costs of the tiered commitment tree.</li>
<li>Explore using different number of leaves under a given node in the tree (currently set to 2).</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/rln-light-verifiers#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h2>
<p>The tiered commitment tree is a promising approach to reduce the number of RPC calls required to sync the tree and reduce the gas costs associated with computing the root of the tree.
Consequently, it allows for a more scalable and efficient RLN verifier.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/rln-light-verifiers#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li>
<li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li>
<li><a href="https://rfc.vac.dev/vac/32/rln-v1" target="_blank" rel="noopener noreferrer">RLN-V1 RFC</a></li>
<li><a href="https://rfc.vac.dev/vac/raw/rln-v2" target="_blank" rel="noopener noreferrer">RLN-V2 RFC</a></li>
<li><a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a></li>
<li><a href="https://vac.dev/rlog/rln-anonymous-dos-prevention" target="_blank" rel="noopener noreferrer">Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku</a></li>
</ul>]]></content>
<author>
<name>Aaryamann</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Strengthening Anonymous DoS Prevention with Rate Limiting Nullifiers in Waku]]></title>
<id>https://vac.dev/rlog/rln-anonymous-dos-prevention</id>
<link href="https://vac.dev/rlog/rln-anonymous-dos-prevention"/>
<updated>2023-11-07T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Rate Limiting Nullifiers in practice, applied to an anonymous p2p network, like Waku.]]></summary>
<content type="html"><![CDATA[<p>Rate Limiting Nullifiers in practice, applied to an anonymous p2p network, like Waku.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Rate Limiting Nullifier (RLN) is a zero-knowledge gadget that allows users to prove 2 pieces of information,</p>
<ol>
<li>They belong to a permissioned membership set</li>
<li>Their rate of signaling abides by a fixed number that has been previously declared</li>
</ol>
<p>The "membership set" introduced above, is in the form of a sparse, indexed merkle tree.
This membership set can be maintained on-chain, off-chain or as a hybrid depending on the network's storage costs.
Waku makes use of a hybrid membership set,
where insertions are tracked in a smart contract.
In addition, each Waku node maintains a local copy of the tree,
which is updated upon each insertion.</p>
<p>Users register themselves with a hash of a locally generated secret,
which is then inserted into the tree at the next available index.
After having registered, users can prove their membership by proving their knowledge of the pre-image of the respective leaf in the tree.
The leaf hashes are also referred to as commitments of the respective users.
The actual proof is done by a <a href="https://ethereum.org/en/developers/tutorials/merkle-proofs-for-offline-data-integrity/" target="_blank" rel="noopener noreferrer">Merkle Inclusion Proof</a>, which is a type of ZK proof.</p>
<p>The circuit ensures that the user's secret does indeed hash to a leaf in the tree,
and that the provided Merkle proof is valid.</p>
<p>After a User generates this Merkle proof,
they can transmit it to other users,
who can verify the proof.
Including a message's hash within the proof generation,
additionally guarantees integrity of that message.</p>
<p>A malicious user could generate multiple proofs per epoch.
they generate multiple proofs per epoch.
However, when multiple proofs are generated per epoch,
the malicious user's secret is exposed, which strongly disincentivizes this attack.
This mechanism is further described in <a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#malicious-user-secret-interpolation-mechanism">malicious User secret interpolation mechanism</a></p>
<p>Note: This blog post describes rln-v1, which excludes the range check in favor of a global rate limit for all users,
which is once per time window. This version is currently in use in waku-rln-relay.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-protocol-parameters">RLN Protocol parameters<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#rln-protocol-parameters" class="hash-link" aria-label="Direct link to RLN Protocol parameters" title="Direct link to RLN Protocol parameters">​</a></h2>
<p>Given below is the set of cryptographic primitives,
and constants that are used in the RLN protocol.</p>
<ol>
<li>Proving System: <a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer"><code>groth16</code></a></li>
<li>Elliptic Curve: <a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer"><code>bn254</code></a> (aka bn128) (not to be confused with the 254 bit Weierstrass curve)</li>
<li>Finite Field: Prime-order subgroup of the group of points on the <code>bn254</code> curve</li>
<li>Default Merkle Tree Height: <code>20</code></li>
<li>Hashing algorithm: <a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer"><code>Poseidon</code></a></li>
<li>Merkle Tree: <a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer"><code>Sparse Indexed Merkle Tree</code></a></li>
<li>Messages per epoch: <code>1</code></li>
<li>Epoch duration: <code>10 seconds</code></li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="malicious-user-secret-interpolation-mechanism">Malicious User secret interpolation mechanism<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#malicious-user-secret-interpolation-mechanism" class="hash-link" aria-label="Direct link to Malicious User secret interpolation mechanism" title="Direct link to Malicious User secret interpolation mechanism">​</a></h2>
<blockquote>
<p>note: all the parameters mentioned below are elements in the finite field mentioned above.</p>
</blockquote>
<p>The private inputs to the circuit are as follows: -</p>
<div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">identitySecret: the randomly generated secret of the user</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">identityPathIndex: the index of the commitment derived from the secret</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">pathElements: elements included in the path to the index of the commitment</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>Following are the public inputs to the circuit -</p>
<div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">x: hash of the signal to the finite field</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">rlnIdentifier: application-specific identifier which this proof is being generated for</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">epoch: the timestamp which this proof is being generated for</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>The outputs of the circuit are as follows: -</p>
<div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">y: result of Shamir's secret sharing calculation</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">root: root of the Merkle tree obtained after applying the inclusion proof</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">nullifier: uniquely identifies a message, derived from rlnIdentifier, epoch, and the user's secret</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>With the above data in mind, following is the circuit pseudocode -</p>
<div class="codeBlockContainer_EB2s codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#F8F8F2;--prism-background-color:rgba(var(--lsd-surface-secondary), 0.08)"><div class="codeBlockContent_ugSV"><pre tabindex="0" class="prism-code language-text codeBlock_TWhw thin-scrollbar"><code class="codeBlockLines_LDrR"><span class="token-line" style="color:#F8F8F2"><span class="token plain">identityCommitment = Poseidon([identitySecret])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">root = MerkleInclusionProof(identityCommitment, identityPathIndex, pathElements)</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">externalNullifier = Poseidon([epoch, rlnIdentifier])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">a1 = Poseidon([identitySecret, externalNullifier])</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">y = identitySecret + a1 * x</span><br></span><span class="token-line" style="color:#F8F8F2"><span class="token plain">nullifier = Poseidon([a1])</span><br></span></code></pre><div class="buttonGroup_Qu4e"><button type="button" aria-label="Copy code to clipboard" title="Copy" class="clean-btn"><span class="copyButtonIcons_an20" aria-hidden="true"><div class="icon_S7Kx m_thRi copyButtonIcon_ZL7v"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M2.917 12.833q-.482 0-.825-.343a1.12 1.12 0 0 1-.342-.823V3.5h1.167v8.167h6.416v1.166zM5.25 10.5q-.481 0-.824-.343a1.12 1.12 0 0 1-.343-.824v-7q0-.48.343-.824t.824-.342h5.25q.481 0 .824.343t.343.823v7q0 .482-.343.825a1.12 1.12 0 0 1-.824.342zm0-1.167h5.25v-7H5.25z"></path></svg></div></span></button></div></div></div>
<p>To interpolate the secret of a user who has sent multiple signals during the same epoch to the same rln-based application, we may make use of the following formula -</p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>a</mi><mn>1</mn></msub><mo>=</mo><mfrac><mrow><mo stretchy="false">(</mo><msub><mi>y</mi><mn>1</mn></msub><mo>−</mo><msub><mi>y</mi><mn>2</mn></msub><mo stretchy="false">)</mo></mrow><mrow><mo stretchy="false">(</mo><msub><mi>x</mi><mn>1</mn></msub><mo>−</mo><msub><mi>x</mi><mn>2</mn></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">a_1 = {(y_1 - y_2) \over (x_1 - x_2)}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.53em;vertical-align:-0.52em"></span><span class="mord"><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.01em"><span style="top:-2.655em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mtight"><span class="mord mathnormal mtight">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:0em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span><span style="top:-3.23em"><span class="pstrut" style="height:3em"></span><span class="frac-line" style="border-bottom-width:0.04em"></span></span><span style="top:-3.485em"><span class="pstrut" style="height:3em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mopen mtight">(</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mbin mtight">−</span><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3173em"><span style="top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em"><span class="pstrut" style="height:2.5em"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.143em"><span></span></span></span></span></span></span><span class="mclose mtight">)</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.52em"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></p>
<p>where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">y_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">x_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">y_2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">2</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> are shares from different messages</p>
<p>subsequently, we may use one pair of the shares, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">y_1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span></span></span></span> to obtain the <code>identitySecret</code></p>
<p><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mi>d</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi>i</mi><mi>t</mi><mi>y</mi><mi>S</mi><mi>e</mi><mi>c</mi><mi>r</mi><mi>e</mi><mi>t</mi><mo>=</mo><msub><mi>y</mi><mn>1</mn></msub><mo>−</mo><msub><mi>a</mi><mn>1</mn></msub><mo>∗</mo><mi>x</mi></mrow><annotation encoding="application/x-tex">identitySecret = y_1 - a_1 * x</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord mathnormal">i</span><span class="mord mathnormal">d</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord mathnormal">i</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal" style="margin-right:0.05764em">S</span><span class="mord mathnormal">ecre</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.7778em;vertical-align:-0.1944em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:-0.0359em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6153em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.4306em"></span><span class="mord mathnormal">x</span></span></span></span></p>
<p>This enables RLN to be used for rate limiting with a <em>global</em> limit. For arbitrary limits,
please refer to an article written by @curryrasul, <a href="https://mirror.xyz/privacy-scaling-explorations.eth/iCLmH1JVb7fDqp6Mms2NR001m2_n5OOSHsLF2QrxDnQ" target="_blank" rel="noopener noreferrer">rln-v2</a>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="wakus-problem-with-dos">Waku's problem with DoS<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#wakus-problem-with-dos" class="hash-link" aria-label="Direct link to Waku's problem with DoS" title="Direct link to Waku's problem with DoS">​</a></h2>
<p>In a decentralized, privacy focused messaging system like <a href="https://waku.org/" target="_blank" rel="noopener noreferrer">Waku</a>,
Denial of Service (DoS) vulnerabilities are very common, and must be addressed to promote network scale and optimal bandwidth utilization.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="dos-prevention-with-user-metadata">DoS prevention with user metadata<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#dos-prevention-with-user-metadata" class="hash-link" aria-label="Direct link to DoS prevention with user metadata" title="Direct link to DoS prevention with user metadata">​</a></h3>
<p>There are a couple of ways a user can be rate-limited, either -</p>
<ol>
<li>IP Logging</li>
<li>KYC Logging</li>
</ol>
<p>Both IP and KYC logging prevent systems from being truly anonymous, and hence, cannot be used as a valid DoS prevention mechanism for Waku.</p>
<p>RLN can be used as an alternative, which provides the best of both worlds, i.e a permissioned membership set, as well as anonymous signaling.
However, we are bound by k-anonymity rules of the membership set.</p>
<p><a href="https://rfc.vac.dev/waku/standards/core/17/rln-relay" target="_blank" rel="noopener noreferrer">Waku-RLN-Relay</a> is a <a href="https://libp2p.io/" target="_blank" rel="noopener noreferrer">libp2p</a> pubsub validator that verifies if a proof attached to a given message is valid.
In case the proof is valid, the message is relayed.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="performance-analysis">Performance analysis<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#performance-analysis" class="hash-link" aria-label="Direct link to Performance analysis" title="Direct link to Performance analysis">​</a></h2>
<blockquote>
<p>Test bench specs: AMD EPYC 7502P 32-Core, 4x32GB DDR4 Reg.ECC Memory</p>
</blockquote>
<p>This simulation was conducted by @alrevuelta, and is described in more detail <a href="https://github.com/waku-org/research/issues/23" target="_blank" rel="noopener noreferrer">here</a>.</p>
<p>The simulation included 100 waku nodes running in parallel.</p>
<p>Proof generation times -
</p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="img" src="https://vac.dev/assets/images/proof_generation_time-195632e4864fa4c5f883895f2ea9e9e3.png" width="1547" height="1096" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>Proof verification times -
</p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="img" src="https://vac.dev/assets/images/proof_verification_time-c95708ef2a4fc0470114fbceebc6bc30.png" width="1564" height="1214" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<p>A spammer node publishes 3000 msg/epoch, which is detected by all connected nodes, and subsequently disconnect to prevent further spam -
</p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="img" src="https://vac.dev/assets/images/spam_prevention_in_action-50221f227e3d94be5aeae45193cc04ea.png" width="1574" height="1108" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="security-analysis">Security analysis<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#security-analysis" class="hash-link" aria-label="Direct link to Security analysis" title="Direct link to Security analysis">​</a></h2>
<p><a href="https://doi.org/10.1007/s00145-018-9280-5" target="_blank" rel="noopener noreferrer">Barbulescu and Duquesne</a>
conclude that that the <code>bn254</code> curve has only 100 bits of security.
Since the bn254 curve has a small embedding degree,
it is vulnerable to the <a href="https://en.wikipedia.org/wiki/MOV_attack" target="_blank" rel="noopener noreferrer">MOV attack</a>.
However, the MOV attack is only applicable to pairings,
and not to the elliptic curve itself.
It is acceptable to use the bn254 curve for RLN,
since the circuit does not make use of pairings.</p>
<p><a href="https://github.com/vacp2p/research/issues/155" target="_blank" rel="noopener noreferrer">An analysis</a> on the number of rounds in the Poseidon hash function was done,
which concluded that the hashing rounds should <em>not</em> be reduced,</p>
<p>The <a href="https://github.com/vacp2p/rln-contract" target="_blank" rel="noopener noreferrer">smart contracts</a> have <em>not</em> been audited, and are not recommended for real world deployments <em>yet</em>.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="storage-analysis">Storage analysis<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#storage-analysis" class="hash-link" aria-label="Direct link to Storage analysis" title="Direct link to Storage analysis">​</a></h2>
<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mn>32</mn><mtext>&nbsp;</mtext><mi>b</mi><mi>y</mi><mi>t</mi><mi>e</mi><mi>s</mi><mspace linebreak="newline"></mspace><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>h</mi><mi>e</mi><mi>i</mi><mi>g</mi><mi>h</mi><mi>t</mi><mo>=</mo><mn>20</mn><mspace linebreak="newline"></mspace><mi>t</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>e</mi><mi>a</mi><mi>v</mi><mi>e</mi><mi>s</mi><mo>=</mo><msup><mn>2</mn><mn>20</mn></msup><mspace linebreak="newline"></mspace><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mi>t</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi><mi mathvariant="normal">_</mi><mi>l</mi><mi>e</mi><mi>a</mi><mi>v</mi><mi>e</mi><mi>s</mi><mo>∗</mo><mi>c</mi><mi>o</mi><mi>m</mi><mi>m</mi><mi>i</mi><mi>t</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>t</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mspace linebreak="newline"></mspace><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><msup><mn>2</mn><mn>20</mn></msup><mo>∗</mo><mn>32</mn><mo>=</mo><mn>33</mn><mo separator="true">,</mo><mn>554</mn><mo separator="true">,</mo><mn>432</mn><mspace linebreak="newline"></mspace><mo>∴</mo><mi>m</mi><mi>a</mi><mi>x</mi><mi mathvariant="normal">_</mi><mi>t</mi><mi>r</mi><mi>e</mi><mi>e</mi><mi mathvariant="normal">_</mi><mi>s</mi><mi>i</mi><mi>z</mi><mi>e</mi><mo>=</mo><mn>33.55</mn><mtext>&nbsp;</mtext><mi>m</mi><mi>e</mi><mi>g</mi><mi>a</mi><mi>b</mi><mi>y</mi><mi>t</mi><mi>e</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">commitment\_size = 32\ bytes \\
tree\_height =20 \\
total\_leaves = 2^{20} \\
max\_tree\_size = total\_leaves * commitment\_size \\
max\_tree\_size = 2^{20} * 32 = 33,554,432 \\
∴max\_tree\_size = 33.55\ megabytes</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord">32</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">b</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal">t</span><span class="mord mathnormal">es</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">h</span><span class="mord mathnormal">e</span><span class="mord mathnormal">i</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">h</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">20</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">e</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">es</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8641em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8641em"><span style="top:-3.113em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1.0044em;vertical-align:-0.31em"></span><span class="mord mathnormal">t</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal" style="margin-right:0.01968em">l</span><span class="mord mathnormal">e</span><span class="mord mathnormal">a</span><span class="mord mathnormal" style="margin-right:0.03588em">v</span><span class="mord mathnormal">es</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">co</span><span class="mord mathnormal">mmi</span><span class="mord mathnormal">t</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">t</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8641em"></span><span class="mord"><span class="mord">2</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8641em"><span style="top:-3.113em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">20</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">∗</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord">32</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em"></span><span class="mord">33</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">554</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord">432</span></span><span class="mspace newline"></span><span class="base"><span class="strut" style="height:0.6922em"></span><span class="mrel amsrm">∴</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9695em;vertical-align:-0.31em"></span><span class="mord mathnormal">ma</span><span class="mord mathnormal">x</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">t</span><span class="mord mathnormal">ree</span><span class="mord" style="margin-right:0.02778em">_</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">ze</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em"></span><span class="mord">33.55</span><span class="mspace">&nbsp;</span><span class="mord mathnormal">m</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.03588em">g</span><span class="mord mathnormal">ab</span><span class="mord mathnormal" style="margin-right:0.03588em">y</span><span class="mord mathnormal">t</span><span class="mord mathnormal">es</span></span></span></span></span>
<p>The storage overhead introduced by RLN is minimal.
RLN only requires 34 megabytes of storage, which poses no problem on most end-user hardware, with the exception of IoT/microcontrollers.
Still, we are working on further optimizations allowing proof generation without having to store the full tree.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="the-bare-minimum-requirements-to-run-rln">The bare minimum requirements to run RLN<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#the-bare-minimum-requirements-to-run-rln" class="hash-link" aria-label="Direct link to The bare minimum requirements to run RLN" title="Direct link to The bare minimum requirements to run RLN">​</a></h2>
<p>With proof generation time in sub-second latency, along with low storage overhead for the tree,
it is possible for end users to generate and verify RLN proofs on a modern smartphone.</p>
<p>Following is a demo provided by @rramos that demonstrates
<a href="https://drive.google.com/file/d/1ITLYrDOQrHQX2_3Q6O5EqKPYJN8Ye2gF/view?usp=sharing" target="_blank" rel="noopener noreferrer">waku-rln-relay used in react native</a>.</p>
<blockquote>
<p>Warning: The react native sdk will be deprecated soon, and the above demo should serve as a PoC for RLN on mobiles</p>
</blockquote>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="rln-usage-guide">RLN usage guide<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#rln-usage-guide" class="hash-link" aria-label="Direct link to RLN usage guide" title="Direct link to RLN usage guide">​</a></h2>
<p><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a> implements api's that allow users to handle operations to the tree,
as well as generate/verify RLN proofs.</p>
<p>Our main implementation of RLN can be accessed via this Rust <a href="https://crates.io/crates/rln" target="_blank" rel="noopener noreferrer">crate</a>,
which is documented <a href="https://docs.rs/rln/0.4.1/rln/public/struct.RLN.html" target="_blank" rel="noopener noreferrer">here</a>.
It can used in other langugages via the FFI API, which is documented <a href="https://docs.rs/rln/0.4.1/rln/ffi/index.html" target="_blank" rel="noopener noreferrer">here</a>.
The usage of RLN in Waku is detailed in our <a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a>,
which provides step-by-step instructions on how to run Waku-RLN-Relay.</p>
<p>Following is a diagram that will help understand the dependency tree -</p>
<p></p><div class="wrapper_SWrM active_qZD5"><img decoding="async" loading="lazy" alt="rln-dep-tree" src="https://vac.dev/assets/images/rln_dep_tree-0bf1837513daecde1a3de4deb9a8855f.jpg" width="631" height="552" class="img_ev3q"><button class="fullscreenButton_Bocn lsd-icon-button lsd-icon-button--medium lsd-icon-button--outlined"><div class="icon_S7Kx m_thRi"><svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="none" viewBox="0 0 14 14"><path fill="#fff" d="M1.75 2.917V5.25h1.167V2.917H5.25V1.75H2.917A1.17 1.17 0 0 0 1.75 2.917M2.917 8.75H1.75v2.333a1.17 1.17 0 0 0 1.167 1.167H5.25v-1.167H2.917zm8.166 2.333H8.75v1.167h2.333a1.17 1.17 0 0 0 1.167-1.167V8.75h-1.167zm0-9.333H8.75v1.167h2.333V5.25h1.167V2.917a1.17 1.17 0 0 0-1.167-1.167"></path></svg></div></button></div><p></p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="future-work">Future work<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#future-work" class="hash-link" aria-label="Direct link to Future work" title="Direct link to Future work">​</a></h2>
<ul>
<li>Optimizations to zerokit for proof generation time.</li>
<li>Incrementing tree depth from 20 to 32, to allow more memberships.</li>
<li>Optimizations to the smart contract.</li>
<li>An ability to signal validity of a message in different time windows.</li>
<li>Usage of proving systems other than Groth16.</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/rln-anonymous-dos-prevention#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h2>
<ul>
<li><a href="https://github.com/rate-limiting-nullifier/circom-rln" target="_blank" rel="noopener noreferrer">RLN Circuits</a></li>
<li><a href="https://github.com/vacp2p/zerokit" target="_blank" rel="noopener noreferrer">Zerokit</a></li>
<li><a href="https://rfc.vac.dev/vac/32/rln-v1" target="_blank" rel="noopener noreferrer">RLN-V1 RFC</a></li>
<li><a href="https://rfc.vac.dev/vac/raw/rln-v2" target="_blank" rel="noopener noreferrer">RLN-V2 RFC</a></li>
<li><a href="https://hackmd.io/7cBCMU5hS5OYv8PTaW2wAQ?view" target="_blank" rel="noopener noreferrer">RLN Implementers guide</a></li>
<li><a href="https://eprint.iacr.org/2016/260.pdf" target="_blank" rel="noopener noreferrer">groth16</a></li>
<li><a href="https://eprint.iacr.org/2013/879.pdf" target="_blank" rel="noopener noreferrer">bn254</a></li>
<li><a href="https://eprint.iacr.org/2019/458.pdf" target="_blank" rel="noopener noreferrer">Poseidon Hash</a></li>
<li><a href="https://github.com/rate-limiting-nullifier/pmtree" target="_blank" rel="noopener noreferrer">Sparse Indexed Merkle Tree</a></li>
<li><a href="https://doi.org/10.1007/s00145-018-9280-5" target="_blank" rel="noopener noreferrer">Updating key size estimations for pairings</a></li>
</ul>]]></content>
<author>
<name>Aaryamann</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks]]></title>
<id>https://vac.dev/rlog/GossipSub Improvements</id>
<link href="https://vac.dev/rlog/GossipSub Improvements"/>
<updated>2023-11-06T12:00:00.000Z</updated>
<summary type="html"><![CDATA[GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks]]></summary>
<content type="html"><![CDATA[<p>GossipSub Improvements: Evolution of Overlay Design and Message Dissemination in Unstructured P2P Networks</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="motivitation">Motivitation<a href="https://vac.dev/rlog/GossipSub%20Improvements#motivitation" class="hash-link" aria-label="Direct link to Motivitation" title="Direct link to Motivitation">​</a></h2>
<p>We have been recently working on analyzing and improving the performance of the GossipSub protocol for large messages,
as in the case of Ethereum Improvement Proposal <a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">EIP-4844</a>.
This work led to a comprehensive study of unstructured P2P networks.
The intention was to identify the best practices that can serve as guidelines for performance improvement and scalability of P2P networks.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/GossipSub%20Improvements#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Nodes in an unstructured p2p network form self-organizing overlay(s) on top of the IP infrastructure to facilitate different services like information dissemination,
query propagation, file sharing, etc. The overlay(s) can be as optimal as a tree-like structure or as enforcing as a fully connected mesh.</p>
<p>Due to peer autonomy and a trustless computing environment, some peers may deviate from the expected operation or even leave the network.
At the same time, the underlying IP layer is unreliable.</p>
<p>Therefore, tree-like overlays are not best suited for reliable information propagation.
Moreover, tree-based solutions usually result in significantly higher message dissemination latency due to suboptimal branches.</p>
<p>Flooding-based solutions, on the other hand, result in maximum resilience against adversaries and achieve minimal message dissemination latency because the message propagates through all (including the optimal) paths.
Redundant transmissions help maintain the integrity and security of the network in the presence of adversaries and high node failure but significantly increase network-wide bandwidth utilization, cramming the bottleneck links.</p>
<p>An efficient alternative is to lower the number of redundant transmissions by D-regular broadcasting, where a peer will likely receive (or relay) a message from up to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> random peers.
Publishing through a D-regular overlay triggers approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">N \times D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> transmissions.
Reducing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> reduces the redundant transmissions but compromises reachability and latency.
Sharing metadata through a K-regular overlay (where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>&gt;</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">K &gt; D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7224em;vertical-align:-0.0391em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">&gt;</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span>) allows nodes to pull missing messages.</p>
<p>GossipSub [<a href="https://arxiv.org/pdf/2007.02754.pdf" target="_blank" rel="noopener noreferrer">1</a>] benefits from full-message (D-regular) and metadata-only (k-regular) overlays.
Alternatively, a metadata-only overlay can be used, requiring a pull-based operation that significantly minimizes bandwidth utilization at the cost of increased latency.</p>
<p>Striking the right balance between parameters like <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mo separator="true">,</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">D, K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8778em;vertical-align:-0.1944em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal" style="margin-right:0.07153em">K</span></span></span></span>, pull-based operation, etc., can yield application-specific performance tuning, but scalability remains a problem.</p>
<p>At the same time, many other aspects can significantly contribute to the network's performance and scalability.
One option is to realize peers' suitability and continuously changing capabilities while forming overlays.</p>
<p>For instance, a low-bandwidth link near a publisher can significantly demean the entire network's performance.
Reshuffling of peering links according to the changing network conditions can lead to superior performance.</p>
<p>Laying off additional responsibilities to more capable nodes (super nodes) can alleviate peer cramming, but it makes the network susceptible to adversaries/peer churn.
Grouping multiple super nodes to form virtual node(s) can solve this problem.</p>
<p>Similarly, flat (single-tier) overlays cannot address the routing needs in large (geographically dispersed) networks.</p>
<p>Hierarchical (Multi-tier) overlays with different intra/inter-overlay routing solutions can better address these needs.
Moreover, using message aggregation schemes for grouping multiple messages can save bandwidth and provide better resilience against adversaries/peer churn.</p>
<p>This article's primary objective is to investigate the possible choices that can empower an unstructured P2P network to achieve superior performance for the broadest set of applications.
We look into different constraints imposed by application-specific needs (performance goals) and investigate various choices that can augment the network's performance.
We explore overlay designs/freshness, peer selection approaches, message-relaying mechanisms, and resilience against adversaries/peer churn.
We consider GossipSub a baseline protocol to explore various possibilities and decisively commit to the ones demonstrating superior performance.
We also discuss the current state and, where applicable, propose a strategic plan for embedding new features to the GossipSub protocol.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal1-low-latency-operation">GOAL1: Low Latency Operation<a href="https://vac.dev/rlog/GossipSub%20Improvements#goal1-low-latency-operation" class="hash-link" aria-label="Direct link to GOAL1: Low Latency Operation" title="Direct link to GOAL1: Low Latency Operation">​</a></h2>
<p>Different applications, like blockchain, streaming, etc., impose strict time bounds on network-wide message dissemination latency.
A message delivered after the imposed time bounds is considered as dropped.
An early message delivery in applications like live streaming can further enhance the viewing quality.</p>
<p>The properties and nature of the overlay network topology significantly impact the performance of services and applications executed on top of them.
Studying and devising mechanisms for better overlay design and message dissemination is paramount to achieving superior performance.</p>
<p>Interestingly, shortest-path message delivery trees have many limitations:</p>
<ol>
<li>Changing network dynamics requires a quicker and continuous readjustment of the multicast tree.</li>
<li>The presence of resource-constrained (bandwidth/compute, etc.) nodes in the overlay can result in congestion.</li>
<li>Node failure can result in partitions, making many segments unreachable.</li>
<li>Assuring a shortest-path tree-like structure requires a detailed view of the underlying (and continuously changing) network topology.</li>
</ol>
<p>Solutions involve creating multiple random trees to add redundancy [<a href="https://ieeexplore.ieee.org/abstract/document/6267905" target="_blank" rel="noopener noreferrer">2</a>].
Alternatives involve building an overlay mesh and forwarding messages through the multicast delivery tree (eager push).</p>
<p>Metadata is shared through the overlay links so that the nodes can ask for missing messages (lazy push or pull-based operation) through the overlay links.
New nodes are added from the overlay on node failure, but it requires non-faulty node selection.</p>
<p>GossipSub uses eager push (through overlay mesh) and lazy push (through IWANT messages).</p>
<p>The mesh degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>L</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>≤</mo><mi>D</mi><mo>≤</mo><msub><mi>D</mi><mrow><mi>H</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{Low} \leq D \leq D_{High}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">L</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8193em;vertical-align:-0.136em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.08125em">H</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> is crucial in deciding message dissemination latency.
A smaller value for <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> results in higher latency due to increased rounds, whereas a higher <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> reduces latency on the cost of increased bandwidth.
At the same time, keeping <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> independent of the growing network size (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span>) may increase network-wide message dissemination latency.
Adjusting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> with <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.10903em">N</span></span></span></span> maintains similar latency on the cost of increased workload for peers.
Authors in [<a href="https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf" target="_blank" rel="noopener noreferrer">3</a>] suggest only a logarithmic increase in <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> to maintain a manageable workload for peers.
In [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>], it is reported that the average mesh degree should not exceed <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub><mo>=</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">D_{avg} = \ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span> for an optimal operation,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span> is a small constant.</p>
<p>Moreover, quicker shuffling of peers results in better performance in the presence of resource-constrained nodes or node failure [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>].</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal2-considering-heterogeneity-in-overlay-design">GOAL2: Considering Heterogeneity In Overlay Design<a href="https://vac.dev/rlog/GossipSub%20Improvements#goal2-considering-heterogeneity-in-overlay-design" class="hash-link" aria-label="Direct link to GOAL2: Considering Heterogeneity In Overlay Design" title="Direct link to GOAL2: Considering Heterogeneity In Overlay Design">​</a></h2>
<p>Random peering connections in P2P overlays represent a stochastic process. It is inherently difficult to precisely model the performance of such systems.
Most of the research on P2P networks provides simulation results assuming nodes with similar capabilities.
The aspect of dissimilar capabilities and resource-constrained nodes is less explored.</p>
<p>It is discussed in GOAL1 that overlay mesh results in better performance if <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> does not exceed <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">\ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span>.
Enforcing all the nodes to have approximately <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo>+</mo><mi>C</mi></mrow><annotation encoding="application/x-tex">\ln(N) + C</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.07153em">C</span></span></span></span> peers makes resource-rich nodes under-utilized, while resource-constrained nodes are overloaded.
At the same time, connecting high-bandwidth nodes through a low-bandwidth node undermines the network's performance.
Ideally, the workload on any node should not exceed its available resources.
A better solution involves a two-phased operation:</p>
<ol>
<li>
<p>Every node computes its available bandwidth and selects a node degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> proportional to its available bandwidth [<a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">4</a>].
Different bandwidth estimation approaches are suggested in literature [<a href="https://ieeexplore.ieee.org/abstract/document/1224454" target="_blank" rel="noopener noreferrer">5</a>,<a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">6</a>].
Simple bandwidth estimation approaches like variable packet size probing [<a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">6</a>] yield similar results with less complexity.
It is also worth mentioning that many nodes may want to allocate only a capped share of their bandwidth to the network.
Lowering <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> according to the available bandwidth can still prove helpful.
Additionally, bandwidth preservation at the transport layer through approaches like µTP can be useful.
To further conform to the suggested mesh-degree average <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>, every node tries achieving this average within its neighborhood, resulting in an overall similar <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>a</mi><mi>v</mi><mi>g</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{avg}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">a</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">vg</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span>.</p>
</li>
<li>
<p>From the available local view, every node tries connecting peers with the lowest latency until <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> connections are made.
We suggest referring to the peering solution discussed in GOAL5 to avoid network partitioning.</p>
</li>
</ol>
<p>The current GossipSub design considers homogeneous peers, and every node tries maintaining <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>D</mi><mrow><mi>L</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>≤</mo><mi>D</mi><mo>≤</mo><msub><mi>D</mi><mrow><mi>H</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">D_{Low} \leq D \leq D_{High}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">L</span><span class="mord mathnormal mtight">o</span><span class="mord mathnormal mtight" style="margin-right:0.02691em">w</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.8193em;vertical-align:-0.136em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">≤</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.9694em;vertical-align:-0.2861em"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em">D</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.08125em">H</span><span class="mord mathnormal mtight">i</span><span class="mord mathnormal mtight" style="margin-right:0.03588em">g</span><span class="mord mathnormal mtight">h</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em"><span></span></span></span></span></span></span></span></span></span> connections.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal3-bandwidth-optimization">GOAL3: Bandwidth Optimization<a href="https://vac.dev/rlog/GossipSub%20Improvements#goal3-bandwidth-optimization" class="hash-link" aria-label="Direct link to GOAL3: Bandwidth Optimization" title="Direct link to GOAL3: Bandwidth Optimization">​</a></h2>
<p>Redundant message transmissions are essential for handling adversaries/node failure. However, these transmissions result in traffic bursts, cramming many overlay links.
This not only adds to the network-wide message dissemination latency but a significant share of the network's bandwidth is wasted on (usually) unnecessary transmissions.
It is essential to explore solutions that can minimize the number of redundant transmissions while assuring resilience against node failures.</p>
<p>Many efforts have been made to minimize the impact of redundant transmissions.
These solutions include multicast delivery trees, metadata sharing to enable pull-based operation, in-network information caching, etc. [<a href="https://dl.acm.org/doi/abs/10.1145/945445.945473" target="_blank" rel="noopener noreferrer">7</a>,<a href="https://link.springer.com/chapter/10.1007/11558989_12" target="_blank" rel="noopener noreferrer">8</a>].
GossipSub employs a hybrid of eager push (message dissemination through the overlay) and lazy push (a pull-based operation by the nodes requiring information through IWANT messages).</p>
<p>A better alternative to simple redundant transmission is to use message aggregation [<a href="https://ieeexplore.ieee.org/abstract/document/8737576" target="_blank" rel="noopener noreferrer">9</a>,<a href="https://dl.acm.org/doi/abs/10.1145/1993636.1993676" target="_blank" rel="noopener noreferrer">10</a>,<a href="https://ieeexplore.ieee.org/abstract/document/4276446" target="_blank" rel="noopener noreferrer">11</a>] for the GossipSub protocol.
As a result, redundant message transmissions can serve as a critical advantage of the GossipSub protocol.
Suppose that we have three equal-length messages <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mn>1</mn><mo separator="true">,</mo><mi>x</mi><mn>2</mn><mo separator="true">,</mo><mi>x</mi><mn>3</mn></mrow><annotation encoding="application/x-tex">x1, x2, x3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em"></span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em"></span><span class="mord mathnormal">x</span><span class="mord">3</span></span></span></span>. Assuming an XOR coding function,
we know two trivial properties: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mn>1</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>=</mo><mi>x</mi><mn>1</mn></mrow><annotation encoding="application/x-tex">x1 \oplus x2 \oplus x2 = x1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:0.6444em"></span><span class="mord mathnormal">x</span><span class="mord">1</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><mi>x</mi><mn>1</mn><mi mathvariant="normal">∣</mi><mo>=</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mn>1</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mo>⊕</mo><mi>x</mi><mn>2</mn><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\vert x1 \vert = \vert x1 \oplus x2 \oplus x2 \vert</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">∣</span><span class="mord mathnormal">x</span><span class="mord">1∣</span><span class="mspace" style="margin-right:0.2778em"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord">∣</span><span class="mord mathnormal">x</span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em"></span><span class="mord mathnormal">x</span><span class="mord">2</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">⊕</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mord mathnormal">x</span><span class="mord">2∣</span></span></span></span>.</p>
<p>This implies that instead of sending messages individually, we can encode and transmit composite message(s) to the network.
The receiver can reconstruct the original message from encoded segments.
As a result, fewer transmissions are sufficient for sending more messages to the network.</p>
<p>However, sharing linear combinations of messages requires organizing messages in intervals,
and devising techniques to identify all messages belonging to each interval.
In addition, combining messages from different publishers requires more complex arrangements,
involving embedding publisher/message IDs, delayed forwarding (to accommodate more messages), and mechanisms to ensure the decoding of messages at all peers.
Careful application-specific need analysis can help decide the benefits against the added complexity.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal4-handling-large-messages">GOAL4: Handling Large Messages<a href="https://vac.dev/rlog/GossipSub%20Improvements#goal4-handling-large-messages" class="hash-link" aria-label="Direct link to GOAL4: Handling Large Messages" title="Direct link to GOAL4: Handling Large Messages">​</a></h2>
<p>Many applications require transferring large messages for their successful operation. For instance, database/blockchain transactions [<a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">12</a>].
This introduces two challenges:</p>
<ol>
<li>Redundant large message transmissions result in severe network congestion.</li>
<li>Message transmissions follow a store/forward process at all peers, which is inefficient in the case of large messages.</li>
</ol>
<p>The above-mentioned challenges result in a noticeable increase in message dissemination latency and bandwidth wastage.
Most of the work done for handling large messages involves curtailing redundant transmissions using multicast delivery trees,
reducing the number of fanout nodes, employing in-network message caching, pull-based operation, etc.</p>
<p>Approaches like message aggregation also prove helpful in minimizing bandwidth wastage.</p>
<p>Our recent work on GossipSub improvements (still a work in progress) suggests the following solutions to deal with large message transmissions:</p>
<ol>
<li>
<p>Using IDontWant message proposal [<a href="https://github.com/libp2p/specs/pull/413" target="_blank" rel="noopener noreferrer">13</a>] and staggered sending.</p>
<p>IDontWant message helps curtail redundant transmissions by letting other peers know we have already received the message.
Staggered sending enables relaying the message to a short subset of peers in each round.
We argue that simultaneously relaying a message to all peers hampers the effectiveness of the IDontWant message.
Therefore, using the IDontWant message with staggered sending can yield better results by allowing timely reception and processing of IDontWant messages.</p>
</li>
<li>
<p>Message transmissions follow a store/forward process at all peers that is inefficient in the case of large messages.
We can parallelize message transmission by partitioning large messages into smaller fragments, letting intermediate peers relay these fragments as soon as they receive them.</p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal5-scalability">GOAL5: Scalability<a href="https://vac.dev/rlog/GossipSub%20Improvements#goal5-scalability" class="hash-link" aria-label="Direct link to GOAL5: Scalability" title="Direct link to GOAL5: Scalability">​</a></h2>
<p>P2P networks are inherently scalable because every incoming node brings in bandwidth and compute resources.
In other words, we can keep adding nodes to the network as long as every incoming node brings at-least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo>×</mo><mi>D</mi></mrow><annotation encoding="application/x-tex">R \times D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7667em;vertical-align:-0.0833em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span><span class="mspace" style="margin-right:0.2222em"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em"></span></span><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> bandwidth,
where <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi></mrow><annotation encoding="application/x-tex">R</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.00773em">R</span></span></span></span> is average data arrival rate.
It is worth mentioning that network-wide message dissemination requires at-least <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">⌈</mo><msub><mrow><mi>log</mi><mo>⁡</mo></mrow><mi>D</mi></msub><mo stretchy="false">(</mo><mi>N</mi><mo stretchy="false">)</mo><mo stretchy="false">⌉</mo></mrow><annotation encoding="application/x-tex">\lceil \log_D (N) \rceil</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em"></span><span class="mopen">⌈</span><span class="mop"><span class="mop">lo<span style="margin-right:0.01389em">g</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2342em"><span style="top:-2.4559em;margin-right:0.05em"><span class="pstrut" style="height:2.7em"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em">D</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2441em"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.10903em">N</span><span class="mclose">)⌉</span></span></span></span> hops.
Therefore, increasing network size increases message dissemination latency, assuming D is independent of the network size.</p>
<p>Additionally, problems like peer churn, adversaries, heterogeneity, distributed operation, etc., significantly hamper the network's performance.
Most efforts for bringing scalability to the P2P systems have focused on curtailing redundant transmissions and flat overlay adjustments.
Hierarchical overlay designs, on the other hand, are less explored.</p>
<p>Placing a logical structure in unstructured P2P systems can help scale P2P networks.</p>
<p>One possible solution is to use a hierarchical overlay inspired by the approaches [<a href="https://link.springer.com/article/10.1007/s12083-016-0460-5" target="_blank" rel="noopener noreferrer">14</a>,<a href="https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16" target="_blank" rel="noopener noreferrer">15</a>,<a href="https://ieeexplore.ieee.org/abstract/document/9826458" target="_blank" rel="noopener noreferrer">16</a>].
An abstract operation of such overlay design is provided below:</p>
<ol>
<li>
<p>Clustering nodes based on locality, assuming that such peers will have relatively lower intra-cluster latency and higher bandwidth.
For this purpose, every node tries connecting peers with the lowest latency until <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi></mrow><annotation encoding="application/x-tex">D</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em"></span><span class="mord mathnormal" style="margin-right:0.02778em">D</span></span></span></span> connections are made or the cluster limit is reached.</p>
</li>
<li>
<p>A small subset of nodes having the highest bandwidth and compute resources is selected from each cluster.
These super nodes form a fully connected mesh and jointly act as a virtual node,
mitigating the problem of peer churn among super nodes.</p>
</li>
<li>
<p>Virtual nodes form a fully connected mesh to construct a hierarchical overlay.
Each virtual node is essentially a collection of super nodes;
a link to any of the constituent super nodes represents a link to the virtual node.</p>
</li>
<li>
<p>One possible idea is to use GossipSub for intra-cluster message dissemination and FloodSub for inter-cluster message dissemination.</p>
</li>
</ol>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary">Summary<a href="https://vac.dev/rlog/GossipSub%20Improvements#summary" class="hash-link" aria-label="Direct link to Summary" title="Direct link to Summary">​</a></h2>
<p>Overlay acts as a virtual backbone for a P2P network. A flat overlay is more straightforward and allows effortless readjustment to application needs.
On the other hand, a hierarchical overlay can bring scalability at the cost of increased complexity.
Regardless of the overlay design, a continuous readjustment to appropriate peering links is essential for superior performance.
At the same time, bandwidth preservation (through message aggregation, caching at strategic locations, metadata sharing, pull-based operation, etc.) can help minimize latency.
However, problems like peer churn and in-network adversaries can be best alleviated through balanced redundant coverage, and frequent reshuffling of the peering links.</p>
<h1>References</h1>
<ul>
<li>[1] D. Vyzovitis, Y. Napora, D. McCormick, D. Dias, and Y. Psaras, “Gossipsub: Attack-resilient message propagation in the filecoin and eth2. 0 networks,” arXiv preprint arXiv:2007.02754, 2020. Retrieved from <a href="https://arxiv.org/pdf/2007.02754.pdf" target="_blank" rel="noopener noreferrer">https://arxiv.org/pdf/2007.02754.pdf</a></li>
<li>[2] M. Matos, V. Schiavoni, P. Felber, R. Oliveira, and E. Riviere, “Brisa: Combining efficiency and reliability in epidemic data dissemination,” in 2012 IEEE 26th International Parallel and Distributed Processing Symposium. IEEE, 2012, pp. 983–994. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/6267905" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/6267905</a></li>
<li>[3] P. T. Eugster, R. Guerraoui, A. M. Kermarrec, and L. Massouli, “Epidemic information dissemination in distributed systems,” IEEE Computer, vol. 37, no. 5, 2004. Retrieved from <a href="https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf" target="_blank" rel="noopener noreferrer">https://infoscience.epfl.ch/record/83478/files/EugGueKerMas04IEEEComp.pdf</a></li>
<li>[4] D. Frey, “Epidemic protocols: From large scale to big data,” Ph.D. dissertation, Universite De Rennes 1, 2019. Retrieved from <a href="https://inria.hal.science/tel-02375909/document" target="_blank" rel="noopener noreferrer">https://inria.hal.science/tel-02375909/document</a></li>
<li>[5] M. Jain and C. Dovrolis, “End-to-end available bandwidth: measurement methodology, dynamics, and relation with tcp throughput,” IEEE/ACM Transactions on networking, vol. 11, no. 4, pp. 537–549, 2003. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/1224454" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/1224454</a></li>
<li>[6] R. Prasad, C. Dovrolis, M. Murray, and K. Claffy, “Bandwidth estimation: metrics, measurement techniques, and tools,” IEEE network, vol. 17, no. 6, pp. 27–35, 2003. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/1248658" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/1248658</a></li>
<li>[7] D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat, “Bullet: High bandwidth data dissemination using an overlay mesh,” in Proceedings of the nineteenth ACM symposium on Operating systems principles, 2003, pp. 282–297. Retrieved from <a href="https://dl.acm.org/doi/abs/10.1145/945445.945473" target="_blank" rel="noopener noreferrer">https://dl.acm.org/doi/abs/10.1145/945445.945473</a></li>
<li>[8] V. Pai, K. Kumar, K. Tamilmani, V. Sambamurthy, and A. E. Mohr, “Chainsaw: Eliminating trees from overlay multicast,” in Peer-to-Peer Systems IV: 4th International Workshop, IPTPS 2005, Ithaca, NY, USA, February 24-25, 2005. Revised Selected Papers 4. Springer, 2005, pp. 127–140. Retrieved from <a href="https://link.springer.com/chapter/10.1007/11558989_12" target="_blank" rel="noopener noreferrer">https://link.springer.com/chapter/10.1007/11558989_12</a></li>
<li>[9] Y.-D. Bromberg, Q. Dufour, and D. Frey, “Multisource rumor spreading with network coding,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019, pp. 2359–2367. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/8737576" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/8737576</a></li>
<li>[10] B. Haeupler, “Analyzing network coding gossip made easy,” in Proceedings of the forty-third annual ACM symposium on Theory of computing, 2011, pp. 293–302. Retrieved from <a href="https://dl.acm.org/doi/abs/10.1145/1993636.1993676" target="_blank" rel="noopener noreferrer">https://dl.acm.org/doi/abs/10.1145/1993636.1993676</a></li>
<li>[11] S. Yu and Z. Li, “Massive data delivery in unstructured peer-to-peer networks with network coding,” in 6th IEEE/ACIS International Conference on Computer and Information Science (ICIS 2007). IEEE, 2007, pp. 592–597. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/4276446" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/4276446</a></li>
<li>[12] V. Buterin, D. Feist, D. Loerakker, G. Kadianakis, M. Garnett, M. Taiwo, and A. Dietrichs, “Eip-4844: Shard blob transactions scale data-availability of ethereum in a simple, forwards-compatible manner,” 2022. Retrieved from <a href="https://eips.ethereum.org/EIPS/eip-4844" target="_blank" rel="noopener noreferrer">https://eips.ethereum.org/EIPS/eip-4844</a></li>
<li>[13] A. Manning, “Gossipsub extension for epidemic meshes (v1.2.0),” 2022. Retrieved from <a href="https://github.com/libp2p/specs/pull/413" target="_blank" rel="noopener noreferrer">https://github.com/libp2p/specs/pull/413</a></li>
<li>[14] Z. Duan, C. Tian, M. Zhou, X. Wang, N. Zhang, H. Du, and L. Wang, “Two-layer hybrid peer-to-peer networks,” Peer-to-Peer Networking and Applications, vol. 10, pp. 1304–1322, 2017. Retrieved from <a href="https://link.springer.com/article/10.1007/s12083-016-0460-5" target="_blank" rel="noopener noreferrer">https://link.springer.com/article/10.1007/s12083-016-0460-5</a></li>
<li>[15] W. Hao, J. Zeng, X. Dai, J. Xiao, Q. Hua, H. Chen, K.-C. Li, and H. Jin, “Blockp2p: Enabling fast blockchain broadcast with scalable peer-to-peer network topology,” in Green, Pervasive, and Cloud Computing: 14th International Conference, GPC 2019, Uberlandia, Brazil, May 26–28, 2019, Proceedings 14. Springer, 2019, pp. 223–237. Retrieved from <a href="https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16" target="_blank" rel="noopener noreferrer">https://link.springer.com/chapter/10.1007/978-3-030-19223-5_16</a></li>
<li>[16] H. Qiu, T. Ji, S. Zhao, X. Chen, J. Qi, H. Cui, and S. Wang, “A geography-based p2p overlay network for fast and robust blockchain systems,” IEEE Transactions on Services Computing, 2022. Retrieved from <a href="https://ieeexplore.ieee.org/abstract/document/9826458" target="_blank" rel="noopener noreferrer">https://ieeexplore.ieee.org/abstract/document/9826458</a></li>
</ul>]]></content>
<author>
<name>Umar Farooq</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Nescience - A zkVM leveraging hiding properties]]></title>
<id>https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties</id>
<link href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties"/>
<updated>2023-08-28T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Nescience, a privacy-first blockchain zkVM.]]></summary>
<content type="html"><![CDATA[<p>Nescience, a privacy-first blockchain zkVM.</p>
<!-- -->
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="introduction">Introduction<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#introduction" class="hash-link" aria-label="Direct link to Introduction" title="Direct link to Introduction">​</a></h2>
<p>Nescience is a privacy-first blockchain project that aims to enable private transactions and provide a general-purpose execution environment for classical applications.
The goals include creating a state separation architecture for public/private computation,
designing a versatile virtual machine based on mainstream instruction sets,
creating proofs for private state updates, implementing a kernel-based architecture for correct execution of private functions,
and implementing core DeFi protocols such as AMMs and staking from a privacy perspective.</p>
<p>It intends to create a user experience that is similar to public blockchains, but with additional privacy features that users can leverage at will.
To achieve this goal, Nescience will implement a versatile virtual machine that can be used to implement existing blockchain applications,
while also enabling the development of privacy-centric protocols such as private staking and private DEXs.</p>
<p>To ensure minimal trust assumptions and prevent information leakage, Nescience proposes a proof system that allows users to create proofs for private state updates,
while the verification of the proofs and the execution of the public functions inside the virtual machine can be delegated to an external incentivised prover.</p>
<p>It also aims to implement a seamless interaction between public and private state, enabling composability between contracts, and private and public functions.
Finally, Nescience intends to implement permissive licensing, which means that the source code will be open-source,
and developers will be able to use and modify the code without any restriction.</p>
<p>Our primary objective is the construction of the Zero-Knowledge Virtual Machine (zkVM). This document serves as a detailed exploration of the multifaceted challenges,
potential solutions, and alternatives that lay ahead. Each step is a testament to our commitment to thoroughness;
we systematically test various possibilities and decisively commit to the one that demonstrates paramount performance and utility.
For instance, as we progress towards achieving Goal 2, we are undertaking a rigorous benchmarking of the Nova proof system against its contemporaries.
Should Nova showcase superior performance metrics, we stand ready to integrate it as our proof system of choice. Through such meticulous approaches,
we not only reinforce the foundation of our project but also ensure its scalability and robustness in the ever-evolving landscape of blockchain technology.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-1-create-a-state-separation-architecture">Goal 1: Create a State Separation Architecture<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-1-create-a-state-separation-architecture" class="hash-link" aria-label="Direct link to Goal 1: Create a State Separation Architecture" title="Direct link to Goal 1: Create a State Separation Architecture">​</a></h2>
<p>The initial goal revolves around crafting a distinctive architecture that segregates public and private computations,
employing an account-based framework for the public state and a UTXO-based structure for the private state.</p>
<p>The UTXO model [<a href="https://bitcoin.org/bitcoin.pdf" target="_blank" rel="noopener noreferrer">1</a>,<a href="https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/" target="_blank" rel="noopener noreferrer">2</a>], notably utilized in Bitcoin, generates new UTXOs to serve future transactions,
while the account-based paradigm assigns balances to accounts that transactions can modify.
Although the UTXO model bolsters privacy by concealing comprehensive balances,
the pursuit of a dual architecture mandates a meticulous synchronization of these state models,
ensuring that private transactions remain inconspicuous in the wider public network state.</p>
<p>This task is further complicated by the divergent transaction processing methods intrinsic to each model,
necessitating a thoughtful and innovative approach to harmonize their functionality.
To seamlessly bring together the dual architecture, harmonizing the account-based model for public state with the UTXO-based model for private state,
a comprehensive strategy is essential.</p>
<p>The concept of blending an account-based structure with a UTXO-based model for differentiating between public and private states is intriguing.
It seeks to leverage the strengths of both models: the simplicity and directness of the account-based model with the privacy enhancements of the UTXO model.</p>
<p>Here's a breakdown and a potential strategy for harmonizing these models:</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-rationale-behind-the-dual-architecture-"><ins> Rationale Behind the Dual Architecture: </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-rationale-behind-the-dual-architecture-" class="hash-link" aria-label="Direct link to -rationale-behind-the-dual-architecture-" title="Direct link to -rationale-behind-the-dual-architecture-">​</a></h3>
<ul>
<li>
<p><strong>Account-Based Model:</strong> This model is intuitive and easy to work with. Every participant has an account,
and transactions directly modify the balances of these accounts. It's conducive for smart contracts and a broad range of applications.</p>
</li>
<li>
<p><strong>UTXO-Based Model:</strong> This model treats every transaction as a new output, which can then be used as an input for future transactions.
By not explicitly associating transaction outputs with user identities, it offers a degree of privacy.</p>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-harmonizing-the-two-systems-"><ins> Harmonizing the Two Systems: </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-harmonizing-the-two-systems-" class="hash-link" aria-label="Direct link to -harmonizing-the-two-systems-" title="Direct link to -harmonizing-the-two-systems-">​</a></h3>
<ol>
<li>
<p>Translation Layer</p>
<ul>
<li>
<p>Role: Interface between UTXO and account-based states.</p>
</li>
<li>
<p><em>UTXO-to-Account Adapter:</em> When UTXOs are spent, the adapter can translate these into the corresponding account balance modifications.
This could involve creating a temporary 'pseudo-account' that mirrors the
UTXO's attributes.</p>
</li>
<li>
<p><em>Account-to-UTXO Adapter:</em> When an account wishes to make a private transaction,
it would initiate a process converting a part of its balance to a UTXO, facilitating a privacy transaction.</p>
</li>
</ul>
</li>
<li>
<p>Unified Identity Management</p>
<ul>
<li>
<p>Role: Maintain a unified identity (or address) system that works across both state models,
allowing users to easily manage their public and private states without requiring separate identities.</p>
</li>
<li>
<p><em>Deterministic Wallets:</em> Use Hierarchical Deterministic (HD) wallets [<a href="https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14" target="_blank" rel="noopener noreferrer">3</a>,<a href="https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki" target="_blank" rel="noopener noreferrer">4</a>], enabling users to generate multiple addresses (both UTXO and account-based) from a single seed.
This ensures privacy while keeping management centralized for the user.</p>
</li>
</ul>
</li>
<li>
<p>State Commitments</p>
<ul>
<li>
<p>Role: Use cryptographic commitments to commit to the state of both models. This can help in efficiently validating cross-model transactions.</p>
</li>
<li>
<p><em>Verkle Trees:</em> Verkle Trees combine Vector Commitment and the KZG polynomial commitment scheme to produce a structure that's efficient in terms of both proofs and verification.
Verkle proofs are considerably small in size (less data to store and transmit), where Transaction and state verifications can be faster due to the smaller proof sizes and computational efficiencies.</p>
</li>
<li>
<p><em>Mimblewimble-style Aggregation</em> [<a href="https://github.com/mimblewimble/grin/blob/master/doc/intro.md" target="_blank" rel="noopener noreferrer">5</a>]: For UTXOs, techniques similar to those used in Mimblewimble can be used to aggregate transactions, keeping the state compact and enhancing privacy.</p>
</li>
</ul>
</li>
<li>
<p>Batch Processing &amp; Anonymity Sets</p>
<ul>
<li>
<p>Role: Group several UTXO-based private transactions into a single public account-based transaction.
This can provide a level of obfuscation and can make synchronization between the two models more efficient.</p>
</li>
<li>
<p><em>CoinJoin Technique</em> [<a href="https://en.bitcoin.it/wiki/CoinJoin" target="_blank" rel="noopener noreferrer">6</a>]: As seen in Bitcoin, multiple users can combine their UTXO transactions into one, enhancing privacy.</p>
</li>
<li>
<p><em>Tornado Cash Principle</em> [<a href="https://github.com/tornadocash/tornado-classic-ui" target="_blank" rel="noopener noreferrer">7</a>]: For account-based systems wanting to achieve privacy, methods like those used in Tornado Cash can be implemented,
providing zk-SNARKs-based private transactions.</p>
</li>
</ul>
</li>
<li>
<p>Event Hooks &amp; Smart Contracts</p>
<ul>
<li>
<p>Role: Implement event-driven mechanisms that trigger specific actions in one model based on events in the other.
For instance, a private transaction (UTXO-based) can trigger a corresponding public notification or event in the account-based model.</p>
</li>
<li>
<p><em>Conditional Execution:</em> Smart contracts could be set to execute based on events in the UTXO system. For instance,
a smart contract might release funds (account-based) once a specific UTXO is spent.</p>
</li>
<li>
<p><em>Privacy Smart Contracts:</em> Using zk-SNARKs or zk-STARKs to bring privacy to the smart contract layer,
allowing for private logic execution.</p>
</li>
</ul>
</li>
</ol>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-challenges-and-solutions-"><ins> Challenges and Solutions </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-challenges-and-solutions-" class="hash-link" aria-label="Direct link to -challenges-and-solutions-" title="Direct link to -challenges-and-solutions-">​</a></h3>
<ol>
<li>
<p>Synchronization Overhead</p>
<ul>
<li>
<p>Challenge: Combining two distinct transaction models creates an inherent synchronization challenge.</p>
</li>
<li>
<p>State Channels: By allowing transactions to be conducted off-chain between participants, state channels can alleviate synchronization stresses.
Only the final state needs to be settled on-chain, drastically reducing the amount of data and frequency of updates required.</p>
</li>
<li>
<p>Sidechains: These act as auxiliary chains to the main blockchain. Transactions can be processed on the sidechain and then periodically synced with the main chain.
This structure helps reduce the immediate load on the primary system.</p>
</li>
<li>
<p>Checkpointing: Introduce periodic checkpoints where the two systems' states are verified and harmonized.
This can ensure consistency without constant synchronization.</p>
</li>
</ul>
</li>
<li>
<p>Double Spending</p>
<ul>
<li>
<p>Challenge: With two models operating in tandem, there's an increased risk of double-spending attacks.</p>
</li>
<li>
<p>Multi-Signature Transactions: Implementing transactions that require signatures from both systems can prevent unauthorized movements.</p>
</li>
<li>
<p>Cross-Verification Mechanisms: Before finalizing a transaction, it undergoes verification in both UTXO and account-based systems.
If discrepancies arise, the transaction can be halted.</p>
</li>
<li>
<p>Timestamping: By attaching a timestamp to each transaction, it's possible to order them sequentially, making it easier to spot and prevent double spending.</p>
</li>
</ul>
</li>
<li>
<p>Complexity in User Experience</p>
<ul>
<li>
<p>Challenge: The dual model, while powerful, is inherently complex.</p>
</li>
<li>
<p>Abstracted User Interfaces: Design UIs that handle the complexity behind the scenes,
allowing users to make transactions without needing to understand the nuances of the dual model.</p>
</li>
<li>
<p>Guided Tutorials: Offer onboarding tutorials to acquaint users with the system's features,
especially emphasizing when and why they might choose one transaction type over the other.</p>
</li>
<li>
<p>Feedback Systems: Implement systems where users can provide feedback on any complexities or challenges they encounter.
This real-time feedback can be invaluable for iterative design improvements.</p>
</li>
</ul>
</li>
<li>
<p>Security</p>
<ul>
<li>
<p>Challenge: Merging two systems can introduce unforeseen vulnerabilities.</p>
</li>
<li>
<p>Threat Modeling: Regularly conduct threat modeling exercises to anticipate potential attack vectors,
especially those that might exploit the interaction between the two systems.</p>
</li>
<li>
<p>Layered Security Protocols: Beyond regular audits, introduce multiple layers of security checks.
Each layer can act as a fail-safe if a potential threat bypasses another.</p>
</li>
<li>
<p>Decentralized Watchtowers: These are third-party services that monitor the network for malicious activities.
If any suspicious activity is detected, they can take corrective measures or raise alerts.</p>
</li>
</ul>
</li>
<li>
<p>Gas &amp; Fee Management:</p>
<ul>
<li>
<p>Challenge: A dual model can lead to convoluted fee structures.</p>
</li>
<li>
<p>Dynamic Fee Adjustment: Implement algorithms that adjust fees based on network congestion and transaction type.
This can ensure fairness and prevent network abuse.</p>
</li>
<li>
<p>Fee Estimation Tools: Provide tools that can estimate fees before a transaction is initiated.
This helps users understand potential costs upfront.</p>
</li>
<li>
<p>Unified Gas Stations: Design platforms where users can purchase or allocate gas for both transaction types simultaneously,
simplifying the gas acquisition process.</p>
</li>
</ul>
</li>
</ol>
<p>By addressing these challenges head-on with a detailed and systematic approach, it's possible to unlock the full potential of a dual-architecture system,
combining the strengths of both UTXO and account-based models without their standalone limitations.</p>
<table><thead><tr><th>Aspect</th><th>Details</th></tr></thead><tbody><tr><td><strong>Harmony</strong></td><td>- <strong>Advanced VM Development:</strong> Design tailored for private smart contracts. - <strong>Leverage Established Architectures:</strong> Use WASM or RISC-V to harness their versatile and encompassing nature suitable for zero-knowledge applications. - <strong>Support for UTXO &amp; Account-Based Models:</strong> Enhance adaptability across various blockchain structures.</td></tr><tr><td><strong>Challenges</strong></td><td>- <strong>Adaptation Concerns:</strong> WASM and RISC-V weren't designed with zero-knowledge proofs as a primary focus, posing integration challenges. - <strong>Complexities with Newer Systems:</strong> Systems like (Super)Nova, STARKs, and Sangria are relatively nascent, adding another layer of intricacy to the integration. - <strong>Optimization Concerns:</strong> Ensuring that these systems are optimized for zero-knowledge proofs.</td></tr><tr><td><strong>Proposed Solutions</strong></td><td>- <strong>Integration of Nova:</strong> Consider Nova's proof system for its potential alignment with project goals. - <strong>Comprehensive Testing:</strong> Rigorously test and benchmark against alternatives like Halo2, Plonky, and Starky to validate choices. - <strong>Poseidon Recursion Technique:</strong> To conduct exhaustive performance tests, providing insights into each system's efficiency and scalability.</td></tr></tbody></table>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-2-virtual-machine-creation">Goal 2: Virtual Machine Creation<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-2-virtual-machine-creation" class="hash-link" aria-label="Direct link to Goal 2: Virtual Machine Creation" title="Direct link to Goal 2: Virtual Machine Creation">​</a></h2>
<p>The second goal entails the creation of an advanced virtual machine by leveraging established mainstream instruction sets like WASM or RISC-V.
Alternatively, the objective involves pioneering a new, specialized instruction set meticulously optimized for Zero-Knowledge applications.</p>
<p>This initiative seeks to foster a versatile and efficient environment for executing computations within the privacy-focused context of the project.
Both WASM and RISC-V exhibit adaptability to both UTXO and account-based models due to their encompassing nature as general-purpose instruction set architectures.</p>
<p><em>WASM</em>, operating as a low-level virtual machine, possesses the capacity to execute code derived from a myriad of high-level programming languages,
and boasts seamless integration across diverse blockchain platforms.</p>
<p>Meanwhile, <em>RISC-V</em> emerges as a versatile option, accommodating both models, and can be seamlessly integrated with secure enclaves like SGX or TEE,
elevating the levels of security and privacy. However, it is crucial to acknowledge that employing WASM or RISC-V might present challenges,
given their original design without specific emphasis on optimizing for Zero-Knowledge Proofs (ZKPs).</p>
<p>Further complexity arises with the consideration of more potent proof systems like (Super)Nova, STARKs, and Sangria, which,
while potentially addressing optimization concerns, necessitate extensive research and testing due to their relatively nascent status within the field.
This accentuates the need for a judicious balance between established options and innovative solutions in pursuit of an architecture harmoniously amalgamating privacy, security, and performance.</p>
<p>The ambition to build a powerful virtual machine tailored to zero-knowledge (ZK) applications is both commendable and intricate.
The combination of two renowned instruction sets, WASM and RISC-V, in tandem with ZK, is an innovation that could redefine privacy standards in blockchain.
Let's dissect the challenges and possibilities inherent in this goal:</p>
<ol>
<li>
<p>Established Mainstream Instruction Sets - WASM and RISC-V</p>
<ul>
<li>
<p>Strengths:</p>
<ul>
<li>
<p><em>WASM</em>: Rooted in its ability to execute diverse high-level language codes, its potential for cross-chain compatibility makes it a formidable contender.
Serving as a low-level virtual machine, its role in the blockchain realm is analogous to that of the Java Virtual Machine in the traditional computing landscape.</p>
</li>
<li>
<p><em>RISC-V</em>: This open-standard instruction set architecture has made waves due to its customizable nature.
Its adaptability to both UTXO and account-based structures coupled with its compatibility with trusted execution environments like SGX and TEE augments its appeal,
especially in domains that prioritize security and privacy.</p>
</li>
</ul>
</li>
<li>
<p>Challenges: Neither WASM nor RISC-V was primarily designed with ZKPs in mind. While they offer flexibility,
they might lack the necessary optimizations for ZK-centric tasks. Adjustments to these architectures might demand intensive R&amp;D efforts.</p>
</li>
</ul>
</li>
<li>
<p>Pioneering a New, Specialized Instruction Set</p>
<ul>
<li>
<p>Strengths: A bespoke instruction set can be meticulously designed from the ground up with ZK in focus,
potentially offering unmatched performance and optimizations tailored to the project's requirements.</p>
</li>
<li>
<p>Challenges: Crafting a new instruction set is a monumental task requiring vast resources, including expertise, time, and capital.
It would also need to garner community trust and support over time.</p>
</li>
</ul>
</li>
<li>
<p>Contemporary Proof Systems - (Super)Nova, STARKs, Sangria</p>
<ul>
<li>
<p>Strengths: These cutting-edge systems, being relatively new, might offer breakthrough cryptographic efficiencies that older systems lack: designed with modern challenges in mind,
they could potentially bridge the gap where WASM and RISC-V might falter in terms of ZKP optimization.</p>
</li>
<li>
<p>Challenges: Their nascent nature implies a dearth of exhaustive testing, peer reviews, and potentially limited community support.
The unknowns associated with these systems could introduce unforeseen vulnerabilities or complexities.
While they could offer optimizations that address challenges presented by WASM and RISC-V, their young status demands rigorous vetting and testing.</p>
</li>
</ul>
</li>
</ol>
<center><table><thead><tr><th style="text-align:center"></th><th style="text-align:center">Mainstream (WASM, RISC-V)</th><th style="text-align:center">ZK-optimized (New Instruction Set)</th></tr></thead><tbody><tr><td style="text-align:center">Existing Tooling</td><td style="text-align:center">YES</td><td style="text-align:center">NO</td></tr><tr><td style="text-align:center">Blockchain-focused</td><td style="text-align:center">NO</td><td style="text-align:center">YES</td></tr><tr><td style="text-align:center">Performant</td><td style="text-align:center">DEPENDS</td><td style="text-align:center">YES</td></tr></tbody></table></center>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-optimization-concerns-for-wasm-and-risc-v-"><ins> Optimization Concerns for WASM and RISC-V: </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-optimization-concerns-for-wasm-and-risc-v-" class="hash-link" aria-label="Direct link to -optimization-concerns-for-wasm-and-risc-v-" title="Direct link to -optimization-concerns-for-wasm-and-risc-v-">​</a></h3>
<ul>
<li>
<p><em>Cryptography Libraries</em>: ZKP applications rely heavily on specific cryptographic primitives. Neither WASM nor RISC-V natively supports all of these primitives.
Thus, a comprehensive library of cryptographic functions, optimized for these platforms, needs to be developed.</p>
</li>
<li>
<p><em>Parallel Execution</em>: Given the heavy computational demands of ZKPs, leveraging parallel processing capabilities can optimize the time taken.
Both WASM and RISC-V would need modifications to handle parallel execution of ZKP processes efficiently.</p>
</li>
<li>
<p><em>Memory Management</em>: ZKP computations can sometimes require significant amounts of memory, especially during the proof generation phase.
Fine-tuned memory management mechanisms are essential to prevent bottlenecks.</p>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-emerging-zkp-optimized-systems-considerations-"><ins> Emerging ZKP Optimized Systems Considerations: </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-emerging-zkp-optimized-systems-considerations-" class="hash-link" aria-label="Direct link to -emerging-zkp-optimized-systems-considerations-" title="Direct link to -emerging-zkp-optimized-systems-considerations-">​</a></h3>
<ul>
<li>
<p><em>Proof Size</em>: Different systems generate proofs of varying sizes. A smaller proof size is preferable for blockchain applications to save on storage and bandwidth.
The trade-offs between proof size, computational efficiency, and security need to be balanced.</p>
</li>
<li>
<p><em>Universality</em>: Some systems can support any computational statement (universal), while others might be tailored to specific tasks.
A universal system can be more versatile for diverse applications on the blockchain.</p>
</li>
<li>
<p><em>Setup Requirements</em>: Certain ZKP systems, like zk-SNARKs, require a trusted setup, which can be a security concern.
Alternatives like zk-STARKs don't have this requirement but come with other trade-offs.</p>
</li>
</ul>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-strategies-for-integration-"><ins> Strategies for Integration: </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-strategies-for-integration-" class="hash-link" aria-label="Direct link to -strategies-for-integration-" title="Direct link to -strategies-for-integration-">​</a></h3>
<ul>
<li>
<p><em>Iterative Development</em>: Given the complexities, an iterative development approach can be beneficial.
Start with a basic integration of WASM or RISC-V for general tasks and gradually introduce specialized ZKP functionalities.</p>
</li>
<li>
<p><em>Benchmarking</em>: Establish benchmark tests specifically for ZKP operations. This will provide continuous feedback on the performance of the system as modifications are made, ensuring optimization.</p>
</li>
<li>
<p><em>External Audits &amp; Research</em>: Regular checks from cryptographic experts and collaboration with academic researchers can help in staying updated and ensuring secure implementations.</p>
</li>
</ul>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-3-proofs-creation-and-verification">Goal 3: Proofs Creation and Verification<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-3-proofs-creation-and-verification" class="hash-link" aria-label="Direct link to Goal 3: Proofs Creation and Verification" title="Direct link to Goal 3: Proofs Creation and Verification">​</a></h2>
<p>The process of generating proofs for private state updates is vested in the hands of the user, aligning with our commitment to minimizing trust assumptions and enhancing privacy.
Concurrently, the responsibility of verifying these proofs and executing public functions within the virtual machine can be effectively delegated to an external prover,
a role that is incentivized to operate with utmost honesty and integrity. This intricate balance seeks to safeguard against information leakage,
preserving the confidentiality of private transactions. Integral to this mechanism is the establishment of a robust incentivization framework.</p>
<p>To ensure the prover’s steadfast commitment to performing tasks with honesty, we should introduce a mechanism that facilitates both rewards for sincere behavior and penalties for any deviation from the expected standards.
This two-pronged approach serves as a compelling deterrent against dishonest behavior and fosters an environment of accountability.
In addition to incentivization, a crucial consideration is the economic aspect of verification and execution.
The verification process has been intentionally designed to be more cost-effective than execution.</p>
<p>This strategic approach prevents potential malicious actors from exploiting the system by flooding it with spurious proofs, a scenario that could arise when the costs align favorably.
By maintaining a cost balance that favors verification, we bolster the system’s resilience against fraudulent activities while ensuring its efficiency.
In sum, our multifaceted approach endeavors to strike an intricate equilibrium between user-initiated proof creation, external verification, and incentivization.
This delicate interplay of mechanisms ensures a level of trustworthiness that hinges on transparency, accountability, and economic viability.</p>
<p>As a result, we are poised to cultivate an ecosystem where users’ privacy is preserved, incentives are aligned,
and the overall integrity of the system is fortified against potential adversarial actions. To achieve the goals of user-initiated proof creation,
external verification, incentivization, and cost-effective verification over execution, several options and mechanisms can be employed:</p>
<ol>
<li>
<p><strong>User-Initiated Proof Creation:</strong> Users are entrusted with the generation of proofs for private state updates, thus ensuring greater privacy and reducing trust dependencies.</p>
<ul>
<li>
<p>Challenges:</p>
<ul>
<li>
<p>Maintaining the quality and integrity of the proofs generated by users.</p>
</li>
<li>
<p>Ensuring that users have the tools and knowledge to produce valid proofs.</p>
</li>
</ul>
</li>
<li>
<p>Solutions:</p>
<ul>
<li>
<p>Offer extensive documentation, tutorials, and user-friendly tools to streamline the proof-generation process.</p>
</li>
<li>
<p>Implement checks at the verifier's end to ensure the quality of proofs.</p>
</li>
</ul>
</li>
</ul>
</li>
<li>
<p><strong>External Verification by Provers:</strong> An external prover verifies the proofs and executes public functions within the virtual machine.</p>
<ul>
<li>
<p>Challenges:</p>
<ul>
<li>
<p>Ensuring that the external prover acts honestly.</p>
</li>
<li>
<p>Avoiding centralized points of failure.</p>
</li>
</ul>
</li>
<li>
<p>Solutions:</p>
<ul>
<li>
<p>Adopt a decentralized verification approach, with multiple provers cross-verifying each other’s work.</p>
</li>
<li>
<p>Use reputation systems to rank provers based on their past performances, creating a trust hierarchy.</p>
</li>
</ul>
</li>
</ul>
</li>
<li>
<p>** Incentivization Framework:** A system that rewards honesty and penalizes dishonest actions, ensuring provers' commitment to the task.</p>
<ul>
<li>
<p>Challenges:</p>
<ul>
<li>
<p>Determining the right balance of rewards and penalties.</p>
</li>
<li>
<p>Ensuring that the system cannot be gamed for undue advantage.</p>
</li>
</ul>
</li>
<li>
<p>Solutions<sup><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#user-content-fn-1-c438e0" id="user-content-fnref-1-c438e0" data-footnote-ref="true" aria-describedby="footnote-label">1</a></sup>:</p>
<ul>
<li>
<p>Implement a dynamic reward system that adjusts based on network metrics and provers' performance.</p>
</li>
<li>
<p>Use a staking mechanism where provers need to lock up a certain amount of assets.
Honest behavior earns rewards, while dishonest behavior could lead to loss of staked assets.</p>
</li>
</ul>
</li>
</ul>
</li>
<li>
<p><strong>Economic Viability through Cost Dynamics:</strong> Making verification more cost-effective than execution to deter spamming and malicious attacks.</p>
<ul>
<li>
<p>Challenges:</p>
<ul>
<li>
<p>Setting the right cost metrics for both verification and execution.</p>
</li>
<li>
<p>Ensuring that genuine users aren’t priced out of the system.</p>
</li>
</ul>
</li>
<li>
<p>Solutions:</p>
<ul>
<li>
<p>Use a dynamic pricing model, adjusting costs in real-time based on network demand.</p>
</li>
<li>
<p>Implement gas-like mechanisms to differentiate operation costs and ensure fairness.</p>
</li>
</ul>
</li>
</ul>
</li>
<li>
<p>** Maintaining Trustworthiness:** Create a system that's transparent, holds all actors accountable, and is economically sound.</p>
<ul>
<li>
<p>Challenges:</p>
<ul>
<li>
<p>Keeping the balance where users feel their privacy is intact, while provers feel incentivized.</p>
</li>
<li>
<p>Ensuring the system remains resilient against adversarial attacks.</p>
</li>
</ul>
</li>
<li>
<p>Solutions:</p>
<ul>
<li>
<p>Implement layered checks and balances.</p>
</li>
<li>
<p>Foster community involvement, allowing them to participate in decision-making, potentially through a decentralized autonomous organization (DAO).</p>
</li>
</ul>
</li>
</ul>
</li>
</ol>
<p>Each of these options can be combined or customized to suit the specific requirements of your project, striking a balance between user incentives,
cost dynamics, and verification integrity. A thoughtful combination of these mechanisms ensures that the system remains robust, resilient,
and conducive to the objectives of user-initiated proof creation, incentivized verification, and cost- effective validation.</p>
<center><table><thead><tr><th>Aspect</th><th>Details</th></tr></thead><tbody><tr><td><strong>Design Principle</strong></td><td>- <strong>User Responsibility:</strong> Generating proofs for private state updates. - <strong>External Prover:</strong> Delegated the task of verifying proofs and executing public VM functions.</td></tr><tr><td><strong>Trust &amp; Privacy</strong></td><td>- <strong>Minimized Trust Assumptions:</strong> Place proof generation in users' hands. - <strong>Enhanced Privacy:</strong> Ensure confidentiality of private transactions and prevent information leakage.</td></tr><tr><td><strong>Incentivization Framework</strong></td><td>- <strong>Rewards:</strong> Compensate honest behavior. - <strong>Penalties:</strong> Deter and penalize dishonest behavior.</td></tr><tr><td><strong>Economic Considerations</strong></td><td>- <strong>Verification vs. Execution:</strong> Make verification more cost-effective than execution to prevent spurious proofs flooding. - <strong>Cost Balance:</strong> Strengthen resilience against fraudulent activities and maintain efficiency.</td></tr><tr><td><strong>Outcome</strong></td><td>An ecosystem where: - Users' privacy is paramount. - Incentives are appropriately aligned. - The system is robust against adversarial actions.</td></tr></tbody></table></center>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-4-kernel-based-architecture-implementation">Goal 4: Kernel-based Architecture Implementation<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-4-kernel-based-architecture-implementation" class="hash-link" aria-label="Direct link to Goal 4: Kernel-based Architecture Implementation" title="Direct link to Goal 4: Kernel-based Architecture Implementation">​</a></h2>
<p>This goal centers on the establishment of a kernel-based architecture, akin to the model observed in ZEXE, to facilitate the attestation of accurate private function executions.
This innovative approach employs recursion to construct a call stack, which is then validated through iterative recursive computations.
At its core, this technique harnesses a recursive Succinct Non-Interactive Argument of Knowledge (SNARK) mechanism, where each function call’s proof accumulates within the call stack.</p>
<p>The subsequent verification of this stack’s authenticity leverages recursive SNARK validation.
While this method offers robust verification of private function executions, it’s essential to acknowledge its associated intricacies.</p>
<p>The generation of SNARK proofs necessitates a substantial computational effort, which, in turn, may lead to elevated gas fees for users.
Moreover, the iterative recursive computations could potentially exhibit computational expansion as the depth of recursion increases.
This calls for a meticulous balance between the benefits of recursive verification and the resource implications it may entail.</p>
<p>In essence, Goal 4 embodies a pursuit of enhanced verification accuracy through a kernel-based architecture.
By weaving recursion and iterative recursive computations into the fabric of our system, we aim to establish a mechanism that accentuates the trustworthiness of private function executions,
while conscientiously navigating the computational demands that ensue.</p>
<p>To accomplish the goal of implementing a kernel-based architecture for recursive verification of private function executions,
several strategic steps and considerations can be undertaken: recursion handling and depth management.</p>
<ins> Recursion Handling </ins>
<ul>
<li>
<p><em>Call Stack Management:</em></p>
<ul>
<li>Implement a data structure to manage the call stack, recording each recursive function call’s details, parameters, and state.</li>
</ul>
</li>
<li>
<p>_Proof Accumulation: _</p>
<ul>
<li>
<p>Design a mechanism to accumulate proof data for each function call within the call stack.
This includes cryptographic commitments, intermediate results, and cryptographic challenges.</p>
</li>
<li>
<p>Ensure that the accumulated proof data remains secure and tamper-resistant throughout the recursion process.</p>
</li>
</ul>
</li>
<li>
<p><em>Intermediary SNARK Proofs:</em></p>
<ul>
<li>
<p>Develop an intermediary SNARK proof for each function call’s correctness within the call stack.
This proof should demonstrate that the function executed correctly and produced expected outputs.</p>
</li>
<li>
<p>Ensure that the intermediary SNARK proof for each recursive call can be aggregated and verified together, maintaining the integrity of the entire call stack.</p>
</li>
</ul>
</li>
</ul>
<ins> Depth management </ins>
<ul>
<li>
<p><em>Depth Limitation:</em></p>
<ul>
<li>
<p>Define a threshold for the maximum allowable recursion depth based on the system’s computational capacity, gas limitations, and performance considerations.</p>
</li>
<li>
<p>Implement a mechanism to prevent further recursion beyond the defined depth limit, safeguarding against excessive computational growth.</p>
</li>
</ul>
</li>
<li>
<p><em>Graceful Degradation:</em></p>
<ul>
<li>
<p>Design a strategy for graceful degradation when the recursion depth approaches or reaches the defined limit.
This may involve transitioning to alternative execution modes or optimization techniques.</p>
</li>
<li>
<p>Communicate the degradation strategy to users and ensure that the system gracefully handles scenarios where recursion must be curtailed.</p>
</li>
</ul>
</li>
<li>
<p><em>Resource Monitoring:</em></p>
<ul>
<li>Develop tools to monitor resource consumption (such as gas usage and computational time) as recursion progresses.
Provide real-time feedback to users about the cost and impact of recursive execution.</li>
</ul>
</li>
<li>
<p><em>Dynamic Depth Adjustment:</em></p>
<ul>
<li>
<p>Consider implementing adaptive depth management that dynamically adjusts the recursion depth based on network conditions, transaction fees, and available resources.</p>
</li>
<li>
<p>Utilize algorithms to assess the optimal recursion depth for efficient execution while adhering to gas cost constraints.</p>
</li>
</ul>
</li>
<li>
<p><em>Fallback Mechanisms:</em></p>
<ul>
<li>Create fallback mechanisms that activate if the recursion depth limit is reached or if the system encounters resource constraints.
These mechanisms could involve alternative verification methods or delayed execution.</li>
</ul>
</li>
<li>
<p><em>User Notifications:</em></p>
<ul>
<li>Notify users when the recursion depth limit is approaching, enabling them to make informed decisions about the complexity of their transactions and potential resource usage.</li>
</ul>
</li>
</ul>
<p>Goal 4 underscores the project's ambition to integrate the merits of a kernel-based architecture with recursive verifications to bolster the reliability of private function executions.
While the approach promises robust outcomes, it's pivotal to maneuver through its intricacies with astute strategies, ensuring computational efficiency and economic viability.
By striking this balance, the architecture can realize its full potential in ensuring trustworthy and efficient private function executions.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-5-seamless-interaction-design">Goal 5: Seamless Interaction Design<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-5-seamless-interaction-design" class="hash-link" aria-label="Direct link to Goal 5: Seamless Interaction Design" title="Direct link to Goal 5: Seamless Interaction Design">​</a></h2>
<p>Goal 5 revolves around the meticulous design of a seamless interaction between public and private states within the blockchain ecosystem.
This objective envisions achieving not only composability between contracts but also the harmonious integration of private and public functions.</p>
<p>A notable challenge in this endeavor lies in the intricate interplay between public and private states,
wherein the potential linkage of a private transaction to a public one raises concerns about unintended information leakage.</p>
<p>The essence of this goal entails crafting an architecture that facilitates the dynamic interaction of different states while ensuring that the privacy and confidentiality of private transactions remain unbreached.
This involves the formulation of mechanisms that enable secure composability between contracts, guaranteeing the integrity of interactions across different layers of functionality.</p>
<p>A key focus of this goal is to surmount the challenge of information leakage by implementing robust safeguards.
The solution involves devising strategies to mitigate the risk of revealing private transaction details when connected to corresponding public actions.
By creating a nuanced framework that com- partmentalizes private and public interactions, the architecture aims to uphold privacy while facilitating seamless interoperability.</p>
<p>Goal 5 encapsulates a multifaceted undertaking, calling for the creation of an intricate yet transparent framework that empowers users to confidently engage in both public and private functions,
without compromising the confidentiality of private transactions. The successful realization of this vision hinges on a delicate blend of architectural ingenuity, cryptographic sophistication, and user-centric design.</p>
<p>To achieve seamless interaction between public and private states, composability, and privacy preservation, a combination of solutions and approaches can be employed.
In the table below, a comprehensive list of solutions that address these objectives:</p>
<center><table><thead><tr><th style="text-align:center"><strong>Solution Category</strong></th><th style="text-align:center"><strong>Description</strong></th></tr></thead><tbody><tr><td style="text-align:center"><strong>Layer 2 Solutions</strong></td><td style="text-align:center">Employ zk-Rollups, Optimistic Rollups, and state channels to handle private interactions off-chain and settle them on-chain periodically. Boost scalability and cut transaction costs.</td></tr><tr><td style="text-align:center"><strong>Intermediary Smart Contracts</strong></td><td style="text-align:center">Craft smart contracts as intermediaries for secure public-private interactions. Use these to manage data exchange confidentially.</td></tr><tr><td style="text-align:center"><strong>Decentralized Identity &amp; Pseudonymity</strong></td><td style="text-align:center">Implement decentralized identity systems for pseudonymous interactions. Validate identity using cryptographic proofs.</td></tr><tr><td style="text-align:center"><strong>Confidential Sidechains &amp; Cross-Chain</strong></td><td style="text-align:center">Set up confidential sidechains and employ cross-chain protocols to ensure private and composability across blockchains.</td></tr><tr><td style="text-align:center"><strong>Temporal Data Structures</strong></td><td style="text-align:center">Create chronological data structures for secure interactions. Utilize cryptographic methods for data integrity and privacy.</td></tr><tr><td style="text-align:center"><strong>Homomorphic Encryption &amp; MPC</strong></td><td style="text-align:center">Apply homomorphic encryption and MPC for computations on encrypted data and interactions between state layers.</td></tr><tr><td style="text-align:center"><strong>Commit-Reveal Schemes</strong></td><td style="text-align:center">Introduce commit-reveal mechanisms for private transactions, revealing data only post necessary public actions.</td></tr><tr><td style="text-align:center"><strong>Auditability &amp; Verifiability</strong></td><td style="text-align:center">Use on-chain tools for auditing and verifying interactions. Utilize cryptographic commitments for third-party validation.</td></tr><tr><td style="text-align:center"><strong>Data Fragmentation &amp; Sharding</strong></td><td style="text-align:center">Fragment data across shards for private interactions and curtailed data exposure. Bridge shards securely with cryptography.</td></tr><tr><td style="text-align:center"><strong>Ring Signatures &amp; CoinJoin</strong></td><td style="text-align:center">Incorporate ring signatures and CoinJoin protocols to mask transaction details and mix transactions collaboratively.</td></tr></tbody></table></center>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="goal-6-integration-of-defi-protocols-with-a-privacy-preserving-framework">Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#goal-6-integration-of-defi-protocols-with-a-privacy-preserving-framework" class="hash-link" aria-label="Direct link to Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework" title="Direct link to Goal 6: Integration of DeFi Protocols with a Privacy-Preserving Framework">​</a></h2>
<p>The primary aim of Goal 6 is to weave key DeFi protocols, such as AMMs and staking, into a user-centric environment that accentuates privacy.
This endeavor comes with inherent challenges, especially considering the heterogeneity of existing DeFi protocols, predominantly built on Ethereum.
These variations in programming languages and VMs exacerbate the quest for interoperability. Furthermore, the success and functionality of DeFi protocols is closely tied to liquidity,
which in turn is influenced by user engagement and the amount of funds locked into the system.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="-strategic-roadmap-for-goal-6-"><ins> Strategic Roadmap for Goal 6 </ins><a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#-strategic-roadmap-for-goal-6-" class="hash-link" aria-label="Direct link to -strategic-roadmap-for-goal-6-" title="Direct link to -strategic-roadmap-for-goal-6-">​</a></h2>
<ol>
<li>
<p><em>** Pioneering Privacy-Centric DeFi Models: **</em> Initiate the development of AMMs and staking solutions that are inherently protective of users' transactional privacy and identity.</p>
</li>
<li>
<p><em>** Specialized Smart Contracts with Privacy: **</em> Architect distinct smart contracts infused with privacy elements, setting the stage for secure user interactions within this new, confidential DeFi landscape.</p>
</li>
<li>
<p><em>** Optimized User Interfaces: **</em> Craft interfaces that resonate with user needs, simplifying the journey through the private DeFi space without compromising on security.</p>
</li>
<li>
<p><em>** Tackling Interoperability: **</em></p>
<ul>
<li>
<p>Deploy advanced bridge technologies and middleware tools to foster efficient data exchanges and guarantee operational harmony across a spectrum of programming paradigms and virtual environments.</p>
</li>
<li>
<p>Design and enforce universal communication guidelines that bridge the privacy-centric DeFi entities with the larger DeFi world seamlessly.</p>
</li>
</ul>
</li>
<li>
<p><em>** Enhancing and Sustaining Liquidity: **</em></p>
<ul>
<li>
<p>Unveil innovative liquidity stimuli and yield farming incentives, compelling users to infuse liquidity into the private DeFi space.</p>
</li>
<li>
<p>Incorporate adaptive liquidity frameworks that continually adjust based on the evolving market demands, ensuring consistent liquidity.</p>
</li>
<li>
<p>Forge robust alliances with other DeFi stalwarts, jointly maximizing liquidity stores and honing sustainable token distribution strategies.</p>
</li>
</ul>
</li>
<li>
<p><em>** Amplifying Community Engagement:**</em> Design and roll out enticing incentive schemes to rally users behind privacy-focused AMMs and staking systems,
thereby nurturing a vibrant, privacy-advocating DeFi community.</p>
</li>
</ol>
<p>Through the integration of these approaches, we aim to achieve Goal 6, providing users with a privacy-focused platform for engaging effortlessly in core DeFi functions such as AMMs and staking,
all while effectively overcoming the obstacles related to interoperability and liquidity concerns.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="summary-of-the-architecture">Summary of the Architecture<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#summary-of-the-architecture" class="hash-link" aria-label="Direct link to Summary of the Architecture" title="Direct link to Summary of the Architecture">​</a></h2>
<p>In our quest to optimize privacy, we're proposing a Zero-Knowledge Virtual Machine (Zkvm) that harnesses the power of Zero-Knowledge Proofs (ZKPs).
These proofs ensure that while private state data remains undisclosed, public state transitions can still be carried out and subsequently verified by third parties.
This blend of public and private state is envisaged to be achieved through a state tree representing the public state, while the encrypted state leaves stand for the private state.
Each user's private state indicates validity through the absence of a corresponding nullifier.
A nullifier is a unique cryptographic value generated in privacy-preserving blockchain transactions to prevent double-spending,
ensuring that each private transaction is spent only once without revealing its details.</p>
<p>Private functions' execution mandates users to offer a proof underscoring the accurate execution of all encapsulated private calls.
For validating a singular private function call, we're leaning into the kernel-based model inspired by the ZEXE protocol.
Defined as kernel circuits, these functions validate the correct execution of each private function call.
Due to their recursive circuit structure, a succession of private function calls can be executed by calculating proofs in an iterative manner.
Execution-relevant data, like private and public call stacks and additions to the state tree, are incorporated as public inputs.</p>
<p>Our method integrates the verification keys for these functions within a merkle tree. Here's the innovation: a user's ZKP showcases the existence of the verification key in this tree, yet keeps the executed function concealed.
The unique function identifier can be presented as the verification key, with all contracts merkleized for hiding functionalities.</p>
<p>We suggest a nuanced shift from the ZEXE protocol's identity function, which crafts an identity for smart contracts delineating its behavior, access timeframes, and other functionalities.
Instead of the ZEXE protocol's structure, our approach pivots to a method anchored in the
security of a secret combined with the uniqueness from hashing with the contract address.
The underlying rationale is straightforward: the sender, equipped with a unique nonce and salt for the transaction, hashes the secret, payload, nonce, and salt.
This result is then hashed with the contract address for the final value. The hash function's unidirectional nature ensures that the input cannot be deduced easily from its output.
A specific concern, however, is the potential repetition of secret and payload values across transactions, which could jeopardize privacy.
Yet, by embedding the function's hash within the hash of the contract address, users can validate a specific function's execution without divulging the function, navigating this limitation.</p>
<p>Alternative routes do exist: We could employ signature schemes like ECDSA, focusing on uniqueness and authenticity, albeit at the cost of complex key management.
Fully Homomorphic Encryption (FHE) offers another pathway, enabling function execution on encrypted data, or Multi-Party Computation (MPC) which guarantees non-disclosure of function or inputs.
Yet, integrating ZKPs with either FHE or MPC presents a challenge. Combining cryptographic functions like SHA-3 and BLAKE2 can also bolster security and uniqueness.
It's imperative to entertain these alternatives, especially when hashing might not serve large input/output functions effectively or might fall short in guaranteeing uniqueness.</p>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="current-state">Current State<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#current-state" class="hash-link" aria-label="Direct link to Current State" title="Direct link to Current State">​</a></h2>
<p>Our aim is to revolutionize the privacy and security paradigms through Nescience.
As we strive to set milestones and achieve groundbreaking advancements,
our current focus narrows onto the realization of Goal 2 and Goal 3.</p>
<p>Our endeavors to build a powerful virtual machine tailored for Zero-Knowledge applications have led us down the path of rigorous exploration and testing.
We believe that integrating the right proof system is pivotal to our project's success, which brings us to Nova [<a href="https://eprint.iacr.org/2021/370" target="_blank" rel="noopener noreferrer">8</a>].
In our project journey, we have opted to integrate the Nova proof system, recognizing its potential alignment with our overarching goals.
However, as part of our meticulous approach to innovation and optimization, we acknowledge the need to thoroughly examine Nova’s performance capabilities,
particularly due to its status as a pioneering and relatively unexplored proof system.</p>
<p>This critical evaluation entails a comprehensive process of benchmarking and comparative analysis <a href="https://github.com/vacp2p/zk-explorations" target="_blank" rel="noopener noreferrer">[9]</a>,
pitting Nova against other prominent proof systems in the field, including Halo2 [<a href="https://electriccoin.co/blog/explaining-halo-2/" target="_blank" rel="noopener noreferrer">10</a>],
Plonky2 [<a href="https://polygon.technology/blog/introducing-plonky2" target="_blank" rel="noopener noreferrer">11</a>], and Starky [<a href="https://eprint.iacr.org/2021/582" target="_blank" rel="noopener noreferrer">12</a>].
This ongoing and methodical initiative is designed to ensure a fair and impartial assessment, enabling us to draw meaningful conclusions about Nova’s strengths and limitations in relation to its counterparts.
By leveraging the Poseidon recursion technique, we are poised to conduct an exhaustive performance test that delves into intricate details.
Through this testing framework, we aim to discern whether Nova possesses the potential to outshine its contemporaries in terms of efficiency, scalability, and overall performance.
The outcome of this rigorous evaluation will be pivotal in shaping our strategic decisions moving forward.
Armed with a comprehensive understanding of Nova’s performance metrics vis-à-vis other proof systems,
we can confidently chart a course that maximizes the benefits of our project’s optimization efforts.</p>
<p>Moreover, as we ambitiously pursue the establishment of a robust mechanism for proof creation and verification, our focus remains resolute on preserving user privacy,
incentivizing honest behaviour, and ensuring the cost-effective verification of transactions.
At the heart of this endeavor is our drive to empower users by allowing them the autonomy of generating proofs for private state updates,
thereby reducing dependencies and enhancing privacy.
We would like to actively work on providing comprehensive documentation, user-friendly tools,
and tutorials to aid users in this intricate process.</p>
<p>Parallelly, we're looking into decentralized verification processes, harnessing the strength of multiple external provers that cross-verify each other's work.
Our commitment is further cemented by our efforts to introduce a dynamic reward system that adjusts based on network metrics and prover performance.
This intricate balance, while challenging, aims to fortify our system against potential adversarial actions, aligning incentives, and preserving the overall integrity of the project.</p>
<h1>References</h1>
<p>[1] Nakamoto, S. (2008). Bitcoin: A Peer-to-Peer Electronic Cash System. Retrieved from <a href="https://bitcoin.org/bitcoin.pdf" target="_blank" rel="noopener noreferrer">https://bitcoin.org/bitcoin.pdf</a></p>
<p>[2] Sanchez, F. (2021). Cardano’s Extended UTXO accounting model. Retrived from <a href="https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/" target="_blank" rel="noopener noreferrer">https://iohk.io/en/blog/posts/2021/03/11/cardanos-extended-utxo-accounting-model/</a></p>
<p>[3] Morgan, D. (2020). HD Wallets Explained: From High Level to Nuts and Bolts. Retrieved from <a href="https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14" target="_blank" rel="noopener noreferrer">https://medium.com/mycrypto/the-journey-from-mnemonic-phrase-to-address-6c5e86e11e14</a></p>
<p>[4] Wuille, P. (012). Bitcoin Improvement Proposal (BIP) 44. Retrieved from <a href="https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki" target="_blank" rel="noopener noreferrer">https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki</a></p>
<p>[5] Jedusor, T. (2020). Introduction to Mimblewimble and Grin. Retrieved from <a href="https://github.com/mimblewimble/grin/blob/master/doc/intro.md" target="_blank" rel="noopener noreferrer">https://github.com/mimblewimble/grin/blob/master/doc/intro.md</a></p>
<p>[6] Bitcoin's official wiki overview of the CoinJoin method. Retrieved from <a href="https://en.bitcoin.it/wiki/CoinJoin" target="_blank" rel="noopener noreferrer">https://en.bitcoin.it/wiki/CoinJoin</a></p>
<p>[7] TornadoCash official Github page. Retrieved from <a href="https://github.com/tornadocash/tornado-classic-ui" target="_blank" rel="noopener noreferrer">https://github.com/tornadocash/tornado-classic-ui</a></p>
<p>[8] Kothapalli, A., Setty, S., Tzialla, I. (2021). Nova: Recursive Zero-Knowledge Arguments from Folding Schemes. Retrieved from <a href="https://eprint.iacr.org/2021/370" target="_blank" rel="noopener noreferrer">https://eprint.iacr.org/2021/370</a></p>
<p>[9] ZKvm Github page. Retrieved from <a href="https://github.com/vacp2p/zk-explorations" target="_blank" rel="noopener noreferrer">https://github.com/vacp2p/zk-explorations</a></p>
<p>[10] Electric Coin Company (2020). Explaining Halo 2. Retrieved from <a href="https://electriccoin.co/blog/explaining-halo-2/" target="_blank" rel="noopener noreferrer">https://electriccoin.co/blog/explaining-halo-2/</a></p>
<p>[11] Polygon Labs (2022). Introducing Plonky2. Retrieved from <a href="https://polygon.technology/blog/introducing-plonky2" target="_blank" rel="noopener noreferrer">https://polygon.technology/blog/introducing-plonky2</a></p>
<p>[12] StarkWare (2021). ethSTARK Documentation. Retrieved from <a href="https://eprint.iacr.org/2021/582" target="_blank" rel="noopener noreferrer">https://eprint.iacr.org/2021/582</a></p>
<!-- -->
<section data-footnotes="true" class="footnotes"><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5 sr-only" id="footnote-label">Footnotes<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#footnote-label" class="hash-link" aria-label="Direct link to Footnotes" title="Direct link to Footnotes">​</a></h2>
<ol>
<li id="user-content-fn-1-c438e0">
<p>Incentive Mechanisms:</p>
<ul>
<li>
<p>Token Rewards: Design a token-based reward system where honest provers are compensated with tokens for their verification services.
This incentivizes participation and encourages integrity.</p>
</li>
<li>
<p>Staking and Slashing: Introduce a staking mechanism where provers deposit tokens as collateral.
Dishonest behavior results in slashing (partial or complete loss) of the staked tokens, while honest actions are rewarded.</p>
</li>
<li>
<p>Proof of Work/Proof of Stake: Implement a proof-of-work or proof-of- stake consensus mechanism for verification,
aligning incentives with the blockchain’s broader consensus mechanism.</p>
</li>
</ul>
<a href="https://vac.dev/rlog/Nescience-A-zkVM-leveraging-hiding-properties#user-content-fnref-1-c438e0" data-footnote-backref="" aria-label="Back to reference 1" class="data-footnote-backref">↩</a>
</li>
</ol>
</section>]]></content>
<author>
<name>Moudy</name>
</author>
</entry>
<entry>
<title type="html"><![CDATA[Device Pairing in Js-waku and Go-waku]]></title>
<id>https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku</id>
<link href="https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku"/>
<updated>2023-04-24T12:00:00.000Z</updated>
<summary type="html"><![CDATA[Device pairing and secure message exchange using Waku and noise protocol.]]></summary>
<content type="html"><![CDATA[<p>Device pairing and secure message exchange using Waku and noise protocol.</p>
<!-- -->
<p>As the world becomes increasingly connected through the internet, the need for secure and reliable communication becomes paramount. In <a href="https://vac.dev/wakuv2-noise" target="_blank" rel="noopener noreferrer">this article</a> it is described how the Noise protocol can be used as a key-exchange mechanism for Waku.</p>
<p>Recently, this feature was introduced in <a href="https://github.com/waku-org/js-noise" target="_blank" rel="noopener noreferrer">js-waku</a> and <a href="https://github.com/waku-org/go-waku" target="_blank" rel="noopener noreferrer">go-waku</a>, providing a simple API for developers to implement secure communication protocols using the Noise Protocol framework. These open-source libraries provide a solid foundation for building secure and decentralized applications that prioritize data privacy and security.</p>
<p>This functionality is designed to be simple and easy to use, even for developers who are not experts in cryptography. The library offers a clear and concise API that abstracts away the complexity of the Noise Protocol framework and provides an straightforward interface for developers to use. Using this, developers can effortlessly implement secure communication protocols on top of their JavaScript and Go applications, without having to worry about the low-level details of cryptography.</p>
<p>One of the key benefits of using Noise is that it provides end-to-end encryption, which means that the communication between two parties is encrypted from start to finish. This is essential for ensuring the security and privacy of sensitive information</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="device-pairing">Device Pairing<a href="https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku#device-pairing" class="hash-link" aria-label="Direct link to Device Pairing" title="Direct link to Device Pairing">​</a></h3>
<p>In today's digital world, device pairing has become an integral part of our lives. Whether it's connecting our smartphones with other computers or web applications, the need for secure device pairing has become more crucial than ever. With the increasing threat of cyber-attacks and data breaches, it's essential to implement secure protocols for device pairing to ensure data privacy and prevent unauthorized access.</p>
<p>To demonstrate how device pairing can be achieved using Waku and Noise, we have examples available at <a href="https://examples.waku.org/noise-js/" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js/</a>. You can try pairing different devices, such as mobile and desktop, via a web application. This can be done by scanning a QR code or opening a URL that contains the necessary data for a secure handshake.</p>
<p>The process works as follows:</p>
<p>Actors:</p>
<ul>
<li>Alice the initiator</li>
<li>Bob the responder</li>
</ul>
<ol>
<li>The first step in achieving secure device pairing using Noise and Waku is for Bob generate the pairing information which could be transmitted out-of-band. For this, Bob opens <a href="https://examples.waku.org/noise-js/" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js/</a> and a QR code is generated, containing the data required to do the handshake. This pairing QR code is timeboxed, meaning that after 2 minutes, it will become invalid and a new QR code must be generated</li>
<li>Alice scans the QR code using a mobile phone. This will open the app with the QR code parameters initiating the handshake process which is described in <a href="https://github.com/waku-org/specs/blob/master/standards/application/device-pairing.md/#protocol-flow" target="_blank" rel="noopener noreferrer">WAKU2-DEVICE-PAIRING</a>. These messages are exchanged between two devices over Waku to establish a secure connection. The handshake messages consist of three main parts: the initiator's message, the responder's message, and the final message, which are exchanged to establish a secure connection. While using js-noise, the developer is abstracted of this process, since the messaging happens automatically depending on the actions performed by the actors in the pairing process.</li>
<li>Both Alice and Bob will be asked to verify each other's identity. This is done by confirming if an 8-digits authorization code match in both devices. If both actors confirm that the authorization code is valid, the handshake concludes succesfully</li>
<li>Alice and Bob receive a set of shared keys that can be used to start exchanging encrypted messages. The shared secret keys generated during the handshake process are used to encrypt and decrypt messages sent between the devices. This ensures that the messages exchanged between the devices are secure and cannot be intercepted or modified by an attacker.</li>
</ol>
<p>The above example demonstrates device pairing using js-waku. Additionally, You can also try building and experimenting with other noise implementations like nwaku, or go-waku, with an example available at <a href="https://github.com/waku-org/go-waku/tree/master/examples/noise" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/go-waku/tree/master/examples/noise</a> in which the same flow described before is done with Bob (the receiver) using go-waku instead of js-waku.</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="conclusion">Conclusion<a href="https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku#conclusion" class="hash-link" aria-label="Direct link to Conclusion" title="Direct link to Conclusion">​</a></h3>
<p>With its easy to use API built on top of the Noise Protocol framework and the LibP2P networking stack, if you are a developer looking to implement secure messaging in their applications that are both decentralized and censorship resistant, Waku is definitely an excellent choice worth checking out!</p>
<p>Waku is also Open source with a MIT and APACHEv2 licenses, which means that developers are encouraged to contribute code, report bugs, and suggest improvements to make it even better.</p>
<p>Don't hesitate to try the live example at <a href="https://examples.waku.org/noise-js" target="_blank" rel="noopener noreferrer">https://examples.waku.org/noise-js</a> and build your own webapp using <a href="https://github.com/waku-org/js-noise" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/js-noise</a>, <a href="https://github.com/waku-org/js-waku" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/js-waku</a> and <a href="https://github.com/waku-org/go-waku" target="_blank" rel="noopener noreferrer">https://github.com/waku-org/go-waku</a>. This will give you a hands-on experience of implementing secure communication protocols using the Noise Protocol framework in a practical setting. Happy coding!</p>
<h3 class="anchor anchorWithHideOnScrollNavbar_WYt5" id="references">References<a href="https://vac.dev/rlog/device-pairing-in-js-waku-and-go-waku#references" class="hash-link" aria-label="Direct link to References" title="Direct link to References">​</a></h3>
<ul>
<li><a href="https://vac.dev/wakuv2-noise" target="_blank" rel="noopener noreferrer">Noise handshakes as key-exchange mechanism for Waku</a></li>
<li><a href="https://github.com/waku-org/specs/blob/master/standards/application/noise.md" target="_blank" rel="noopener noreferrer">Noise Protocols for Waku Payload Encryption</a></li>
<li><a href="https://github.com/waku-org/specs/blob/master/standards/application/noise-sessions.md" target="_blank" rel="noopener noreferrer">Session Management for Waku Noise</a></li>
<li><a href="https://github.com/waku-org/specs/blob/master/standards/application/device-pairing.md" target="_blank" rel="noopener noreferrer">Device pairing and secure transfers with Noise</a></li>
<li><a href="https://github.com/waku-org/go-waku/tree/master/examples/noise" target="_blank" rel="noopener noreferrer">go-waku Noise's example</a></li>
<li><a href="https://github.com/waku-org/js-waku-examples/tree/master/examples/noise-js" target="_blank" rel="noopener noreferrer">js-waku Noise's example</a></li>
<li><a href="https://github.com/waku-org/js-noise/" target="_blank" rel="noopener noreferrer">js-noise</a></li>
<li><a href="https://github.com/waku-org/js-noise/" target="_blank" rel="noopener noreferrer">go-noise</a></li>
</ul>]]></content>
<author>
<name>Richard</name>
</author>
</entry>
</feed>
Reference in New Issue View Git Blame Copy Permalink