Introduction​

Secure Group Messaging (SGM) is resource-intensive when aiming for robust security features like forward secrecy (FS) and post-compromise security (PCS).

One straightforward approach to SGM is a pairwise group chat, where each pair of group members establishes a unique encryption key using Diffie-Hellman. While this method ensures security, it falls short in terms of practicality:

• High storage requirements: Each participant must store encryption keys for every other participant.
• Inefficient encryption: Each message must be encrypted separately for every participant, leading to significant computational overhead.
• Inefficient message storage and delivery: Each separately encrypted message must then be sent over the wire, whatever this wire might be. Or stored in database.
• Cumbersome group management: Adding or removing users and refreshing keys becomes increasingly inefficient as the group grows.

One scalable for Secure Group Messaging (SGM) is Message Layer Security (MLS), as standardized in RFC 9420. Leveraging TreeKEM, MLS organizes group members in a cryptographic tree structure, where each participant is responsible for maintaining specific parts of the tree.

While MLS offers scalability and strong security guarantees, its reliance on server-based delivery services poses limitations for fully decentralized environments.

In this post, we present the implementation details of the first version of Decentralized MLS (de-MLS) which is an SGM protocol. De-MLS can serve groups that cannot rely on central servers, such as journalists and activists seeking secure communication. It is also well suited for DAOs, where Ethereum-based authentication can restrict access to members holding a minimum ETH balance, and for NGOs or research consortia that prefer not to host their own servers while still requiring end-to-end encrypted group messaging. Decentralized MLS (de-MLS) satisfies the following features:

• Decentralized
• Scalable
• End-to-end encrypted (E2EE)
• FS and PCS provided
• Ethereum authenticated

Background​

MLS​

The Message Layer Security (MLS) protocol offers scalable and secure group messaging protocol by organizing participants into a cryptographic tree structure, enabling efficient operations like adding or removing members with logarithmic time complexity relative to the group size. MLS provides strong security guarantees, including FS and PCS.

MLS assumes that two services are provided:

• An Authentication Service (AS): It enables group members to authenticate the credentials presented by other group members.
• A Delivery Service (DS) that routes MLS messages among the participants in the protocol in the correct order and manage the keyPackage of the users where the keyPackage is the objects that provide some public information about a user.

Despite its scalability, MLS has a notable limitation: it is inherently designed for server-based federated architectures for delivery service (DS), even when the servers themselves don't need to be trusted. To achieve a decentralized protocol, the functionality of DS must be reimagined to eliminate reliance on a central server while preserving the protocol's security properties. Thus, we proposed decentralized MLS (de-MLS), leveraging Waku nodes as peer-to-peer communication protocols to eliminate reliance on centralized servers.

Lastly, MLS operates on an epoch-based model, where group state changes (e.g., adding/removing users or key refreshes) occur between epochs that are always required to be conducted by a single entity. For example, if a user is removed in epoch E, the rest of the group members generate a new key in epoch E + 1 by passing the new entropy. The removed user cannot decrypt messages sent after epoch E + 1.

Waku​

Waku is a decentralized messaging protocol designed for secure and efficient communication in peer-to-peer networks. It operates as a broadcast-based routing layer where content topics can be used to tag and filter messages. Users join channels by subscribing to specific content topics, which determine the scope and type of messages exchanged. This enables flexible and efficient communication patterns in a decentralized environment.

de-MLS​

Decentralized MLS (de-MLS) is a peer-to-peer secure group messaging protocol that can work with any delivery service (DS) meeting a minimal set of requirements. In this post, we highlight its integration with Waku as the messaging protocol, while emphasizing that de-MLS itself remains agnostic to the underlying DS. Further technical details can be found in the de-MLS RFC.

Decentralization is achieved not only at the delivery service (DS) level but also within the authentication service (AS). Multiple special nodes named Steward in the group serve as authorized identities to authenticate users before they join or are removed from the group transparently.

de-MLS provides two different user management configurations, both utilizing the Waku protocol for DS:

1. Single Steward:
   • A single authorized identity (Steward) manages the group, including removing or adding users with agreement among users by a voting-based consensus.
2. Multi-Steward:
   • Multiple Stewards have equal authority to add or remove users.
   • A consensus mechanism ensures consistency by resolving concurrent changes within the same epoch and preventing possible conflicts. In each epoch, all modifications are managed exclusively by a single Steward.

Note: We chose the term Steward to reflect the role of transparently coordinating and organizing passengers at stations, much like Stewards do in transit systems.

In multi-Steward settings, de-MLS requires a consensus among Stewards that have equal rights in the group since changes in an epoch in MLS are required to be conducted by a single identity, that is the Steward.

For the consensus integration, ongoing research explores two promising approaches:

1. On-chain consensus mechanisms: Outsourcing consensus to a smart contract solution for transparent and immutable agreement.
2. Off-chain consensus mechanisms: Utilizing off-chain consensus protocols to design efficient, decentralized protocols.

Waku Integration​

Waku integration is a crucial step in the construction of de-MLS, aiming to replace traditional client-server communication with decentralized messaging. The specifics of Waku integration will be detailed in a separate RFC; for now, our main priority is the de-MLS RFC.

The main challenge in this transition is transforming the centralized Delivery Service (DS) into a decentralized equivalent, which performs two essential functions:

1. Message Delivery and Ordering: The DS is responsible not only for delivering messages to the correct recipients, but also for preserving the correct order of these messages, which is critical for the consistency of group state.
2. Key Package Management: The DS manages key packages, which are essential for adding members securely to a group.

To maintain a truly decentralized architecture, key packages cannot be stored in a centralized location. Initially, we considered using a smart contract (SC) as a decentralized substitute for server-side key package storage. However, this approach proved impractical. Blockchains are immutable by design—once data is written, it cannot be fully removed. This contradicts a core requirement of MLS: each key package must be used exactly once and then deleted, to prevent replay or reuse attacks. Instead, our solution is to require users to actively provide their key packages upon request, allowing validation at the moment of use without persistent storage. While this approach may lose some benefits of asynchronicity, we plan to address this in the future by introducing store nodes that can temporarily hold key packages. This ensures both compliance with MLS's security model and alignment with decentralized system principles.

Flow​

The flow section explains the processes that when a user wants to join a group in both Steward and users side also their interactions. The flow of de-MLS is as follows:

1. Steward joins the welcome topic​

The welcome topic is a topic created and monitored by the Steward for a specific secure messaging group, allowing any Waku node to subscribe permissionlessly. Being in the welcome topic does not imply group membership, it acts as a waiting room where users can send their key material, which the Steward listens for and processes before granting access to the secure group.

2. Group initialization​

Steward initalizes a group with parameters such as cipher suite and group ID.

3. Emitting Group Anouncement (GA) by Steward​

Steward creates group announcement (GA) periodically to the welcome channel that the users can find the who the Steward is. This will be important for the next step.

4. User joins the welcome topic​

As first, the users who wants to be part of the decentralized MLS should subscribe the welcome channel. Then user can find the group name and also corresponding GA message from Steward. This GA message helps the user to create a valid keyPackages which define in section 10 in RFC9420 for the group.

5. User creates its key package​

User creates the keyPackage and encrypt by public key of the Steward then send it to the Steward. Since the message is encrypted, stay secure though the welcome (permissionless) topic.

6. Steward receives the User's key package​

Steward receives the user's keyPackage and decrypt it. After decrypted, Steward also verifies the validity of the keyPackage by signature verification. If the keyPackage is not valid, the Steward just drops the message, otherwise it moves to the next step which is proposal creation.

7. Creation of Voting proposals​

Voting proposals are special MLS application messages that may come from any participant, including the Steward. In this context, any member can create a proposal corresponding to the user’s keyPackage. In regular MLS, proposals are automatically converted into commit messages, which can change the structure of the tree. However, in de-MLS, since the process is decentralized, proposals must be voted on before being converted into a commitment.

8. Voting for proposal​

Voting applies decentralization by protecting small groups can control. Therefore, proposals must be voted on before committing. The consensus mechanism should be a lightweight consensus that cannot be a bottleneck for treeKEM scalability. Basically, the consensus returns the binary result for a given proposal. If voting result is NO, the proposal is dropped; otherwise, the Steward transforms it into an MLS proposal. MLS proposal message is a distinct type of MLS application message, where the Steward attaches the voting result instead of directly releasing a commit message.

9. Creating commit message​

Commit messages are the messages that start new epochs. They include key and tree material that existing members can use to generate the new state of the tree.

After Steward gets the YES from consensus, Steward creates commit messages that injects new entropy for the existing group members.

10. Sending messages​

After Steward creates and then sends two messages:

1. Commit message informs existing group member to update their key to align with the new member’s key for the upcoming epoch.
2. The welcome message informs the newly joined user to generate a group key that matches the key existing members will use in the upcoming epoch.

Although existing users had different group keys in the previous epoch and the new user had none, the Steward message ensures that both existing and new users converge on the same group key in the next epoch.

11. Applying welcome message​

User can generate the next epoch group key by using the welcome message as well as existing users extract the same groupKey by using commit messages.

The commit message helps existing members generate the next group key Gk+1, while the welcome message helps the newly joining user generate the same Gk+1. This provides two important security properties:

1. Forward Secrecy (FS): The new user cannot read previous messages since they were encrypted with the old key Gk

2. Post-Compromise Security (PCS): If a user is removed from the group, they cannot read future messages since those messages will be encrypted with the new key Gk+1

Benchmark​

This section presents the performance evaluation of de-MLS. One of the key advantages of the MLS protocol is its efficiency, as it eliminates the need for pairwise message exchanges between all participants. Instead, the decentralized DS enables the addition of new participants by sending only two messages to the group: a commit message and a welcome message. However, despite this advantage, the protocol does have certain bottlenecks, which are as follows:

• Firstly, the Steward must receive the key packages from each member wishing to join the group. This process requires sequential message exchanges and involves computationally intensive tasks such as encryption, decryption, and digital signature verification. Even when multiple users are added to the group simultaneously, the process is essentially sequential. The tree structure is updated one user at a time, followed by sending the final commit message to the existing group members and a single welcome message to the new members.
• Secondly adding a member to a group requires rebuilding the tree and computing new keys.

The following measurements were made as follows:

1. The time required for the entire sequence of receiving a user key package is presented here. This includes generating the Steward key, creating messages with signatures and encryption, and processing these messages.

Share Key Package - 1.8395 ms

Note that these measurements do not account for the time taken to forward messages.

1. The time required for creating the commit and welcome message from a ready-made package bunches is shown in this table.

The tests were conducted on the following configuration: Apple M3 Pro @ 4.05GHz and 12-Core CPU/18-Core GPU.

Here, the network latency and the time taken by users to apply the received commits are also excluded. These aspects are planned to be measured and evaluated in future work.

Potential drawbacks and countermeasures​

Since de-MLS replace the servers by P2P, we could lose some good features of servers based MLS. In this section we present the potential drawbacks and possible countermeasures of de-MLS.

• Offline users: keyPackages are provided by the users directly without any storing, this is required each user must be online for joining to a group.
  • We can consider to use Waku sync nodes that are nodes has storing ability for a temporary storing of keyPackages.
• DoS attack to Steward: Steward is known in welcome message from periodic group announcement message so Steward can be targeted for DoS attack.
  • As always we consider to use Rate-Limiting Nullifier (RLN) with Waku to protect network from spam.
• Message loss or delay : Because of P2P and consensus settings, message can be lost or delayed.,
  • We can integrate reliability mechanisms to Waku such as scalable data sync (SDS)
  • Consensus mechanism requires to provide liveness property against offline nodes, for example, it may provides default YES or NO options for a silent users who do not vote.
• Enchanced authentication
  • Ethereum authentication could be inefficient. We can configure the authentication mechanism for example asking minimum balance or etc.

Conclusion​

To summarize, the approach to solving decentralized DS tasks with Waku can be outlined as shown in the comparison table:

Future Work​

In the next iterations, the implementations are planned as following:

• Dual-Consensus Multi-Steward Support: One consensus mechanism selects an Steward from all users, while a second governs group decisions among the elected Stewards
• Consensus mechanism for handling concurrent changes within the same epoch
• Key rotation support
• Benchmarking for the multi-Steward configuration including the network time

References​

• [1] RFC 9420: The Messaging Layer Security (MLS) Protocol. Retrieved from https://datatracker.ietf.org/doc/rfc9420/
• [2] OpenMLS. Retrived from https://github.com/openmls/openmls
• [3] Waku. Retrived from https://waku.org/
• [4] de-MLS. Retrived from https://github.com/vacp2p/de-mls/

Overview​

For each subscribed topic, GossipSub nodes maintain a full-message mesh of peers with a target degree $D$, bounded by lower and upper thresholds $D_{low}$ and $D_{high}$​, respectively. In parallel, a metadata-only (gossip) mesh includes additional $K$ peers for metadata exchange. All messages flow through the full-message mesh, and metadata flows through the gossip mesh.

The metadata contains IHAVE messages, announcing IDs of seen messages. Upon receiving an IHAVE announcement about an unseen message ID, a receiver responds with an IWANT request to retrieve the announced message. IHAVE announcements serve multiple purposes:

1. Offer additional resilience in situations involving non-conforming peers or network partitions.
2. Speed up message propagation by allowing far-off peers to fetch overdue messages.

Since GossipSub v1.1 [1], replying to IWANT requests is optional, to safeguard honest peers from adversaries. This change encourages peers to make redundant IWANT requests for the same message. While this arrangement works well for small messages, it can be inefficient for bigger ones. This inefficiency arises from an increase in duplicates and a higher number of IWANT requests due to longer transmission times for large messages. As a result, we observe a significant rise in bandwidth utilization and message dissemination times across the network.

IDONTWANT messages in GossipSub v1.2 [2] help reduce some duplicates. However, further efforts are necessary to mitigate this problem [3, 4]. That is why many recent works focus on improving GossipSub's performance when handling large messages [5, 6, 7, 8, 9, 10, 11, 12].

In this post, we evaluate the push-pull phase transition (PPPT) mechanism [10], the GossipSub v1.4 proposal [11, 5], and the GossipSub v2.0 proposal [12] for performance against Gossipsub v1.2. For this purpose, we implement minimal proof-of-concept (PoC) versions of these protocols in nim-libp2p [13] and use the shadow simulator to provide performance evaluation results.

We begin by discussing the issues of message dissemination times and duplicate messages in GossipSub. Next, we provide a brief overview of the proposals under review, followed by the experiments and findings from our performance evaluations.

Message Transfer Time and Duplicates​

Assuming uniform link characteristics, message dissemination to full-message mesh concludes in $\tau_D \approx (D \times \tau_{tx}) + \tau_p$ time, where $\tau_{tx} = \frac{S}{R}$, with $S$, $R$, and $\tau_p$ being the message size, data rate, and link latency, respectively.

This simplifies network-wide dissemination time to $\tau_{N} \approx \tau_D \times h$, with $h$ indicating the number of hops along the longest path. This implies that a tenfold increase in message size results in an eightyfold rise in $D \times \tau_{tx}$ for a mesh with $D = 8$, which accumulates across $h$ hops. This leads to two fundamental problems:

1. A longer contention interval ($\tau_D$) increases the chance that peers receive the same message from multiple mesh members during that interval,
   leading to redundant transmissions and more duplicates. Reducing $D$ inadvertently increases $h$ — potentially slowing propagation overall.

2. Peers are unaware of ongoing message receptions and may generate redundant IWANT requests for the messages they are already receiving. Most of these requests are entertained by early message receivers, which increases message dissemination time by increasing the workload at peers situated along the optimal path.

Talking about duplicates, a network comprising $N$ peers, each with a degree $D$, has a total of $\frac {N \times D}{2}$ edges (links), as every link connects two peers. Assuming that a message traverses every link exactly once, we get at least $\frac {N \times D}{2}$ transmissions.

Only $N-1$ transmissions are necessary for delivering a message to all peers. As a result, we get $\frac {N \times D}{2} -(N-1)$ duplicates in the network. We can simplify average duplicates received by a single peer to $\bar{d}_{min} \approx \frac{D}{2}-1$. Here, $\bar{d}_{min}$ represents the lower bound on average duplicates because we assume that the send and receive operations are mutually exclusive. This assumption requires that message transmission times (and link latencies) are so small that no two peers simultaneously transmit the same message to each other.

However, a large message can noticeably increase the contention interval, which increases the likelihood that many peers will simultaneously transmit the same message to each other.

Authors in [14] explore the upper bound ($\bar{d}_{max}$) on duplicates. They argue that a node can forward a received message to a maximum of $D-1$ peers while the original publisher sends it to $D$ peers. As a result, we get $N(D-1)+1$ transmissions in the network. Only $N-1$ transmissions are necessary to deliver a message to all peers. Remaining $N(D-1)+1-(N-1)$ transmissions are duplicates, which simplifies the upper bound on average duplicates received by each peer to $\bar{d}_{max} \approx D-2$. This rise indicates that larger messages can lead to more duplicates due to longer contention intervals. It is essential to highlight that the impact of IWANT/IDONTWANT messages is not considered in $\bar{d}$ computations above.

Protocols Considered​

Push-Pull Phase Transition (PPPT)​

In PPPT, authors argue that most redundant transmissions occur during the later stages of message propagation. As a message traverses through the network, the peers forwarding the message should gradually reduce the number of mesh members that directly receive the message (push) and send immediate IHAVE announcements to the remaining peers in the mesh. The remaining mesh members can fetch any missing messages using IWANT requests (Pull). The authors also suggest two strategies to estimate the message propagation stage:

1. Include a hop count in the message header to identify the number of hops traversed by that message. When a peer forwards a received message, it performs a pull operation for a subset of mesh members that equals the specified hop count and a push operation for the remaining mesh members.

2. Infer the message propagation stage by looking into the number of received IHAVE announcements and duplicates for that message. Use this information to choose a balance between pull-based and push-based message forwarding.

The authors suggest that instead of simultaneously pushing a message to the selected peers, sequentially initiating transmission to each peer after a short delay enhances the likelihood of timely receiving a higher number of IDONTWANT requests.

Key Considerations and PoC Implementation​

The use of hop count is a more effective and straightforward method for identifying the message propagation stage than relying on the duplicate count. However, this approach may compromise the publisher's anonymity and reveal information about the publisher and its mesh members. Additional due diligence may be needed to address these issues.

In the PoC implementation [15], we use hop count to determine the message propagation stage. When forwarding a message, every peer selects a subset of mesh members equal to the advertised hop count for pull operation and forwards the message to the remaining mesh members. If the advertised hop count exceeds the number of mesh members chosen for message forwarding, the sender relays the message to all selected peers.

GossipSub v1.4 Proposal​

GossipSub v1.4 proposal considers longer large-message transfer times ($\tau_D$) as contention intervals and argues that most duplicates occur during these intervals for two fundamental reasons:

1. Peers are unaware of ongoing message receptions and may generate redundant IWANT requests for messages they are already receiving.

2. Peers can send IDONTWANT announcements only after receiving the entire message. However, a large contention interval increases the likelihood that many redundant transmissions will start before IDONTWANT messages are issued.

GossipSub v1.4 proposal eliminates contention interval with help from two new control messages: PREAMBLE and IMRECEIVING. A PREAMBLE precedes every large message transmission. Upon receiving a preamble, a peer learns about the messages it is receiving and performs two actions:

1. Notify mesh members about ongoing message reception using an IMRECEIVING announcement. On receiving an IMRECEIVING announcement from a peer, mesh members defer sending the announced message to that peer.

2. Defer IWANT requests for messages that are currently being received. Peers also limit the outstanding IWANT requests for any message to one.

Key Considerations and PoC Implementation​

The use of PREAMBLE/IMRECEIVING addresses the limitation of IDONTWANT messages. For instance, consider a peer $X$ begins receiving a message $m$ at time $t_1$. It can transmit IDONTWANT only after receiving $m$, i.e., at time $t_1+\tau_D$. Therefore, it can not cancel any duplicate receptions of $m$ that start before $t1 + \tau_D + \tau_p$. In contrast, IMRECEIVING announcements for $m$ start at $t_1 + \Delta$, where $\Delta$ denotes PREAMBLE processing time and satisfies $\Delta \ll \tau_D$. As a result, peer $X$ can eliminate all duplicate receptions of $m$ that start after $t_1 + \Delta +\tau_p$, which noticeably reduces duplicates.

The use of PREAMBLE also allows deferring IWANT requests for messages we are already receiving, which can also improve message dissemination time by reducing the workload on peers along the optimal message forwarding path.

It is worth mentioning that a malicious peer can exploit this approach by sending a PREAMBLE and never completing (or deliberately delaying) the promised message transfer. The optional safety strategy in GossipSub v1.4 proposal suggests using a peer score threshold for PREAMBLE processing and a behavior penalty for broken promises. A timeout strategy helps recover such messages.

It is essential to mention that sending and processing of PREAMBLE and IMRECEIVING messages is optional. This flexibility allows for the use of custom safety strategies in various implementations. For example, the ongoing production-grade implementation of GossipSub v1.4 in nim-libp2p allows peers to ignore PREAMBLEs unless they come from mesh members with higher data rates (bandwidth estimation becomes trivial with PREAMBLEs) and good peer scores. This implementation also lets peers choose between a push or pull strategy for handling broken promises.

For the performance evaluations in this post, we utilize the PoC implementation of GossipSub v1.4 [16]. A complete, production-grade version is currently undergoing testing and validation [17].

GossipSub v2.0 Proposal​

GossipSub v2.0 introduces a hybrid method for message dissemination that combines both push and pull strategies through two new control messages: IANNOUNCE and INEED. These messages are analogous to IHAVE and IWANT messages, respectively. However, IANNOUNCE messages are issued to the mesh members immediately after validating a received message without waiting for the heartbeat interval. Similarly, INEED requests are made exclusively to mesh members, and a peer generates only one INEED request for a received message.

The balance between push and pull approaches is determined by the $D_{announce}$ parameter. On receiving a message, a peer forwards it to $D - D_{announce}$ mesh members and sends IANNOUNCE messages to the remaining mesh peers. On receiving an IANNOUNCE for an unseen message, a peer can request it using an INEED message.

Key Considerations and PoC Implementation​

The $D_{announce}$ parameter governs the balance between push and pull operations. Setting $D_{announce} = D$ results in a pull-only operation, which can eliminate duplicates at the cost of increased message dissemination time. In contrast, setting $D_{announce}$ to zero reverts to standard GossipSub v1.2 operation. The authors suggest setting $D_{announce} = D-1$ to moderately decrease dissemination time while incurring only a small number of duplicate transmissions.

It is important to note that malicious peers can exploit this approach by delaying or entirely omitting responses to INEED requests. Similarly, sending INEED requests to suboptimal or overwhelmed peers can further increase message dissemination time. The authors propose using a timeout strategy and negative peer scoring to address these issues. If a message transfer does not complete within the specified interval, the receiver decreases the sender's peer score and issues a new INEED request to an alternative mesh member.

For the performance evaluations in this post, we utilize the PoC implementation of GossipSub v2.0 [18] from nim-libp2p. We set $D_{announce} = D-1$ and allow any peer to send a single IWANT request for a message, only if it has not previously sent an INEED request for the same message.

Experiments​

We conducted a series of experiments under various configurations to evaluate the performance of the GossipSub v1.4 proposal, the PPPT approach, and the GossipSub v2.0 proposal against the baseline GossipSub v1.2 protocol. To support these evaluations, we extended the nim-libp2p implementation to include minimal PoC implementations of the considered protocols [16, 15, 18]
and used the Shadow simulator [19] to carry out performance evaluations.

For GossipSub v1.4 and PPPT, we also report results from delayed forwarding, where peers introduce a short delay before relaying a message to every mesh member. This delay helps reduce the number of redundant transmissions by increasing the likelihood of timely receiving a higher number of IDONTWANT notifications.

We evaluate performance using network-wide message dissemination time (latency), network-wide bandwidth utilization (bandwidth), and the average number of duplicates received by a peer for every transmitted message. We also report the average number of IWANT requests transmitted by a peer for a single message.

For each experiment, we transmit multiple messages in the network. We average the network-wide dissemination time for these messages to report latency. Bandwidth refers to the total volume of traffic in the network, encompassing control messages and data transmissions (including duplicates and IWANT replies). A peer usually receives multiple copies of any transmitted message. Excluding the first received message, all copies are duplicates. We compute average duplicates received by a peer as $\frac{1}{N M} \sum_{j=1}^{M} \sum_{i=1}^{N} d_{i,j}$ where $N$ and $M$ denote the number of peers and the number of transmitted messages, respectively, and $d_{i,j}$ represents the number of duplicates received by peer $i$ for message $j$. A similar mechanism computes average IWANT requests.

Three simulation scenarios are considered:

• Scenario 1: The number of publishers and message size are kept constant while the network size gradually increases.

• Scenario 2: The number of publishers and the network size remain constant while the message size gradually increases.

• Scenario 3: The number of nodes and message size remain constant while the number of publishers gradually increases.

In all experiments, we transmit multiple messages such that every publisher sends exactly one message to the network. After a publisher transmits a message, each subsequent publisher waits for a specified interval (inter-message delay) before sending the next message.

Rotating publishers ensures that every message traverses a different path, which helps achieve fair performance evaluation. On the other hand, changing inter-message delays allows for creating varying traffic patterns. A shorter inter-message delay implies more messages can be in transit simultaneously, which helps evaluate performance against large message counts. A longer delay ensures every message is fully disseminated before introducing a new message. Similarly, increasing message size stresses the network. As a result, we evaluate performance across a broader range of use cases.

The simulation details are presented in the tables below. The experiments are conducted using the shadow simulator. We uniformly set peer bandwidths and link latencies between 40-200 Mbps and 40-130 milliseconds in five variations.

Table 1: Simulation Scenarios.

Table 2: Simulation Parameters.

Results​

Scenario1: Increasing Network Size​

Bandwidth	Latency	Average Duplicates	Average IWANT Requests

Scenario2: Increasing Message Size​

Bandwidth	Latency	Average Duplicates	Average IWANT Requests

Scenario3: Increasing Number of Publishers​

Bandwidth	Latency	Average Duplicates	Average IWANT Requests

Findings​

• The number of IWANT requests increases with the message size. Limiting ongoing IWANT requests for each message to one can be beneficial. Additionally, the use of message PREAMBLEs can help eliminate IWANT requests for messages that are already being received.

• Pull-based approaches can substantially reduce bandwidth utilization, but may result in much longer message dissemination times. However, these approaches can achieve simultaneous propagation of multiple messages by implicitly rotating among outgoing messages. As a result, increasing the number of messages yields similar dissemination times.

• Transition from push to pull operation during the later stages of message propagation can reduce bandwidth consumption, without compromising latency. However, determining the propagation stage is challenging. Methods like hop counts may compromise anonymity, while using IHAVE announcements can be misleading. For instance, in the case of large messages, peers may receive IHAVE announcements much earlier than the actual message spreads through the network.

• Push-based approaches achieve the fastest message dissemination but often produce a higher number of duplicate messages. Employing mechanisms like PREAMBLE/IMRECEIVING messages for guided elimination of duplicate messages can significantly reduce bandwidth consumption. This reduction not only minimizes redundant transmissions but also decreases the overall message dissemination time by lessening the workload on peers located along optimal message forwarding paths.

Please feel free to join the discussion and leave feedback regarding this post in the VAC forum.

References​

[1] GossipSub v1.1 Specifications

[2] GossipSub v1.2 Specifications

[3] Number of Duplicate Messages in Ethereum’s Gossipsub Network

[4] Impact of IDONTWANT in the Number of Duplicates

[5] PREAMBLE and IMRECEIVING for Improved Large Message Handling

[6] Staggering and Fragmentation for Improved Large Message Handling

[7] GossipSub for Big Messages

[8] FullDAS: Towards Massive Scalability with 32MB Blocks and Beyond

[9] Choke Extension for GossipSub

[10] PPPT: Fighting the GossipSub Overhead with Push-Pull Phase Transition

[11] GossipSub v1.4 Specifications Proposal

[12] GossipSub v2.0 Specifications Proposal

[13] Libp2p Implementation in nim

[14] The Streamr Network: Performance and Scalability

[15] PPPT: PoC Implementation in nim-libp2p

[16] GossipSub v1.4: PoC Implementation in nim-libp2p

[17] GossipSub v1.4: Production-Grade Implementation in nim-libp2p

[18] GossipSub v2.0: PoC Implementation in nim-libp2p

[19] nim-libp2p GossipSub Test Node

Background​

Our friends over at Waku are particularly enthusiastic about anonymous spam prevention technologies. They have been using the Rate Limiting Nullifier (RLN) tooling that Zerokit provides to enforce a message-rate policy among users—a crucial feature unless we want a community bombarded with "totally legit, not scams" messages on repeat. However, as is often the case with new technology, some problematic delays began to surface. Node recalculations, a common operation, were taking tens of seconds at the scales being tested and deployed—even exceeding 40 seconds at times. These delays accumulate, leading to total delays on the order of three hours under certain conditions.

Naturally, we couldn't just let this sit. While we've touched on the issue of spam prevention, it's important to recognize that this technology is foundational that challenges conventional wisdom on how things must be done. Does the idea of "smart contracts without gas" catch your attention? Don't hold your breath just yet: the really interesting applications of this tech will be dead in the water, unless we can meet the challenge put to us.

The Challenge​

The plan of attack that the team put together was twofold: get rid of redundant operations and data taking up precious resources, and make the remaining operations go Blazingly Fast™.

Introducing the star of the show for part 1: The main point of having this tree is to generate proofs so that peers can verify the claims being made. That doesn’t require the whole Merkle tree, just a single path, from leaf to root. The engineering work took us in a direction where these paths were the primary context in which ZK proofs operated, relegating the tree itself to an off-network reference. This reduced the burden imposed on the network significantly. Updating the data on the tree has similarly reduced, with the exception being that the siblings of each node were retained. This is called the stateless approach.

Well, stateless in the context of proof generation and verification. This is the critical context when it comes to performance, and the stateless approach does a great job, but these proofs have to come from somewhere. Each participant still needs to maintain the Merkle tree in their local environment. Without this tree, one cannot generate proofs or verify the proofs provided to them. Fortunately, one does not need to track the entire tree, but can be limited to a subset of the tree needed. With millions of participants on-chain, this can make the difference needed to make Zerokit empowered technologies accessible to those running raspberry Pis. Combine this with the fact that the heavy lifting operations of proof gen/verification being modular and separate, each participant can optimise to run things according to the strengths and requirements of their native hardware, easing the way to allow each participants to run their tree implementation at the speed of mach-ludicrous.

Fortunately, the core of our already existing implementation was sane and well put together. Double-lucky for us, the talents of newly minted VAC/ACZ team members Sylvain and Vinh were readily available. Sylvain, with a solid background in the Rust programming language, having already graduated from the most challenging parts of its infamous learning curve. He quickly got to work zeroing in on some subtle performance pathologies. Something as simple as using a mutable iterator to change values directly. Clever use of references to avoid copying data, and other memory optimization techniques that can be hidden to those that cannot “see the matrix” when working in Rust lead to very promising bench-marking results.

Vinh, having recently graduated from his CS studies, was presented with the challenge of parrelising computations. For those not familiar with Rust, this might seem unreasonable, but thanks to the rayon crate, and Rusts promise of "fearless concurrency" afforded by its type and ownership system, this kind of refactor becomes surprisingly easy, even for a talented individual at the start of their career. Of particular note: These parallelisations have been made available to the browser. Browser threads are relatively now, and by diving into this bleeding-edge technology, and making use of libraries that are still in early development stages, Blazingly Fast™ is now available within the browser. With all that in the bag, all these performance gains are gift-wrapped in the use of browser-native WASM runtimes.

Well done, everyone!

The importance of benchmarks​

No performance project is complete without high quality benchmark data. Writing a quick benchmark for tracking improvements through development is one thing, but having a system of telemetry that allows you to credibly assert claims of superior performance is what completes the project. With such credible claims in hand, these efforts can bring about significant impact on the field at large. The key word being credible. Credibility cannot depend on “trust me bro” (obviously). The truth of these claims must come out of the directed efforts of a multitude of thought-disciplines. The engineer must have a solid model to understand the nature of the system. The statistician sets the quality standards of the data. The Scientist must diligently put relevant hypothesis to the test. The advocate must see that the reports made reach out to where it makes the most impact, the list goes on. Much of this is out of scope for this article, and so I will treat you with a link. Here’s your chance to see a hardcore OS engineer at the top of their chosen field speak on the subject of their passion.

All this is to say we are not the only team implementing Merkle tree tech, which also includes the Poseidon hash function it needs. In order to be a premier research destination, key aspect of why VAC exists, the fruits of our labor is just the beginning. We must prove the merit of our efforts through comparative benchmarks that satisfies the skeptics and decision makers.

Comparative benchmarks are among the most high-stakes element of performance critical projects. Get it right, and quality research output can become industry standard technology. Get it wrong, and be ready to lose the trust the field has in you as your precious R&D fades into obscurity.

For the time being, our comparative benchmarks have been used internally to inform decision-makers. As benchmarks become standardised, independently verified and executed, this initial effort may be the first of many steps to a community-wide ecosystem. A thunderdome of benchmarks, leaving us with a single champion that cannot be denied, but which technology will claim this mantle? May the bits be ever in your favor...

Benchmarking with Rust's criterion crate​

Rust, much like Nim, offers unadulterated, fine-grained, and direct control over performance, but with Rust, this control is even more immediate. With its sophisticated ownership model, powerful type system, and comprehensive tooling, Rust has earned an unrivaled reputation for enabling "fearless concurrency," ease of refactoring, and providing tooling that effectively "pair programs with you" to help avoid common pitfalls, includeing those of the performance veriety.

The criterion crate is considered the go-to library for micro-benchmarking within the Rust ecosystem, and is generally regarded as an invaluable tool for obtaining high-quality telemetry. Through its ergonomic idioms and well-thought-out API, writing high-quality benchmarks becomes straightforward once you become familiar with its features. Criterion helps avoid common traps such as inappropriate compiler optimizations, improper performance sampling, and failing to prune telemetry overhead. As is typical for the Rust ecosystem, the documentation is thorough, leaving little to be desired, and the examples are extensive, making the initial learning process a pleasant experience.

Most importantly, it automatically generates tables and graphs from this data, making the crucial task of analysis straightforward and accessible. At this point, we are ready to appreciate the results of our efforts.

Promising results​

When it comes to Merkle trees, we have two elements to consider: The tree itself, and the hashing function that is plugged into it. In the benchmarks we put together for the benefit of internal processes, we put our implementation up against a corresponding FOSS implementation. Scenarios were developed to isolate key performance telemetry, obtain a statistically usable sampling, with the resulting data rendered into a human readable form that can be read with a reasonable degree of confidence: enjoy! The brief summary: It appears that our in house implementation consistently outperforms others, and we’ve decided to continue committing to the R&D of our in-house implementations. Congratulations to the Zerokit team for this accomplishment.

Despite the promising results, these “micro-benchmarks” form just some of the many pieces of the whole system performance when it comes to product needs. How the system performs as a whole is all that matters. This is a promising on it’s own, but watching the performance benefits being realized in the wild is the true goal.

Which brings us back to what started all this: Waku came to us with concerns about performance issues within Zerokit limiting the scope and scale in which it can be used. The engineering talent brought to bear on this issue has successfully achieved the performance goals needed, and the results of these effort have demonstrated there is merit in continuing our commitment to this project.

Conclusion​

We’ve covered a story that starts with crippling performance bottlenecks in Waku, and ends on this high-note: The problematic performance scaling issues are no more, and in the process of resolving this critical pain-point, we have established internal benchmarks that allow us to confidently state that what we are doing, we are doing well. These accomplishments come down to a solid team effort. The open communication coming in from Waku, the talented engineers working together to bring their skills and contributions to bear, the community developed tools and prior works that allowed it all to happen, and those working quietly in the background providing the leadership, resources, and coordination needed to bring this all together. Two VAC/ACZ engineers in particular call for specific mention: Ekaterina for her role in taking lead in the R&D of the Zerokit ecosystem, and Sylvain for his efforts in squeezing out some impressive optimizations. Vinh for unleashing the power of multiple threads, not only for native, but for when running in the browser as well.

Perhaps you want to get involved! Maybe you have some ideas about what the community needs for standard benchmarks. Would you like to see another implementation added to the thunderdome? Raise an issue, or join us on our forum. We look forward to seeing your voice added.

This is just one story, coming out of one relatively small project from VAC research. The two driving directives of the team is to be a conduit of expertise within IFT, and to be a premier research destination within the domains we work in. You might be independent of IFT with an interest in what we do, an IFT core contributor, or anything in between: our services are at your disposal. Join us on discord to start the conversation, email one of our team members, or maybe you might hear a knock on your door, should something in your field of work catch our interest.

If you have comments or suggestions, feel free to reach out to the authors directly or start a thread in the Logos Discord server.

Nim 2.2 – Better Stability, Smarter Memory, and Smoother Development​

The Nim 2.2 release series focuses on improving language stability, fixing long-standing bugs, and optimizing performance—particularly in the ORC memory management system. The latest patch in this series, version 2.2.4, continues to build on these goals.

Here are some of the key highlights from the 2.2 series:

• More powerful generics and type expressions: Stabilization of generics, typedesc, and static types. These features now support arbitrary expressions that previously only worked in limited cases, making them more reliable.
• Improved tuple unpacking: Tuple unpacking now supports discarding values using underscores (_) and allows inline type annotations for unpacked elements.
• Memory leak fixes: Issues with memory leaks when using std/nre’s regular expressions or nested exceptions have been resolved.
• More efficient async code: Futures no longer always copy data, resulting in better performance in asynchronous workflows.
• String bug fixes: Several issues involving string and cstring usage have been corrected.

In addition to core language improvements:

• NimSuggest stability: The language server has received multiple fixes, improving the experience in IDEs and editors that rely on NimSuggest for autocompletion and error checking.
• Better code generation: Numerous issues related to invalid or broken C and C++ code generation and backend-specific bugs have been addressed, improving Nim’s reliability when targeting other languages.

You can read the full release announcement and changelog here

Error Handling in Nim: Why Results Beat Exceptions​

Error handling is one of the most critical aspects of writing reliable software, yet it remains a contentious topic in many programming languages. In Nim, developers face a unique challenge: multiple error handling paradigms are supported, leading to confusion about which approach to choose. For robust, maintainable code, our answer at Logos is increasingly clear—favor Result types over exceptions.

The Exception Problem​

While exceptions might seem convenient for quick scripts and prototypes, they introduce significant challenges in complex, long-running applications:

• Silent API Changes: One of the most dangerous aspects of exception-based error handling is that changes deep within dependencies can break your code without any compile-time warning. When a function suddenly starts throwing a new exception type, your code may fail at runtime under exceptional circumstances—often when you least expect it.
• Resource Management Issues: Exceptions create unpredictable control flow that can lead to resource leaks, security vulnerabilities, and unexpected crashes. When an exception unwinds the stack, resources may not be properly cleaned up.
• Refactoring Difficulties: The compiler provides little assistance when working with exception-based code. Adding a new exception type breaks the ABI but leaves the API unchanged, making it nearly impossible to track down all the places that need updating.

The Result Advantage​

The Result type offers a compelling alternative that makes error handling explicit, predictable, and compiler-verified:

# Enforce that no exceptions can't be raised in this module
{.push raises: [].}

import results

proc doSomething(): Result[void, string] =
# Implementation here

proc getRandomInt(): Result[int, string] =
# Implementation here

doSomething().isOkOr:

echo "Failed doing something, error: ", & error

randomInt = getRandomInt().valueOr:

echo "Failed getting random int, error: ", & error

Notice that this usage of Result is much more concise and easier to follow than try-except blocks

Best Practices for Result-Based Error Handling​

• Make Errors Explicit: Use Result when multiple failure paths exist and calling code needs to differentiate between them. This makes error handling visible at the call site and forces developers to consciously handle failure cases.
• Handle Errors Locally: Address errors at each abstraction level rather than letting them bubble up through multiple layers. This prevents spurious abstraction leakage and keeps error handling logic close to where problems occur.
• Use Exception Tracking: Enable exception tracking with {.push raises: [].} at the module level. This helps identify any remaining exception-throwing code and ensures new code follows the Result pattern.

When to Break the Rules​

While Result should be your default choice, exceptions still have their place:

• Assertions and Logic Errors: Use assertions for violated preconditions or situations where recovery isn't possible or expected.
• Legacy Integration: When interfacing with exception-heavy libraries, you may need to use exceptions at integration boundaries, but convert them to Result types as quickly as possible. To ensure safe exception handling, explicitly declare which exceptions a procedure may raise using the {.raises: [SpecificException].} pragma.

Error handling in Nim continues to evolve, but the trend is clear: explicit error handling through Result types provides better safety, maintainability, and debugging experience than exceptions. By making errors part of your function signatures and forcing explicit handling at call sites, you create more robust software that fails gracefully and predictably.

Debugging in Nim​

Nowadays, analyzing the behavior of a Nim program is not as straightforward as debugging a C++ application, for example.

GDB​

GDB can be used, and step-by-step debugging with GDB and VSCode is possible. However, the interaction is not very smooth. You can set breakpoints in VSCode and press F5 to run the program up to the breakpoint and continue debugging from there. That said, the state of variables is not fully demangled. For example:

For that reason, GDB is not the preferred option in Logos

Logs - Chronicles​

At Logos, we primarily debug Nim applications using log outputs. In particular, we make extensive use of the nim-chronicles library.

nim-chronicles is a robust logging library that automatically includes the following contextual information in each log entry:

• Calling thread ID
• Current timestamp
• Log level (e.g., TRACE, DEBUG, INFO, WARN, ERROR, FATAL)
• Source file name
• Line number of the log statement

Additionally, chronicles supports attaching custom log messages along with relevant variable values, which proves especially useful for debugging. For instance, in the following example, the log message is "Configuration. Shards", and it includes the value of an additional variable, shard.

INF 2025-07-01 09:56:57.705+02:00 Configuration. Shards                      topics="waku conf" tid=28817 file=waku_conf.nim:147 shard=64

There are also useful techniques for displaying more detailed information about specific variables:

• repr(p) — Returns a string representation of the variable p, providing a more comprehensive view of its contents.
• name(typeof(p)) — Extracts the type of the variable p as a string. This is particularly helpful when working with pointers or generics.

Logs - echo​

The echo statement in Nim serves as a basic debugging tool, although it is less powerful and flexible compared to nim-chronicles.

Besides, debugEcho is an interesting alternative, which behaves similarly to echo but it allows working on routines marked with no side effects.

Heaptrack​

This technique enables precise insight into where memory is being consumed within a Nim application.

It is particularly useful for identifying potential memory leaks and is widely employed in nwaku (Nim Waku). For more details, refer to the documentation: Heaptrack Tutorial.

Formatting code in Nim​

Maintaining a consistent code format is essential for readability and for facilitating clear diff comparisons during code reviews.

To support this, Logos strongly recommends using nph across all Nim projects.

Introduction​

Maximum distance separable (MDS) matrices play a significant role in algebraic coding theory and symmetric cryptography. In particular, square MDS matrices are commonly used in affine permutation layers of partial substitution-permutation networks (P-SPNs). These are widespread designs of the modern symmetric ciphers and hash functions. A classic example of the latter is Poseidon [1], a well-known hash function used in zk-SNARK proving systems.

Square MDS matrices differ in terms of security that they are able to provide for P-SPNs. The use of some such matrices in certain P-SPNs may result in existence of infinitely long subspace trails of small period for the latter, which make them vulnerable to differential cryptanalysis [2].

Two methods for security checking of square MDS matrices for P-SPNs have been proposed in [2]. The first one, which is referred to as the three tests method in the rest of the article, is aimed at security checking for a specified structure of the substitution layer of a P-SPN. The second method, which is referred here as the sufficient test method, has been designed to determine whether a square MDS matrix satisfies a sufficient condition of being secure regardless of the structure of a P-SPN substitution layer, i.e. to check whether the matrix belongs to the class of square MDS matrices, which are referred to as unconditionally secure in the current article.

This article aims to introduce MDSECheck method — a novel approach to checking square MDS matrices for unconditional security, which has already been implemented in the Rust programming language as the library crate [3]. The next sections explain the notions mentioned above, describe the MDSECheck method as well as its mathematical foundations, provide a brief overview of the MDSECheck library crate and outline possible future research directions.

MDS matrix: how to define and construct​

An $m$ x $n$ matrix $M$ over a finite field is called MDS, if and only if for distinct $n$-dimensional column vectors $v_1$ and $v_2$ the column vectors $v_1 \: | \: M v_1$ and $v_2 \: | \: M v_2$, where $|$ stands for vertical concatenation, do not coincide in $n$ or more components. The set of all possible column vectors $v \: | \: M v$ for some fixed matrix $M$ is a systematic MDS code, i.e. a linear code, which contains input symbols on their original positions and achieves the Singleton bound. The latter property results in good error-correction capability.

There are several equivalent definitions of MDS matrices, but the next one is especially useful for constructing them directly by means of algebraic methods. A matrix over a finite field is called MDS, if and only if all its square submatrices are nonsingular. The matrix entries and the matrix itself are also considered submatrices.

One of the most efficient and straightforward methods to directly construct an MDS matrix is generating a Cauchy matrix [4]. Such an $m$ x $n$ matrix is defined using $m$-dimensional vector $x$ and $n$-dimensional vector $y$, for which all entries in the concatenation of $x$ and $y$ are distinct. The entries of the Cauchy matrix are described by the formula $M_{i, j} = 1 \: / \: (x_i - y_j)$. It is obvious that any submatrix of a Cauchy matrix is also a Cauchy matrix. The Cauchy determinant formula [5] implies that every square Cauchy matrix is nonsingular. Thus, Cauchy matrices satisfy the second definition of MDS matrices.

Partial substitution-permutation networks​

Describing SPNs in algebraic terms is convenient, so this approach has been chosen for this article. SPNs are designs of the symmetric cryptoprimitives, which operate on an internal state, which is represented as an $n$-dimensional vector over some finite field, and update this state iteratively by means of the round transformations described below.

Each round begins with an optional update of the internal state by adding to its components some input data or extraction of some of these components as the output data. This optional step depends on the specific cryptoprimitive and the current round number. The next step is called the nonlinear substitution layer and lies in replacing the $i$-th component of the internal state with $S_i(c)$ for each $i \in [1..n]$, where $c$ is the component value and $S_i(x)$ is a nonlinear invertible function over the finite field. The function $S_i(x)$ is specific to the cryptoprimitive and called an S-Box. The final step, which is known as the affine permutation layer, replaces the internal state with $M X + c$, where $X$ is the current internal state, $M$ is a nonsingular square matrix and $c$ is the vector of the round constants. The value of $c$ is specific to the cryptoprimitive and the current round number, while $M$ typically depends only on the cryptoprimitive. The data flow diagram for an SPN is given below.

..................................                         
   │        │        │        │                           
   ▼        ▼        ▼        ▼                           
┌────────────────────────────────┐                         
│ Optional addition / extraction │ <─────> Input / output
└──┬────────┬────────┬────────┬──┘                         
   ▼        ▼        ▼        ▼                            
┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐                         
│S₁(x)│  │S₂(x)│  │ ... │  │Sₙ(x)│                         
└──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘                         
   ▼        ▼        ▼        ▼                            
┌────────────────────────────────┐                         
│       Affine permutation       │                         
└──┬────────┬────────┬────────┬──┘                         
   ▼        ▼        ▼        ▼                            
┌────────────────────────────────┐                         
│ Optional addition / extraction │ <─────> Input / output
└──┬────────┬────────┬────────┬──┘                         
   ▼        ▼        ▼        ▼                            
┌─────┐  ┌─────┐  ┌─────┐  ┌─────┐                         
│S₁(x)│  │S₂(x)│  │ ... │  │Sₙ(x)│                         
└──┬──┘  └──┬──┘  └──┬──┘  └──┬──┘                         
   ▼        ▼        ▼        ▼                            
┌────────────────────────────────┐                         
│       Affine permutation       │                         
└──┬────────┬────────┬────────┬──┘                         
   ▼        ▼        ▼        ▼                            
..................................                         

Partial SPNs are modifications of SPNs, where for certain rounds some S-Boxes are replaced with the identity functions to reduce computational efforts [2]. For example, the nonlinear substitution layers of the partial rounds of Poseidon update only the first internal state component [1]. In the case of P-SPNs, security considerations commonly demand to choose $M$ as a square MDS matrix, because these matrices provide perfect diffusion property for the affine permutation layer [6]. Possessing this property means ensuring that any two $n$-dimensional internal states, which differ in exactly $t$ components, are mapped by the affine permutation layer to two new internal states that differ in at least $n - t + 1$ components.

Square MDS matrix security check in the context of P-SPNs​

Certain square MDS matrices should not be used in certain P-SPNs to avoid making them vulnerable to differential cryptanalysis, since it may exploit the existence of infinitely long subspace trails of small period for vulnerable P-SPNs. [2]. Such matrices are called insecure with respect to particular P-SPNs.

An infinitely long subspace trail of period $l$ exists for a P-SPN, if and only if there is a proper subspace of differences of internal state vectors, such that if for a pair of initial internal states the difference belongs to this subspace, then the difference for the new internal states, which are obtained from the initial ones by means of the same $l$-round transformation, also belongs to this subspace [2].

Two methods for checking square MDS matrices for suitability for P-SPNs in terms of existence of infinitely long subspace trails have been proposed in [2]. The three tests method is aimed at checking whether using a specified matrix for a P-SPN with a specified structure of the substitution layer leads to existence of infinitely long subspace trails of period $p$ for this P-SPN for all $p$ no larger than a given $l$. The sufficient test method has been designed to determine whether a square MDS matrix satisfies a sufficient condition of non-existence of infinitely long subspace trails of period $p$ for P-SPNs using this matrix for all $p$ no larger than a specified $l$.

The sufficient test method is a direct consequence of Theorem 8 in [2] and consists in checking that the minimal polynomial of the $p$-th power of the tested matrix has maximum degree and is irreducible for all $p \in [1..l]$. The aforesaid sufficient non-existence condition is satisfied by the matrix, if and only if all the checks yield positive results.

It is convenient to define the unconditional P-SPN security level of the square MDS matrix as follows: this level is $l$ for the matrix $M$, if and only if the minimal polynomials of $M$, $M^2$, $...$, $M^l$ have maximum degree and are irreducible, but for $M^{l \: + \: 1}$ the minimal polynomial does not have this property. Using this definition, the purpose of the sufficient test method can be described as checking whether the unconditional P-SPN security level of the specified matrix is no less than a given bound.

MDSECheck method: getting rid of the matrix powers​

The MDSECheck method, whose name is derived from the words "MDS", "security", "elaborated" and "check", has the same purpose as the sufficient test method, but achieves it differently. The differences of the first method from the latter and approaches to implementing them can be described as follows:

1. Computation and verification of minimal polynomials of $M^2$, $M^3$, $...$, $M^l$, where $M$ is the tested $n$ x $n$ matrix over $GF(q)$ and $l$ is the security level bound, has been replaced with checks for the corresponding powers of a root of the characteristic polynomial of $M$ for non-presence in nontrivial subfields of $GF(q^n)$.

   1. The non-presence check is performed without straightforward consideration of all nontrivial subfields of $GF(q^n)$. The root is checked only for non-presence in the subfields $GF(q^{n \: / \: p_1})$, $GF(q^{n \: / \: p_2})$, $...$, $GF(q^{n \: / \: p_d})$, where $p_1$, $p_2$, $...$, $p_d$ are all prime divisors of $n$.

   2. The non-presence check reuses some data computed during the checking for irreducibility the minimal polynomial of $M$, which in this case coincides with $f(y)$ designating the characteristic polynomial of $M$. The values of $y^{q^{n \: / \: p_j}} \mod f(y)$ are saved for each $j \in [1..d]$ during the irreducibility check to replace exponentiations with sequential computations of $(y^i)^{q^{n \: / \: p_j}} \mod f(y)$ from $(y^{(i \: - \: 1)})^{q^{n \: / \: p_j}} \mod f(y)$ as its product with $y^{q^{n \: / \: p_j}} \mod f(y)$.

2. The check of the minimal polynomial of $M$ for irreducibility and maximum degree is performed without unconditional computation of this polynomial. This computation has been replaced with the Krylov method fragment, which consists in building and solving only one system of linear equations over $GF(q)$. If $M$ has an irreducible minimal polynomial of maximum degree, then its coefficients are trivially determined from the system solution. If the system is degenerate, then the minimal polynomial of $M$ does not have such properties.

The correctness of the first distinctive feature can be proven as follows. Verifying that the minimal polynomial of a matrix is of maximum degree and irreducible is equivalent to verifying that the characteristic polynomial of this matrix is irreducible, because the minimal polynomial divides the characteristic one. Also, it is trivially proven that for a matrix with such a minimal polynomial it is equal to the characteristic polynomial. Thus, the required checks for the matrices $M^2$, $M^3$, $...$, $M^l$ can be done by checking their characteristic polynomials for irreducibility.

Let $M$ be $n$ x $n$ matrix over $GF(q)$, whose minimal polynomial is of maximum degree and irreducible. The statements in the previous paragraph imply that $f(y)$, which is the $n$-degree characteristic polynomial of $M$, is irreducible. Consider $M$ over the extension field $GF(q^n)$, which is the splitting field of $f(y)$. Let $α \in GF(q^n)$ be a root of $f(y)$. According to standard results from the Galois field theory, $α$, $α^q$, $α^{q^2}$, $...$, $α^{q^{n \: - \: 1}}$ are distinct roots of $f(y)$ [7]. Thus, these powers of $α$ are $n$ distinct eigenvalues of $M$. Hence, due to matrix similarity properties, there is some matrix $S$ such that $S M S^{-1} = D$, where $D$ is the diagonal matrix, whose nonzero elements are $α$, $α^q$, $α^{q^2}$, $...$, $α^{q^{n \: - \: 1}}$. Therefore, $S M^i S^{-1} = D^i$, so the roots of the characteristic polynomial of $M^i$ are $α^i$, $(α^q)^i$, $(α^{q^2})^i$, $...$, $(α^{q^{n \: - \: 1}})^i$. If the minimal polynomial of $α^i$ has degree less than $n$, then the characteristic polynomial of $M^i$ is divisible by this minimal polynomial, while $α^i$ lies in some nontrivial subfield of $GF(q^n)$. One of the fields isomorphic to this subfield is a residue class ring of polynomials modulo the minimal polynomial of $α^i$ [7]. If the minimal polynomial of $α^i$ is of degree $n$, then the characteristic polynomial of $M^i$ equals this minimal polynomial and therefore is irreducible, while $α^i$ does not lie in any nontrivial subfield of $GF(q^n)$. In this case, $1$, $α^i$, $(α^i)^2$, $...$, $(α^i)^{n \: - \: 1}$ are linearly independent as distinct roots of an irreducible polynomial over a finite field [7], so any field containing $α^i$ has at least $q^n$ elements and therefore cannot be a trivial subfield of $GF(q^n)$. Thus, checking the characteristic polynomials of the matrices $M^2$, $M^3$, $...$, $M^l$ for irreducibility is equivalent to verifying that $α^2$, $α^3$, $...$, $α^l$ do not lie in any nontrivial subfield of $GF(q^n)$.

The last sentences of the two previous paragraphs imply the following: verifying that the minimal polynomials of $M^2$, $M^3$, $...$, $M^l$ are of maximum degree and irreducible can be performed by verifying that the corresponding powers of a root of the characteristic polynomial of the $n$ x $n$ matrix $M$ over $GF(q)$ do not belong to any nontrivial subfield of $GF(q^n)$. $\blacksquare$

The approaches to implementing the first distinctive feature can be explained and proven to be correct as follows. Since $GF(q^w)$ is a nontrivial subfield of $GF(q^u)$ if and only if $w$ divides $u$ and $w < u$ [7], the presence of some $ε$ in $GF(q^h)$, which is a nontrivial subfield of $GF(q^n)$, implies that $ε \in GF(q^{n \: / \: ν})$ for some prime $ν$ dividing $n$, because $h$ divides the quotient of $n$ and some of its prime factors. Thus, checking that some value does not belong to subfields $GF(q^{n \: / \: p_1})$, $GF(q^{n \: / \: p_2})$, $...$, $GF(q^{n \: / \: p_d})$, where $p_1$, $p_2$, $...$, $p_d$ are all prime divisors of $n$, is equivalent to checking this value for non-presence in nontrivial subfields of $GF(q^n)$.

Checking for irreducibility the minimal polynomial of $M$ is performed by means of Algorithm 2.2.9 in [8] and consists in sequential computation of $y^p \mod f(y)$, $y^{p^2} \mod f(y)$, $...$, $y^{p^{\lfloor n \: / \: 2 \rfloor}} \mod f(y)$ and checking that $GCD(y^{p^i} \mod f(y) \: - \: y, f(y)) = 1$ for each $i \in [1..\lfloor n \: / \: 2 \rfloor]$, where $f(y)$ is the characteristic polynomial of $M$ and coincides with the minimal polynomial in this case. The optimized root non-presence check is performed by checking that for each $i \in [2..l]$ for each $j \in [1..d]$ the value of $((y^i)^{q^{n \: / \: p_j}} - y^i) \mod f(y)$ is nonzero. This approach is based on the following standard results from the Galois field theory [7]:

• $GF(q^n)$ is isomorphic to the residue class ring of univariate polynomials in $y$ modulo $f(y)$, because at this point $f(y)$ is known to be irreducible, and some root of $f(y)$ is mapped by this isomorphism to the residue class the polynomial $y$ in this ring.

• All elements of a finite field $GF(q^w)$ and only they are roots of $y^{q^w} - y$.

The expression $((y^i)^{q^{n \: / \: p_j}} - y^i) \mod f(y)$ can be rewritten as $((y^{q^{n \: / \: p_j}})^i - y^i) \mod f(y)$, which can be computed without exponentiation as the product of $(y^{(i \: - \: 1)})^{q^{n \: / \: p_j}} \mod f(y)$ and $y^{q^{n \: / \: p_j}} \mod f(y)$, which has been saved during the irreducibility check.

The second distinctive feature can be explained and proven to be correct in following way. The $n$ x $n$ matrix $M$ does not have a minimal polynomial of maximum degree, if some Krylov subspace of order $n$ for it is not $n$-dimensional. Indeed, the minimal polynomial of the matrix is divisible by the minimal polynomial of the restriction of this linear operator to an arbitrary subspace, and in the considered case the latter polynomial has degree less than $n$, because the degree of the minimal polynomial of a linear operator cannot exceed the dimension of the subspace the operator acts on. Thus, an unconditional computation of the minimal polynomial of $M$ is not required to determine whether this polynomial is irreducible and has maximum degree. Using this computation has been replaced with the Krylov method fragment, which consists in choosing any nonzero $n$-dimensional column vector $v$ and solving the system of linear equations $A X = b$, where $A$ is an $n$ x $n$ matrix, whose columns are $v$, $M v$, $M^2 v$, $...$, $M^{n \: - \: 1} v$, and $b$ is $M^n v$. If $A$ is singular, the minimal polynomial of $M$ is reducible or does not have maximum degree, so checking $M$ has been accomplished; otherwise, $f(y)$, which is the minimal and characteristic polynomial of $M$, can be expressed as $y^n - X_{n \: - \: 1} y^{n \: - \: 1} - X_{n \: - \: 2} y^{n \: - \: 2} - … - X_1 y - X_0$.

The steps of MDSECheck method can be summarized as follows:

1. The square MDS matrix $M$ over $GF(q)$ and the unconditional P-SPN security level bound $l$ are received as inputs.

2. The Krylov method fragment is used to compute the minimal polynomial of $M$. If the computation fails, then $M$ is not unconditionally secure, so the check of $M$ is complete. If it succeeds, then the minimal polynomial has maximum degree and, therefore, coincides with the characteristic polynomial of $M$.

3. Algorithm 2.2.9 is used to check for irreducibility the minimal polynomial of $M$, which is also the characteristic polynomial of $M$ in this case. Some data computed during this step is saved to be reused at the next one. If the polynomial is reducible, then the check of $M$ is complete, because $M$ has been found to be not unconditionally secure.

4. The values of $α^2$, $α^3$, $...$, $α^l$, where $α$ is a root of the characteristic polynomial of $M$, are sequentially checked for non-presence in nontrivial subfields of $GF(q^n)$ as described above. If $α^i$ belongs to some nontrivial subfield of $GF(q^n)$, then the unconditional P-SPN security level of $M$ is $i \: - \: 1$, so the check of $M$ is complete. If all the values do not belong to such a subfield, then the unconditional P-SPN security level is at least $l$.

MDSECheck library crate: implementation in Rust​

The library crate [3] provides tools for generating random square Cauchy MDS matrices over prime finite fields and applying the MDSECheck method to check such matrices for unconditional security. The used data types of field elements and polynomials are provided by the crates ark-ff [9] and ark-poly [10]. The auxiliary tools in the crate modules are accessible as well.

Generating by means of this crate a 10 x 10 MDS matrix, which is defined over the BN254 scalar field [11] and has unconditional P-SPN security level is 1000, takes less than 60 milliseconds on average for the laptop with the processor Intel® Core™ i9-14900HX, whose maximum clock frequency is 5.8 GHz.

Conclusion​

The MDSECheck method proposed in this article is a novel approach to checking square MDS matrices for unconditional security as the components of affine permutation layers of P-SPNs. It has been implemented as a practical library crate for generating unconditionally secure square MDS matrices for P-SPNs over prime finite fields.

The future research directions may include theoretical and experimental studies of performance of approaches, which use the MDSECheck method to generate unconditionally secure square MDS matrices for P-SPNs.

References​

1. L. Grassi, D. Khovratovich, C. Rechberger, A. Roy, M. Schofnegger. "POSEIDON: A New Hash Function for Zero-Knowledge Proof Systems (Updated Version)".
2. L. Grassi, C. Rechberger, M. Schofnegger. "Proving Resistance Against Infinitely Long Subspace Trails: How to Choose the Linear Layer".
3. The page "mdsecheck" on crates.io.
4. Y. Kumar, P. Mishra, S. Samanta, K. Chand Gupta, A. Gaur. "Construction of all MDS and involutory MDS matrices".
5. The page "Value of Cauchy Determinant" on proofwiki.org.
6. T. Silva, R. Dahab "MDS Matrices for Cryptography".
7. S. Huczynska, M. Neunhöffer. "Finite Fields"
8. R. Crandall, C. Pomerance. "Prime Numbers: A Computational Perspective" (2nd edition).
9. The page "ark-ff" on crates.io.
10. The page "ark-poly" on crates.io.
11. The page "ark-bn254" on crates.io.

With 2024 now behind us and a new year ahead, Vac is proud to reflect on the milestones and breakthroughs that defined another year of researching and developing free and open digital public goods for the Institute of Free Technology and wider web3 ecosystem.

Vac comprises various subteams and service units, each with its own focus. Below, we celebrate each unit's achievements and look forward to its 2025 plans.

Nescience​

Nescience is our state separation architecture that aims to enable private transactions and provide a general-purpose execution environment for classical applications.

Highlights​

This year, the Nescience state separation architecture moved from exploration to real progress, taking significant steps towards building a functional and reliable system. The team focused on turning ideas into something real, testing the proposed architecture, and understanding its strengths and weaknesses.

• ZkVM exploration and benchmarks
  • Published deep reviews of 23 existing zkVMs
  • Benchmarked the performance of the six zkVMs that best fit Nescience
• Defined the NSSA architecture
  • Brought clarity to NSSA’s design and explained the system’s architecture in a lengthy exploratory blog post
• Built the sandboxed testnet
  • Designed the first version of the node specification
  • All core components (execution types, UTXOs, cryptographic primitives) implemented and being tested
  • Testing the performance of all execution types in various scenarios

We also made progress on the essential parts of NSSA’s system, including:

• Key protocol for secure key management
• Execution types and circuits for reliable computation
• UTXO specification to manage state transitions effectively
• Cryptography module to ensure privacy and security

Looking forward​

In 2025, the Nescience team plans to double down on what works, fix what doesn’t, and push NSSA closer to real-world use.

• Sandboxed testnet data analysis – the sandboxed testnet will be our primary data source that we will analyse to identify issues, limitations, and areas for improvement.
• Expanding the node – expand sandboxed components into a full node implementation with rigorous testing and iterative optimization (to bridge the gap between proof of concept and production readiness).
• Finalizing the architecture and RFC – after completing NSSA’s architecture, we will draft an RFC to ensure transparency and enable greater collaboration with the broader ecosystem.
• Testing real-life scenarios – applying NSSA to diverse, practical use cases to assess its adaptability, performance, and impact.
• Ongoing optimization – ensure NSSA is robust, efficient, and ready to scale.

Token Economics (TKE)​

The TKE Service Unit works closely with IFT portfolio projects to design and implement crypto-economic incentive structures.

Highlights​

• Formalized and implemented Codex economic incentives in the Litepaper and simulations
• Orchestrated Status Network incentive structure and smart contract implementation
• Started building Nomos’s economic model
• Consulted and provided analysis of incentives for the Logos Operators ordinals project
• Drove discussions on the economic sustainability of Waku; helped define RLN membership and its payment mechanism

Looking forward​

In 2025, TKE will continue to support IFT portfolio projects, working toward economic sustainability while strengthening relationships within the organization. Additionally, the service unit aims to continue building its external reputation through partnerships and publications of relevant work on the Vac forum.

Quality Assurance (QA)​

The QA Service Unit focuses on the development and execution of comprehensive test plans, including implementing unit and interoperability testing.

Highlights​

• Matured Waku interoperability testing framework with coverage for all major protocols and features
• Began collaboration with Nomos, contributing to unit and integration testing
• Partnered with the Status team to test message reliability under unstable network conditions

Looking forward​

• Extend collaboration with the Waku team on go-waku bindings and message reliability testing
• Cement working relationship with the Nomos team through the building of an E2E testing framework for higher-level node validation
• Work closely with Status’s QA team to enhance the functional testing framework
• Continue work on nim-libp2p testing
• Expand collaboration to additional projects

RFC​

The RFC Service Unit takes on the responsibility of shepherding and editing specifications for IFT projects. The unit acts as a linchpin for ensuring standardized and interoperable protocols within the IFT ecosystem.

Highlights​

• Working to implement RFC culture across the IFT ecosystem
• Began editorial work for several IFT portfolio projects: Status, Nomos, Waku, and Codex.
• Reworked our standards with regard to writing RFCs to a consensus-oriented specification system

Looking forward​

• Continue to implement RFC culture across the IFT ecosystem
• Broaden the number of RFCs produced – particularly for IFT portfolio projects nearing public releases
• Include new projects with the rfc-index
• Encourage external projects requiring RFCs to establish relationships with the service unit

Applied Cryptography and ZK (ACZ)​

The ACZ Service Unit focuses on cryptographic solutions and zero-knowledge proofs, enhancing the security, privacy, and trustworthiness of IFT portfolio projects and contributing to the overall integrity and resilience of the decentralized web ecosystem.

Highlights​

• Researched a libp2p mix protocol and first proof-of-concept implementation (including ping and GossipSub over mix)
• Researched a decentralized version of MLS (message layer security) with a first proof of concept
• Released Zerokit v0.6.0 and v0.5.0
• Added gnark RLN implementation
• Released Stealth Address Kit v0.3.1, v0.2.0, and v0.1.0
• Published:
  • Verifying RLN Proofs in Light Clients with Subtrees
  • RLN-v3: Towards a Flexible and Cost-Efficient Implementation

Looking forward​

• Ensure libp2p mix protocol is production-ready and support with the publishing of a paper and blog posts
• Ensure decentralized MLS is production-ready and support with the publishing of a paper and blog posts
• Begin explorations of additional research topics
• Release Zerokit v0.7 and future versions

P2P​

The P2P Service Unit specializes in peer-to-peer technologies and develops nim-libp2p, improves the libp2p GossipSub protocol, and assists IFT portfolio projects with the integration of P2P network layers.

Highlights​

• Analysis and work on libp2p GossipSub improvements
• Published:
  • Libp2p GossipSub IDONTWANT Message Performance Impact
• PR to libp2p specifications about specific lib2p GossipSub improvements we researched and tested https://github.com/libp2p/specs/pull/654

Looking forward​

• Add new features to nim-libp2p: QUIC transport, web transport
• Update specifications for libp2p GossipSub, aiming to significantly improve its performance

Distributed Systems Testing (DST)​

The DST Service Unit’s primary objective is to assist IFT portfolio projects in understanding the scaling behavior of their nodes within larger networks. By conducting thorough regression testing, the DST unit helps ensure the reliability and stability of projects.

Highlights​

• DST compute resources transitioned from a hosted environment to a dedicated Vac Lab, enabling better customization of resources and adding significantly more compute power – enabled much higher and more stable simulations (several hundred nodes to several thousand) and enhanced environmental control.
• Maintained monthly regression simulations for both Waku and Nim-libp2p, helping us to detect several issues and ensure that future versions do not introduce new ones.
• Successfully simulated and obtained results for all Waku protocols, relaying feedback to the team.

Looking forward​

• More testing and simulations for Codex and Nomos
• Develop useful tools for all IFT portfolio projects – e.g. a Log Parser tool and data dashboard

Nim​

Several IFT portfolio projects use the Nim ecosystem for its efficiency. The Nim Service Unit is responsible for the development and maintenance of Nim tooling.

Highlights​

• Released Nim-libp2p (v1.7.1, v1.7.0, v1.6.0, v1.5.0, v1.4.0, v1.3.0, v1.2.0)
• Introduced SAT solver to the Nimble package manager that significantly improves dependency resolution
• Nim VSCode Extension
• Stabilized Nim Language Server

Smart Contracts (SC)​

Vac's Smart Contracts Service Unit ensures the smart contracts deployed across the various IFT portfolio projects are secure, robust, and aligned with project requirements.

Highlights​

• Deployed the SNT staking protocol testnet following Status's governance vote to develop SNT staking and Status Network
• Wrote specifications for Codex's architectural components and Status's staking contracts
• Delivered several learn-up sessions on a variety of topics for IFT contributors, including:
  • Stealth addresses
  • Tokenized vaults
  • Rental NFTs
  • Merkle trees
  • Account abstraction
  • EVM deep dive

Looking forward​

• Deploy the SNT staking protocol on the Status Network testnet
• Encourage community security audits via contests
• Provide smart contract consultation services for IFT portfolio products
• Engage in more learn-up sessions to promote org-wide knowledge sharing.

Heading into 2025​

This year has seen Vac involved with many research, development, and testing undertakings in support of IFT portfolio projects. The digital public goods that emerge from our efforts not only support the organization itself but are open and free to use by any project that would benefit.

As we move into 2025, we aim to nurture a stronger RFC culture across the IFT to encourage greater collaboration and knowledge sharing among portfolio projects. Our goal is to serve as an internal conduit of expertise within the organization, supported by a strong RFC culture, maintaining a repository of internal knowledge creation, and identifying and facilitating IFT project synergies. Such an approach should lead to greater efficiencies across the organization.

We also aim to establish a diverse research community around Vac, and our efforts in this regard are already underway. In the final quarter of 2024, Vac stepped up its collaboration with the libp2p community and made a concerted effort to engage the community on the Vac forum. In 2025, we aim to continue working closely with those communities to which we already have ties, such as the libp2p, Ethereum, and Nim ecosystems.

We look forward to continuing our journey with you!

Follow Vac on X, join us in the Vac Discord, or take part in the discussions on the Vac forum to stay up to date with our research and development progress.

Introduction​

A large amount of data is swapped between users on a blockchain in the form of transactions. Over the entire life of a blockchain, the storage space required to maintain a copy of every transaction becomes untenable for most users. However, the integrity of a blockchain relies on a large pool of users that can validate the blockchain's history from its inception to its present state. The data representing the blockchain's state is compressed. This compression addresses the issue of scalability that would otherwise greatly restrict the pool of users.

Data compression alone is not the end goal. As mentioned, it is essential for users to be able to validate the blockchain's history. The property of compression and validation was solved in Bitcoin by the use of Merkle trees. Merkle trees were introduced first by Ralph Merkle in his dissertation [1]. A Merkle tree is a data structure that compresses a digest of data to a constant size while still providing a method for proving membership of elements of the digest. A previous rlog[2] described how Merkle trees with their proof of membership could be used for lightweight clients for RLN.

Tree structure​

A tree is a special data structure that organizes nodes so that there is exactly one path between any two nodes. The trees that we consider can be arranged in layers with multiple nodes (children) merged into a single node (parent) in the preceding layer. A single node exists in the base layer; this special node is called the root node. The highest level of the tree consists of childless nodes called leaves.

A binary tree has one additional property: each nonleaf node has exactly two children nodes. That is, we assume that nodes in a binary tree are either a parent node with two children or a leaf. As strange as it sounds, each child node has exactly one parental node.

A binary tree with $2^n$ leaves consists of $n+1$ layers. Additionally, such a tree has $2^{n+1}-1$ nodes.

Merkle trees​

A Merkle tree is a specialized tree in which each node contains the evaluation of a hash function. Merkle trees are usually taken to have a binary tree structure. As such, the presentation we provide in this section will be for binary trees.

Construction​

In this section, we show how Merkle trees are constructed to compress a digest $D$. Suppose that the digest $D$ consists of $2^n$ entries; we assume that the digest $D$ has this many entries since a Merkle tree is a binary tree. Additionally, each digest can be padded to ensure that $D$ has the desired number of entries.

Each leaf of the Merkle tree contains the hash of a digest entry. Each parent node contains the hash of the concatenation of their child nodes. Through this iterative construction, we reach the root of the tree. The value contained in the root node is called the root hash. The root hash is a compressed representation of the digest $D$.

Each node in the Merkle tree is computed by taking a hash. Since a binary tree with $2^n$ leaves has $2^{n+1}-1$ nodes, then we need to evaluate $2^{n+1}-1$ hashes to construct the Merkle tree.

Merkle tree intregrity​

A large quantity of data can be compressed to a single hash value. A natural question to ask is: could a clever party find another digest that yields a Merkle tree with the same root hash? If possible, this would compromise the ledger since the blockchain's history could be altered. Fortunately, Merkle trees are quite secure. In fact, Merkle trees can be used to both bind and hide a digest.

The Merkle tree is able to bind a digest with one of the properties of hash functions (see our previous Vac 101 [3] for information on hash functions). A hash function is collision resistant; it is infeasible for a malicious party to find two values share the same hash value.

This collision resistance property, essentially, fixes the input to each leaf and into their parent, their parent's parent, and so on.

In certain applications, it may be desirable for the digest of a Merkle tree to be kept confidential. This is achieved with the preimage resistant property of hash functions. A hash function is preimage resistant provided that it is difficult to reverse the hashing operation. It would be necessary for a malicious party to find preimages to each node starting from the root node to determine the original digest.

Now, we see that Merkle trees are secured structures that are tamper resistant.

Proof of membership​

An interesting and critical property of Merkle trees is their ability to prove that any piece of data is part of its digest. This can be done with logarithmic storage and logarithmic computation time.

Suppose that we want to show that data $\ell$ is part of the Merkle tree's digest. Additionally, suppose that $\mathsf{hash}$ is the hash function used to construct the tree. We assume that the hash function $\mathsf{hash}$ can be computed in constant-time for any input.

Suppose that a prover provides data $\ell$ to a verifier, and tells the verifier that $\ell$ corresponds to the $i$th leaf of the Merkle tree. For the verifier to be convinced that $\ell$ is part of the digest, he needs to be able to construct the tree's root hash using $\mathsf{hash}$, $i$, $\ell$ and some additional information from the prover. Specifically, the prover must provide the sibling hashes for each value that the verifier can compute. This enables the verifier to compute the parents of the siblings that the prover provides and the values that he was able to produce himself. The last of the computed parents is the root.

The leaf index $i$ indicates whether a hash value provided by the prover is a left or right sibling. This is done by looking at the binary expansion of $i$.

The verifier can compute the leaf $h_0 = \mathsf{hash}(\ell)$. Next, using $h_0$'s sibling, $h'_0$, provided by the prover, the verifier can compute $h_1 = \mathsf{hash}(h_0 \|h'_0)$ or $h_1 = \mathsf{hash}(h'_0 \| h_0)$ depending on whether $h'_0$ is a left or right sibling. This pathing continues until the verifier either successfully computes the root hash (in $n+1$ hashes) or fails to do so.

The prover has to provide $n$ sibling nodes for the proof of membership.

There is a key detail that is essential for the proof of membership to be secure. The root hash has to be provided to the verifier prior to the selection of data $\ell$. Otherwise, the prover could generate a series of hash values (with the corresponding root hash) to forge a proof of membership.

Capped proof of membership​

Polygon provides an implementation [4] of a shortened proof of membership with a slight modification. A specific layer of the Merkle tree is published instead of just the root hash. By doing this, a capped proof of membership is just the path from leaf to the published layer.

Extensions of Merkle trees​

Merkle trees can be extended in multiple ways. In this section, we explore a select few of these extensions.

Sparse Merkle trees​

A sparse Merkle tree (SMT) is a special Merkle tree that can be used to represent digests with nonconsecutive entries. Specifically, each digest entry has a particular leaf index. For simplicity, we assume that the index value is computed by taking the hash of the entry. We note that this is a sorted SMT.

Let $n$ denote the number of bits that a hash value can possess. This means that our SMT can have at most $2^n$ leaves.

An SMT is treated as a Merkle tree in which each entry is placed in the leaf corresponding to its hash value, and the other entries have a $\mathsf{null}$ marker inserted in. This means that we can prove membership in the way described. However, we can also prove nonmembership of an element by showing that $\mathsf{null}$ is located in the element's hash location. The crucial difference between a sorted and unsorted SMT is that the unsorted variant cannot be used to prove nonmembership.

We can take advantage of the sparse nature of SMTs to provide shortened proofs. Specifically, it is unlikely for entries to cluster together. Thus, it is efficient to maintain a list of values:

Each of the $d_i$'s represents the root hash of a Merkle tree with $2^i$ leaves containing $\mathsf{null}$. These values can be used to shorten the time needed to construct an SMT and the length of proofs.

Proof of nonmembership​

In the first Vac 101 [5], we examined Bloom and Cuckoo filters that could be used for proof of membership and nonmembership. However, the proof of membership may result in false positives due to collisions. This would affect nonmembership proofs as well. Sparse Merkle trees can be adapted to provide greater assurance that a given piece of data is not a member of the digest.

Why is sorting essential? The sorting mechanism of data can be arbitrarily chosen. However, it is essential that there are no gaps in the ordering. The maximum number of elements that could ever exist in the digest must be known. A simple method for this is to use a hash function to provide fingerprints to the data. Each hash using either SHA-256 or Keccak has 256-bits. Our entire digest could consist of a maximum of $2^{256}$ entries. This assumes that our digest does not contain collisions.

The fingerprint of a piece of data $\ell$ indicates which leaf of the SMT it is contained in. This means that a nonmembership of $\ell$ in the SMT becomes a matter of proving that $\mathsf{null}$ is contained in $\ell$'s location.

It is crucial for the SMT to be sorted. Otherwise, a malicious party can append the entry $\ell$ to a random location. This allows for the malicious party to provide contradictory proofs that prove both membership and nonmembership. We note that the requirement that an SMT is sorted may be too strong of an assumption in centralized cases. However, sortedness is a necessary property of SMTs for decentralized systems.

Verkle Trees​

A proof of membership grows in length as the Merkle tree grows. The most obvious approach to remedy this scalability issue is to use Merkle trees in which each node has more than two children. However, this does not fix the issue. A proof of membership in a $k$-nary Merkle tree [6] (each node has $k$ children) has a proof size $\log_k(n)(k-1)$. The multiple $k-1$ is the number of silbings that a node has on each layer. Hence, the proof size grows faster than a logarithmic function of the digest size.

An alternate approach is to use a different data structure: Verkle trees [6]. A Verkle tree replaces hash functions with polynomial commitments [7, 8]. We will explore Verkle trees in a future Vac 101 edition.

References​

• 1. Secrecy, Authentication, and Public Key Systems
• 2. Verifying RLN Proofs in Light Clients with Subtrees
• 3. Vac 101: Transforming an Interactive Protocol to a Noninteractive Argument
• 4. Capped merkle tree in Plonky2
• 5. Vac 101: Membership with Bloom Filters and Cuckoo Filters
• 6. Verkle Trees
• 7. Using polynomial commitments to replace state roots
• 8. KZG polynomial commitments
• 9. O1 labs' Verkle Tree repo

Motivation​

The challenge of large message transmissions in GossipSub leads to longer than expected network-wide message dissemination times (and relatively higher fluctuations). It is particularly relevant for applications that require on-time, network-wide dissemination of large messages, such as Ethereum and Waku [1,2].

This matter has been extensively discussed in the libp2p community [3, 4, 5, 6, 7, 8], and numerous improvements have been considered (or even incorporated) for the GossipSub protocol to enable efficient large-message propagation [3, 7, 9, 10].

Problem description​

Sending a message to N peers involves approximately $\lceil \log_D(N) \rceil$ rounds, with approximately $(D-1)^{X-1} \times D$ transmissions in each round, where $X, D$ represent the round number and mesh size.

Transmitting to a higher number of peers (floodpublish) can theoretically reduce latency by increasing the transmissions in each round to $(D-1)^{X-1} \times (F+D)$, where $F$ represents the number of peers included in floodpublish.

This arrangement works fine for relatively small/moderate message sizes. However, as message sizes increase, significant rises and fluctuations in network-wide message dissemination time are seen.

Interestingly, a higher $D$ or $F$ can also degrade performance in this situation.

Several aspects contribute to this behavior:

1. Ideally, a message transmission to a single peer concludes in $\tau_1 = \frac {L}{R}+P$ (ignoring any message processing time), where $L, R, P$ represent message size, data rate, and link latency. Therefore, the time required for sending a message on a 100Mbps link with 100ms latency jumps from $\tau_1^{10KB} = 100.8ms$ for a 10KB message to $\tau_1^{1MB} = 180ms$ for a 1MB message. For $D$ peers, the transmission time increases to $\tau_D^{1MB} = (80 \times D) + 100ms$, triggering additional queuing delays (proportional to the transmission queue size) during each transmission round.

2. In practice, $\tau_1^{1MB}$ sometimes rises to several hundred milliseconds, further exaggerating the abovementioned queuing delays. This rise is because TCP congestion avoidance algorithms usually limit maximum in-flight bytes in a round trip time (RTT) based on the congestion window ($C_{wnd}$) and maximum segment size (MSS) to approximately ${C_{wnd} \times MSS}$, with $C_{wnd}$ rising with the data transfer for each flow. Consequently, sending the same message through a newly established (cold) connection takes longer. The message transfer time lowers as the $C_{wnd}$ grows. Therefore, performance-improvement practices such as floodpublish, frequent mesh adjustment, and lazy sending typically result in longer than expected message dissemination times for large messages (due to cold connections). It is also worth mentioning that some TCP variants reset their $C_{wnd}$ after different periods of inactivity.

3. Theoretically, the message transmission time to D peers $(\tau_D)$ remains the same even if the message is relayed sequentially to all peers or simultaneous transmissions are carried out, i.e., $\tau_D = \sum_{i=1}^{D} \tau_i$ However, sequential transmissions finish early for individual peers, allowing them to relay early. This may result in quicker network-wide message dissemination.

4. A realistic network comprises nodes with dissimilar capabilities (bandwidth, link latency, compute, etc.). As the message disseminates, it's not uncommon for some peers to receive it much earlier than others. Early gossip (IHAVE announcements) may bring in many IWANT requests to the early receivers (even from peers already receiving the same message), which adds to their workload.

5. A busy peer (with a sizeable outgoing message queue) will enqueue (or simultaneously transfer) newly scheduled outgoing messages. As a result, already scheduled messages are prioritized over those published by the peer itself, introducing a significant initial delay to the locally published messages. Enqueuing IWANT replies to the outgoing message queue can further exaggerate the problem. The lack of adaptiveness and standardization in outgoing message prioritization are key factors that can lead to noticeable inconsistency in message dissemination latency at each hop, even in similar network conditions.

6. Message size directly contributes to peers' workloads in terms of processing and transmission time. It also raises the probability of simultaneous redundant transmissions to the same peer, resulting in bandwidth wastage, congestion, and slow message propagation to the network. Moreover, the benefits of sequential message relaying can be compromised by prioritizing slow (or busy) peers.

7. Most use cases necessitate validating received messages before forwarding them to the next-hop peers. For a higher message transfer time $(\tau )$, this store-and-forward delay accumulates across the hops traveled by the message.

Possible improvements​

1. Minimizing transfer time for large messages​

The impact of message size and achievable data rate on message transmit time $\tau$ is crucial as this time accumulates due to the store-and-forward delay introduced at intermediate hops.

Some possible improvements to minimize overall message dissemination latency include:

a. Message fragmentation​

In a homogeneous network, network-wide message dissemination time (ignoring any processing delays) can be simplified to roughly $\delta \approx \delta_{Tx} + P_h$, where $\delta_{Tx}$ represents accumulative message transmit time denoted as $\delta_{Tx} = \frac{S}{R} \times h$, with $S, R$ being the data size and data rate, and $h, P_h$ being the number of hops in the longest path and message propagation time through the longest path.

Partitioning a large message into n fragments reduces a single fragment transmit time to $\frac{\delta_{Tx}}{n}$. As a received fragment can be immediately relayed by the receiver (while the sender is still transmitting the remaining fragments), it reduces the transmit time to $\delta_{Tx} = \frac{S}{R} \times \frac{2h-1}{n}$.

This time reduction is mainly attributed to the smaller store-and-forward delay involved in fragment transmissions.

However, it is worth noting that many applications require each fragment to be individually verifiable. At the same time, message fragmentation allows a malicious peer to never relay some fragments of a message, which can lead to a significant rise in the application's receive buffer size.

Therefore, message fragmentation requires a careful tradeoff analysis between time and risks.

b. Message staggering​

Considering the same bandwidth, the time $\tau_D$ required for sending a message to D peers stays the same, even if we relay to all peers in parallel or send sequentially to the peers, i.e., $\tau_D = \sum_{i=1}^{D} \tau_i$.

However, sequential relaying results in quicker message reception at individual peers ($\tau_1 \approx \frac{\tau_D}{D}$) due to bandwidth concentration for a particular peer. So, the receiver can start relaying early to its mesh members while the original sender is still sending the message to other peers.

As a result, after every $\frac{\tau_D}{D}$ milliseconds, the number of peers receiving the message increases by $2^X\ \forall\ X \in \{0, D-1\}$ and by $\sum_{k=X-D}^{X-1} \lambda_k\ \forall\ X \geq D$. Here, $X$ represents message transmission round $X = i \cdot \frac{\tau_D}{D} \mid i \in \mathbb{N}_0$, and $\lambda_k$ represents the number of peers that received the message in round $k$.

It is worth noting that a realistic network imposes certain constraints on staggering for peers. For instance, in a network with dissimilar peer capabilities, placing a slow peer (also in cases where many senders simultaneously select the same peer) at the head of the transmission queue may result in head-of-line blocking for the message queue.

At the same time, early receivers get many IWANT requests, increasing their workload.

c. Message prioritization for slow senders​

A slow peer often struggles with a backlog of messages in the outgoing message queue(s) for mesh members. Any new message transmission at this stage (especially the locally published messages) gets delayed. Adaptive message-forwarding can help such peers prioritize traffic to minimize latency for essential message transfers.

For instance, any GossipSub peer will likely receive every message from multiple senders, leading to redundant transmissions [11]. Implementing efficient strategies (only for slow senders) like lazy sending and prioritizing locally published messages/IWANT replies over already queued messages can help minimize outgoing message queue sizes and optimize bandwidth for essential message transfers.

A peer can identify itself as a slow peer by using any bandwidth estimation approach or simply setting an outgoing message queue threshold for all mesh members.

Eliminating/deprioritizing some messages can lower a peer's score, but it also earns the peer an overall better score by achieving some early message transfers.
For instance, sending many near-first messages can only save a peer from a deficit penalty. On the other hand, sending only one message (assuming MeshMessageDeliveriesThreshold defaults to 1) as the first delivered message can add to the accumulative peer score.

2. Mitigating transport issues​

Congestion avoidance algorithms used in various TCP versions directly influence achievable throughput and message transfer time as maximum unacknowledged in-flight bytes are based on the congestion window $(C_{wnd})$ size.

Rapid adaptation of $C_{wnd}$ to the available network conditions can help lower message dissemination latency.

Therefore, selecting a more suitable TCP variant like BBR, which is known for its ability to dynamically adjust the congestion window based on network conditions, can significantly enhance GossipSub's performance.

At the same time, parameters like receive window scaling and initial $C_{wnd}$ also impact message transfer time, but these are usually OS-specific system-wide choices.

One possible solution is to raise $C_{wnd}$ by exchanging data over the newly established connection. This data may involve useful details like peer exchange information and gossip to build initial trust, or GossipSub can use some dummy data to raise $C_{wnd}$ to a reasonable level.

It's important to understand that some TCP variants reset $C_{wnd}$ after specific periods of inactivity [12]. This can lead to a decline in TCP's performance for applications that generate traffic after intervals long enough to trigger the resetting of the congestion window.

Implementing straightforward measures like transport-level ping-pong messages can effectively mitigate this problem [13].

The limitations faced with $C_{wnd}$ scaling also impact some performance optimizations in GossipSub. For instance, floodpublishing is an optimization relying on additional transmissions by the publisher to minimize message dissemination latency.

However, a small $C_{wnd}$ value in (new/cold) TCP connections established with floodpublish peers significantly increases message transmission time [4]. Usually, these peers also receive the same message from other sources during this time, wasting the publisher's bandwidth.

The same is the case with IWANT replies.

Maintaining a bigger mesh (with warm TCP connections) and relaying to $D$ peers can be a better alternative to this problem.

3. Eliminating redundant transmissions​

For every received packet, a peer makes roughly $D$ transmissions to contribute its fair share to the spread of messages. However, the fact that many recipients had already received the message (from some other peer) makes this message propagation inefficient.

Although the $D$-spread is attributed to quicker dissemination and resilience against non-conforming peers, many potential solutions can still minimize redundant transmissions while preserving the resilience of GossipSub.

These solutions, ranging from probabilistic to more knowledgeful elimination of messages from the outgoing message queue, not only address the issue of redundancy but also provide an opportunity for bandwidth optimization, especially for resource-constrained peers.

For instance, an IDONTWANT message, a key component of GossipSub (v1.2) [10], can significantly reduce redundant transmissions.

It allows any node to notify its mesh members that it has already received a message, thereby preventing them from resending the same message. This functionality is useful when a node receives a message larger than a specified threshold.

In such cases, the node promptly informs its mesh peers about the successful reception of the message by sending IDONTWANT messages.

It's important to note that an IDONTWANT message is essentially an IHAVE message, but with a crucial difference, i.e., IHAVEs are only transmitted during the heartbeat intervals, whereas IDONTWANTs are sent immediately after receiving a large message.

This prompt notification helps curtail redundant large message transmissions without compromising the GossipSub resilience.

However, the use of IDONTWANT messages alone has an inherent limitation. For instance, a peer can only send an IDONTWANT after receiving the complete message.

A large message transmission consumes significant time. For example, transmitting a 1MB message at 100 Mbps bandwidth may consume 80 to several hundred milliseconds (depending upon $C_{wnd}$ and latency).

As a result, other mesh members may also start transmitting the same message during this interval. A few potential solutions include:

a. Staggering with IDONTWANT messages​

As previously discussed, staggering can significantly reduce network-wide message dissemination latency. This is primarily due to the relatively smaller store-and-forward delays that are inherent in this approach.

Using both staggering and IDONTWANT messages can further enhance efficiency by reducing redundant transmissions. This is because a node only saturates its bandwidth for a small subset of mesh peers, leading to early transmissions and prompt IDONTWANT message notifications to the mesh members.

It is worth highlighting that staggering can be implemented in various ways.

For example, it can be applied to peers (peer staggering) where a node sequentially relays the same message to all peers one by one.

Alternatively, a node can send a different message to every peer (message staggering or rotational sending), allowing IDONTWANTs for other messages to arrive during this time. The message staggering approach is beneficial when several messages are introduced to the network within a short interval of time.

As the peers in staggered sending are sequentially covered (with a faster speed due to bandwidth concentration), this leads to another problem.

The early covered peers send IHAVE (during their heartbeat intervals) for the messages they have received. IHAVE announcements for newly received large messages trigger IWANTs from nodes (including those already receiving the same message), leading to an additional workload for early receivers [14].

Potential solutions to mitigate these problems include:

1. Defering IHAVE announcements for large messages.

Deferring IHAVE announcements can indirectly prioritize message transmission to the mesh peers over IWANT replies. However, deciding on a suitable deferred interval is crucial for optimal performance. One possible solution is to generate IHAVEs only after the message is relayed to all the mesh peers.

2. Defering IWANT requests for messages that are currently being received.

This requires prior knowledge of msgIDs for the messages under reception. Knowing the message length is also essential in deciding a suitable defer interval to handle situations where a sender starts sending a message and never completes the transmission.

3. Not issuing IWANT for a message if at least $K$ peers have transmitted IDONTWANT for the same message (as this indicates that these peers will eventually relay this message).

However, this approach can inadvertently empower a group of non-conforming mesh peers to send IDONTWANT for a message and never complete message transmission. A delayed IWANT, along with negative peer scoring, can remedy this problem.

b. IMReceiving message​

A peer can issue an IDONTWANT only after it has received the entire message. However, a large message transmission may take several hundred milliseconds to complete. During this time, many other mesh members may start relaying the same message.

Therefore, the probability of simultaneously receiving the same message from multiple senders increases with the message size, significantly compromising the effectiveness of IDONTWANT messages.

Sending a short preamble (containing msgID and length) before the message transmission can provide valuable information about the message. If a receiver is already receiving the same message from another sender, the receiver can request to defer this transmission by sending a brief IMReceiving message.

An IDONTWANT from the receiver will indicate successful message reception. Otherwise, the waiting sender can initiate transmission after a specific wait interval.

However, waiting for IMReceiving after sending the preamble can delay the message transmission. On the other hand, proceeding with message transfer (after sending the preamble) leads to another problem: it is difficult to cancel ongoing message transmission after receiving IMReceiving for the same message.

To streamline this process, a peer can immediately send an IMReceiving message (for every received preamble), urging other mesh peers to defer sending the same message [15, 16].

The other peers can send this message if IDONTWANT is not received from the receiver during the wait interval. This approach can boost IDONTWANT benefits by considering ongoing transmissions for large messages.

While IMReceiving messages can bring about substantial improvements in terms of latency and bandwidth utilization, it's crucial to be aware of the potential risks.

A malicious user can exploit this approach to disrupt message transmission either by never completing a message or by intentionally sending a message at an extremely slow rate to numerous peers.

This could ultimately result in network-wide slow message propagation.

However, carefully calibrating the deferring interval (based on message size) and negative peer scoring can help mitigate these risks.

c. IDONTWANT message with reduced forwarding​

It is common for slow peers to pile up outgoing message queues, especially for large message transfers. This results in a significant queuing delay for outgoing messages. Reduced message forwarding can help decrease the workload of slower peers.

On receiving a message longer than the specified threshold, a slow peer can relay it to only $K \in D$ peers and send an IDONTWANT message to all the peers in $D$.

In this arrangement, the IDONTWANT message serves an additional purpose: to promptly announce data availability, reinforcing redundancy in the presence of adversaries.

When a peer receives an IDONTWANT for an unseen message, it learns about the new message and can request it by sending an IWANT request without waiting for the heartbeat (gossip) interval. As a result, a significantly smaller number of transmissions is sufficient for propagating the message to the entire network.

This approach conserves peer bandwidth by minimizing redundant transmissions while ensuring GossipSub resilience at the cost of one RTT (for missing peers).

Interestingly, curtailing queuing delays can also help lower network-wide message dissemination latency (for huge messages).

However, finding an appropriate value for $K$ is crucial for optimal performance. A smaller $K$ saves peer bandwidth, while a larger $K$ achieves quicker spread until outgoing message queues pile up. Setting $K = D_{low}$ can be one option.

It is worth mentioning that such behavior may negatively impact peer scoring (by missing message delivery rewards from $D-K$ peers). However, a minimized workload enables early message dissemination to the remaining peers. These early transmissions and randomized $K$ set selection can help achieve an overall better peer score.

4. Message prioritization​

Despite the standardized specifications of the GossipSub protocol, the message forwarding mechanisms can significantly impact network-wide message dissemination latency and bandwidth utilization.

It is worth mentioning that every node is responsible for transmitting different types of packets, including control messages, locally published messages, messages received from mesh members, IWANT replies, etc.

As long as traffic volume is lower than the available data rate, the message forwarding mechanisms yield similar results due to negligible queuing delays.

However, when the traffic volume increases and exceeds the available peer bandwidth (even for short traffic bursts), the outgoing message queue(s) sizes rise, potentially impacting the network's performance.

In this scenario, FIFO-based traffic forwarding can lead to locally published messages being placed at the end of the outgoing message queue, introducing a queuing delay proportional to the queue size. The same applies to other delay-sensitive messages like IDONTWANT, PRUNE, etc.

On the other hand, the segregation of traffic into priority and non-priority queues can potentially starve low-priority messages. One possible solution is to use weighted queues for a fair spread of messages.

Message prioritization can be a powerful tool to ensure that important messages reach their intended recipients on time and allow for customizable message handling.

For example, staggering between peers and messages can be better managed by using priority queues. However, it is important to note that message prioritization also introduces additional complexity to the system, necessitating sophisticated algorithms for better message handling.

5. Maximizing benefits from IWANT messages​

During heartbeat intervals, GossipSub nodes transmit IHAVE messages (carrying IDs of seen messages) to the peers not included in the full-message mesh. These peers can use IWANT messages to request any missing messages. A budget counter ensures these messages never exceed a specified threshold during each heartbeat interval.

The IHAVE/IWANT messages are a crucial tool in maintaining network connectivity. They bridge the information gap between nearby and far-off peers, ensuring that information can be disseminated to peers outside the mesh. This function is essential in protecting against network partitions and indirectly aids in safeguarding against Sybil and eclipse attacks.

However, it is essential to understand that high transmission times for large messages require careful due diligence when using IWANT messages for reasons not limited to:

1. A large message reception may take several hundred milliseconds to complete. During this time, an IHAVE message announcing the same message ID will trigger an IWANT request.

2. A peer can send IWANT requests for the same message to multiple nodes, leading to simultaneous transmissions of the same message.

3. Replying to (potentially many) IWANT requests can delay the transmission of the same message to mesh peers, resulting in lower peer scores and slower message propagation.

A few possible solutions to mitigate this problem may include:

1. Issuing IHAVE announcements only after the message is delivered to many mesh peers.

2. Allocating a volume-based budget to service IWANT requests during each heartbeat interval.

3. Deferring IWANT requests for messages that are currently being received.

4. Deferring IWANT requests if at least $K$ IDONTWANTs are received for the same message.

5. A large message transmission can yield high $C_{wnd}$; preferring such peers during mesh maintenance can be helpful.

Summary​

This study investigates the pressing issue of considerable fluctuations and rises in network-wide dissemination times for large messages.

We delve into multiple factors, such as increased message transmit times, store-and-forward delays, congestion avoidance mechanisms, and prioritization between messages, to establish a comprehensive understanding of the problem.

The study also explores the performance of optimization efforts like floodpublishing, IHAVE/IWANT messages, and message forwarding strategies in the wake of large message transmissions.

A key finding is that most congestion avoidance algorithms lack optimization for peer-to-peer networks. Coupling this constraint with increased message transmission times results in notable store-and-forward delays accumulating at each hop.

Furthermore, the probabilistic message-forwarding nature of GossipSub further exacerbates the situation by utilizing a considerable share of available bandwidth on redundant transmissions.

Therefore, approaches focused on eliminating redundant transmissions (IDONTWANT, IMReceiving, lazy sending, etc.) can prove helpful. At the same time, strategies aimed at reducing store-and-forward delays (fragmentation, staggering, prioritization, etc.) can prove beneficial.

It is worth mentioning that many of the strategies suggested in this post are ideas at different stages. Some of these have already been explored and discussed to some extent [5, 17, 18]. We are nearing the completion of a comprehensive performance evaluation of these approaches and will soon share the results of our findings.

Please feel free to join the discussion and leave feedback regarding this post in the VAC forum.

References​

[1] EIP-4844: Shard Blob Transactions. Retrieved from https://eips.ethereum.org/EIPS/eip-4844

[2] Message Propagation Times With Waku-RLN. Retrieved from https://docs.waku.org/research/research-and-studies/message-propagation/

[3] Lenient Flood Publishing. Retrieved from https://github.com/libp2p/rust-libp2p/pull/3666

[4] Disable Flood Publishing. Retrieved from https://github.com/sigp/lighthouse/pull/4383

[5] GossipSub for Big Messages. Retrieved from https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw

[6] GossipSub: Lazy Sending. Retrieved from https://github.com/status-im/nim-libp2p/issues/850

[7] GossipSub: Limit Flood Publishing. Retrieved from https://github.com/vacp2p/nim-libp2p/pull/911

[8] GossipSub: Lazy Prefix Detection. Retrieved from https://github.com/vacp2p/nim-libp2p/issues/859

[9] Potential Gossip Improvement List for EIP4844. Retrieved from https://hackmd.io/@gRwfloEASH6NWWS_KJxFGQ/B18wdnNDh

[10] GossipSub Specifications v1.2: IDONTWANT Message. Retrieved from https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52

[11] Number of Duplicate Messages in Ethereum’s GossipSub Network. Retrieved from https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921

[12] TCP Congestion Control: Re-starting Idle Connections. Retrieved from https://datatracker.ietf.org/doc/html/rfc2581#section-4.1

[13] PING/PONG Control Messages. Retrieved from https://github.com/libp2p/specs/pull/558

[14] IHAVE/IWANT Message Impact. Retrieved from https://github.com/vacp2p/nim-libp2p/issues/1101

[15] Large Message Handling IDONTWANT + IMReceiving Messages. Retrieved from https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281

[16] IDONTWANT Message Impact. Retrieved from https://forum.vac.dev/t/idontwant-message-impact/283

[17] IWANT Message Impact. Retrieved from https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366

[18] IDONTWANT Message Performance. Retrieved from https://vac.dev/rlog/gsub-idontwant-perf-eval/

Overview​

IDONTWANT messages are introduced to curtail redundant transmissions without compromising resilience. Cutting down on duplicates can potentially render two significant advantages:

1. Reducing bandwidth requirements

2. Reducing message dissemination time (latency)

For IDONTWANTs to be effective, they must be received and processed by the sender before the sender starts relaying the respective message.

Duplicates investigation reveals that the average time difference between the first message arrival and the first duplicate arrival is higher than the average round trip time in Ethereum's GossipSub network.

This allows for timely IDONTWANT reception and canceling of many duplicate transmissions, showing a potential for a significant drop in bandwidth utilization. On the other hand, lowering message dissemination time is only possible by minimizing queuing delays at busy peers.

Experiments​

We conducted a series of experiments with different arrangements (changing heartbeat_interval and message size) to precisely identify the impact of IDONTWANT messages on bandwidth utilization and message dissemination time.

The experiments are performed on nim-libp2p using the shadow simulator. The peer bandwidth and link latency are uniformly set between 50-150 Mbps and 40-130 milliseconds in five stages.

In all experiments, ten messages are transmitted to the network, i.e., ten peers (publishers) are selected as the message transmitters. Every publisher transmits exactly one message, and inter-packet spacing (delay) is set to four seconds for each published message. For a fair assessment, we ensure that the publishers are uniformly selected from each bandwidth class.

At the start of each experiment, two additional messages are transmitted to increase the TCP $C_{wnd}$. These messages are not included in latency computations.

The simulation details are presented in the table below.

Findings​

We use bandwidth utilization and latency as evaluation metrics. Bandwidth utilization represents total network-wide traffic (including gossip and other control messages). Latency refers to network-wide message dissemination time. The total number of IWANT requests and the number of message transmissions saved by IDONTWANT messages are also presented for detailed insights.

Experiments reveal that IDONTWANT messages yield a noticeable (up to 21%) drop in bandwidth utilization. A higher drop is seen with a higher heartbeat interval. Interestingly, a relatively low bandwidth reduction (12-20%) is seen for 1MB messages, compared to 500KB messages (18-21%).

This is because downloading a large message may consume several hundred milliseconds. During this time, a receiver will likely generate multiple IWANT requests for the same message, increasing bandwidth utilization.

Moreover, a peer can generate IDONTWANTs only after it has finished downloading the message. A longer download time will result in simultaneous reception of the same message from other mesh members.

These IWANT requests mainly overwhelm early message receivers, which can negatively impact message dissemination time on some occasions. Therefore, a similar message dissemination time is seen with and without IDONTWANT messages.

Similar results are seen on our large-scale deployment runs (running Waku nodes in Kubernetes).

Please feel free to join the discussion and leave feedback regarding this post in the VAC forum.

References​

• GossipSub Specifications v1.2
• GossipSub v1.2: IDONTWANT Control Message
• Number Duplicate Messages in Ethereum’s Gossipsub Network
• IWANT Messages Impact on Latency
• Large Message Handling (IDONTWANT + IMReceiving)
• IDONTWANT Message Impact Before/After Message Validation
• GossipSub for Big Messages
• Regression Test Results: nim-libp2p

Introduction​

The set of interactive protocols form a class of protocols that consist of communication between two parties: the Prover and the Verifier. The Prover tries to convince the Verifier of a given claim. For example, the Prover may want to convince the Verifier that she owns a specific Unspent Transaction Output (UTXO); that is, the Prover possesses the ability to spend the UTXO. In many instances, there is information that the Prover does not wish to reveal to the Verifier. In our example, it is critical that the Prover does not provide the Verifier with the spending key associated with her UTXO. In addition to the Prover's claim and secret data, there is additional data, public parameters, that the claimed statement is expressed in terms of. The public parameters can be thought of as the basis for all similar claims.

In an interactive protocol, the Prover and the Verifier are in active communication. Specifically, the Prover and the Verifier exchange messages so that the Verifier can validate the Prover's claim. However, this communication is not practical for many applications. It is necessary that any party can verify the Prover's claim in decentralized systems. It is impractical for the Prover to be in active communication with a large number of verifying parties. Instead, it is desirable for the Prover to generate a proof on their own that can convince any party. To achieve this, it is necessary for the Prover to generate the Verifier's messages in such a way that the Prover cannot manipulate the Verifier's messages for her benefit. The Fiat-Shamir heuristic 1 is used for this purpose. Even though much of our discussion will focus on $\Sigma$-protocols, the Fiat-Shamir heuristic is not limited to $\Sigma$-protocols. The Fiat-Shamir heuristic has been applied to zk-SNARKs, but the security in this setting has been the subject of discussion and research in recent years. Block et al. 2 provide the first formal analysis of Fiat-Shamir heuristic in zk-SNARKs.

Sigma Protocols​

A $\Sigma$-protocol is a family of interactive protocols that consists of three publicly transmitted messages between the Prover and the Verifier. In particular, the protocol has the following framework:

These three messages form the protocol's transcript: $(\mathsf{commitment}, \mathsf{challenge}, \mathsf{response})$. The Verifier uses all three of these messages to validate the Prover's original claim. The Verifier's challenge should be selected uniform random from all possible challenges. Based on this selection, a dishonest Prover can only convince the Verifier with a negligible probability.

The Schnorr Protocol​

The Schnorr protocol 3 is usually the first $\Sigma$-protocol that one studies. Additionally, the Schnorr protocol can be used as an efficient signature scheme. The Schnorr protocol provides a framework that enables the Prover to convince the Verifier that: for group elements $g$ and $X$, the Prover knows the power to raise $g$ to obtain $X$. Specifically, the Prover possesses some integer $x$ so that $X = g^x$. Cryptographic resources may use either multiplicative or additive notation for groups; we will use multiplicative notation. Briefly, the element $g$ being combined with itself in multiplicative notation is $g \cdot g = g^2$, while in additive notation it is $g + g = 2g$. We assume that our group is of prime order $p$, and is sufficiently large to satisfy the discrete logarithm assumption.

The Schnorr protocol proceeds as follows:

Chaum-Pedersen protocol​

A tuple of group elements $(U,V,W)$ is a DH-triple if and only if there exists some $x \in \mathbb{Z}_p$ so that $V = g^x$ and $W = U^x$. The Chaum-Pedersen protocol provides a framework that enables a Prover to convince a Verifier that she possesses such a $x$ for a claimed DH-triple $(U,V,W)$. The Chaum-Pedersen protocol proceeds as follows:

Hash Functions​

Cryptographic hash functions serve as the backbone to the Fiat-Shamir heuristic. A hash function, $\mathsf{Hash}$, is a special function that takes in an arbitrary binary string and outputs a binary string of a predetermined fixed length. Specifically, $\mathsf{Hash} : \{0,1\}^* \rightarrow \{0,1\}^n$.

The security of cryptographic hash functions will rely on certain tasks being computationally infeasible. A task is computationally infeasible provided that there is no deterministic algorithm that can conclude the task in polynomial-time.

A cryptographic hash function satisfies the following properties:

• Succinct: The hash function should be easy to compute; the hash $\mathsf{Hash}({\bf{b}})$ can be efficiently computed for any binary string ${\bf{b}}$.
• Preimage Resistance: It should be computationally infeasible to work backwards given the output of a hash function. Let ${\bf{y}}$ be a binary string of length $n$. It should be 'impossible' to find some binary string ${\bf{x}}$ so that ${\bf{y}} = \mathsf{Hash}({\bf{x}})$.
• Collision Resistance: It should be difficult to find two strings that hash to the same value. It should be computationally infeasible to find two binary strings ${\bf{x}_1}$ and ${\bf{x}_2}$ so that $\mathsf{Hash}({\bf{x}_1}) = \mathsf{Hash}({\bf{x}_2}).$

A related class of functions is one-way functions. A one-way function satisfies the first two conditions of a cryptographic hash function (succinct and preimage resistance). All cryptographic hash functions are a one-way functions. However, one-way functions do not necessarily satisfy collision-resistance. We will simply refer to cryptographic hash functions as hash functions for the rest of this blog. Commonly used hash functions include SHA-256 5, Keccak 6, and Poseidon 7.

The Fiat-Shamir heuristic​

The Fiat-Shamir heuristic is the technique used to convert an interactive protocol to a noninteractive protocol. This is done by replacing each of the Verifier's messages with a hashed value. Specifically, the Prover generates the Verifier's message by evaluating the hash function $\mathsf{Hash}$ with the concatenation of all public values that appear in the protocol thus far. If $m_0, \dots, m_t$ denote the public values in the protocol thus far, then the Verifier's message is computed as $m_{t+1} := \mathsf{Hash}(m_0|| \cdots || m_t)$.

Since $\mathsf{Hash}$ can be efficiently computed, and the messages $m_0, \dots, m_t$ are public, then any verifying party can compute $m_{t+1}$. Critically, since $\mathsf{Hash}$ is preimage resistant and collision resistant, the Prover cannot manipulate her choices of the messages $m_0,\dots, m_t$ to influence the message $m_{t+1}$. Hence, verifying parties can trust that $m_{t+1}$ is sufficiently random with respect to the preceding messages.

There are two variants of the Fiat-Shamir heuristic: weak and strong. The weak variant uses all of the publicly traded messages in computing the Verifier's messages but does not include the public parameters. However, in the strong variant all of the publicly traded messages and public parameters are used to compute the Verifier's messages. We will provide a discussion on issues that can arise from using the weak Fiat-Shamir heuristic.

Schnorr Protocol with the strong Fiat-Shamir​

When the strong Fiat-Shamir heuristic is applied to the Schnorr protocol, the message $c = \mathsf{Hash}(g||X||T)$. This choice of $c$ provides security since it should be computationally infeasible to find collisions for the outputs of $\mathsf{Hash}$. Thus, $c$ fixes the group elements $g$, $X$ and $T$.

The elements that would be omitted in the hash by applying weak Fiat-Shamir heuristic are $g$ and $X$.

Chaum-Pedersen Protocol with the strong Fiat-Shamir​

The message $c = Hash(g||U||V||W||T||S)$ when the Prover applies the strong Fiat-Shamir heuristic to the Chaum-Pedersen protocol. The properties of $\mathsf{Hash}$ fixes the generator $g$ and the Prover's statement $(U,V,W)$.