mirror of
https://github.com/vacp2p/vac.dev.git
synced 2026-01-06 21:34:17 -05:00
128 lines
6.7 KiB
Plaintext
128 lines
6.7 KiB
Plaintext
---
|
||
title: 'Libp2p GossipSub IDONTWANT Message Performance Impact'
|
||
|
||
|
||
date: 2024-10-28 12:00:00
|
||
authors: farooq
|
||
published: true
|
||
slug: gsub-idontwant-perf-eval
|
||
|
||
categories: research
|
||
discuss: https://forum.vac.dev/t/libp2p-gossipsub-idontwant-message-performance-impact/374
|
||
|
||
|
||
toc_min_heading_level: 2
|
||
toc_max_heading_level: 5
|
||
---
|
||
|
||
This post provides quick insights into the IDONTWANT message performance and highlights minor tweaks that can further contribute to performance gains.
|
||
{/* truncate */}
|
||
|
||
## Overview
|
||
|
||
[IDONTWANT](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md?plain=1#L52) messages are introduced to curtail redundant transmissions without compromising resilience.
|
||
Cutting down on duplicates can potentially render two significant advantages:
|
||
|
||
|
||
1. Reducing bandwidth requirements
|
||
|
||
2. Reducing message dissemination time (latency)
|
||
|
||
|
||
For IDONTWANTs to be effective, they must be received and processed by the sender before the sender starts relaying the respective message.
|
||
|
||
[Duplicates investigation](https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921#arrival-time-of-duplicates-9) reveals that
|
||
the average time difference between the first message arrival and the first duplicate arrival is higher than the average round trip time in Ethereum's GossipSub network.
|
||
|
||
This allows for timely IDONTWANT reception and canceling of many duplicate transmissions,
|
||
showing a potential for a significant drop in bandwidth utilization.
|
||
On the other hand, lowering message dissemination time is only possible by minimizing queuing delays at busy peers.
|
||
|
||
## Experiments
|
||
|
||
We conducted a series of experiments with different arrangements (changing heartbeat_interval and message size)
|
||
to precisely identify the impact of IDONTWANT messages on bandwidth utilization and message dissemination time.
|
||
|
||
The experiments are performed on nim-libp2p using the [shadow simulator](https://github.com/vacp2p/dst-gossipsub-test-node/pull/4).
|
||
The peer bandwidth and link latency are uniformly set between 50-150 Mbps and 40-130 milliseconds in five stages.
|
||
|
||
In all experiments, ten messages are transmitted to the network, i.e.,
|
||
ten peers (publishers) are selected as the message transmitters.
|
||
Every publisher transmits exactly one message,
|
||
and inter-packet spacing (delay) is set to four seconds for each published message.
|
||
For a fair assessment, we ensure that the publishers are uniformly selected from each bandwidth class.
|
||
|
||
At the start of each experiment, two additional messages are transmitted to increase the TCP $C_{wnd}$.
|
||
These messages are not included in latency computations.
|
||
|
||
The simulation details are presented in the table below.
|
||
|
||
|
||
| **Parameter** | **Value** | **Parameter** | **Value** |
|
||
| ------------- | --------- | ------------- | --------- |
|
||
| Peers | 2000 | Publishers | 10 |
|
||
| Peer bandwidth | 50-150 Mbps | Link latency | 40-130 ms |
|
||
| Message size | 1KB, 50KB, 500KB, 1MB | $D$ | 8 |
|
||
| Heartbeat interval | 700ms, 1000ms, 1500ms | $D_{low}$ | 6 |
|
||
| FloodPublish | False | $D_{high}$ | 12 |
|
||
| Gossip factor | 0.05 | Muxer | yamux |
|
||
|
||
|
||
## Findings
|
||
|
||
We use bandwidth utilization and latency as evaluation metrics.
|
||
Bandwidth utilization represents total network-wide traffic (including gossip and other control messages).
|
||
Latency refers to network-wide message dissemination time.
|
||
The total number of IWANT requests and the number of message transmissions saved by IDONTWANT messages are also presented for detailed insights.
|
||
|
||
|
||
Experiments reveal that IDONTWANT messages yield a noticeable (up to 21%) drop in bandwidth utilization.
|
||
A higher drop is seen with a higher heartbeat interval.
|
||
Interestingly, a relatively low bandwidth reduction (12-20%) is seen for 1MB messages,
|
||
compared to 500KB messages (18-21%).
|
||
|
||
|
||
|  |  |  |
|
||
|-----------------------------|-----------------------------|-----------------------------|
|
||
|
||
|
||
This is because downloading a large message may consume several hundred milliseconds.
|
||
During this time, a receiver will likely
|
||
[generate multiple IWANT requests](https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366)
|
||
for the same message, increasing bandwidth utilization.
|
||
|
||
Moreover, a peer can generate
|
||
[IDONTWANTs only after it has finished downloading the message](https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281).
|
||
A longer download time will result in simultaneous reception of the same message from other mesh members.
|
||
|
||
|
||
|  |  |
|
||
|-----------------------------|-----------------------------|
|
||
|
||
|
||
These IWANT requests mainly overwhelm early message receivers,
|
||
which can negatively impact message dissemination time on some occasions.
|
||
Therefore, a similar message dissemination time is seen with and without IDONTWANT messages.
|
||
|
||
|
||
|  |  |  |
|
||
|-----------------------------|-----------------------------|-----------------------------|
|
||
|
||
Similar results are seen on our large-scale deployment runs
|
||
([running Waku nodes in Kubernetes](https://zealous-polka-dc7.notion.site/Nim-libp2p-v1-5-0-regression-testing-August-2024-25edba733c704ccaa411919555c5db1a)).
|
||
|
||
Please feel free to join the discussion and leave feedback regarding this post in the
|
||
[VAC forum](https://forum.vac.dev/t/libp2p-gossipsub-idontwant-message-performance-impact/374).
|
||
|
||
|
||
## References
|
||
|
||
- [GossipSub Specifications v1.2](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.2.md)
|
||
- [GossipSub v1.2: IDONTWANT Control Message](https://github.com/libp2p/specs/pull/548)
|
||
- [Number Duplicate Messages in Ethereum’s Gossipsub Network](https://ethresear.ch/t/number-duplicate-messages-in-ethereums-gossipsub-network/19921)
|
||
- [IWANT Messages Impact on Latency ](https://forum.vac.dev/t/iwant-messages-may-have-negative-impact-on-message-dissemination-latency-for-large-messages/366)
|
||
- [Large Message Handling (IDONTWANT + IMReceiving)](https://forum.vac.dev/t/large-message-handling-idontwant-imreceiving/281)
|
||
- [IDONTWANT Message Impact Before/After Message Validation](https://forum.vac.dev/t/idontwant-message-impact/283)
|
||
- [GossipSub for Big Messages](https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw#2)
|
||
- [Regression Test Results: nim-libp2p](https://zealous-polka-dc7.notion.site/Nim-libp2p-v1-5-0-regression-testing-August-2024-25edba733c704ccaa411919555c5db1a)
|