AtHeartEngineer/specs
1
0
Fork 0
mirror of https://github.com/vacp2p/specs.git synced 2026-01-07 22:44:07 -05:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #478 from libp2p/marco/perf

Browse Source 
Introduce `perf` spec
This commit is contained in:
Marco Munizaga
2023-05-31 10:05:14 -07:00
committed by GitHub
parent 5393c21c4f 4f9ae25ca9
commit 917be26c1d
1 changed files with 131 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

131
perf/perf.md Normal file
View File

@@ -0,0 +1,131 @@
# Perf <!-- omit in toc -->
| Lifecycle Stage | Maturity | Status | Latest Revision |
| --------------- | ------------- | ------ | --------------- |
| 1A | Working Draft | Active | r0, 2022-11-16 |
Authors: [@marcopolo]
Interest Group: [@marcopolo], [@mxinden], [@marten-seemann]
[@marcopolo]: https://github.com/marcopolo
[@mxinden]: https://github.com/mxinden
[@marten-seemann]: https://github.com/marten-seemann
# Table of Contents <!-- omit in toc -->
- [Context](#context)
- [Protocol](#protocol)
- [Benchmarks](#benchmarks)
- [Single connection throughput](#single-connection-throughput)
- [Handshakes per second](#handshakes-per-second)
- [Security Considerations](#security-considerations)
- [Prior Art](#prior-art)
# Context
The `perf` protocol represents a standard benchmarking protocol that we can use
to talk about performance within and across libp2p implementations. This lets us
analyze peformance, guide improvements, and protect against regressions.
# Protocol
The `/perf/1.0.0` protocol (from here on referred to as simply `perf`) is a
client driven set of benchmarks. To not reinvent the wheel, this perf protocol
is almost identical to Nick Bank's [_QUIC Performance_
Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3)
but adapted to libp2p.
The protocol first performs an upload of a client-chosen amount of bytes. Once
that upload has finished, the server sends back as many bytes as the client
requested.
The bytes themselves should be a predetermined arbitrary set of bytes. Zero is
fine, but so is random bytes (as long as it's not a different set of random
bytes, because then you may be limited by how fast you can generate random
bytes).
The protocol is as a follows:
Client:
1. Open a libp2p stream to the server.
2. Tell the server how many bytes we want the server to send us as a single
big-endian uint64 number. Zero is a valid number, so is the max uint64 value.
3. Write some amount of data to the stream.
Zero is a valid amount.
4. Close the write side of our stream.
5. Read from the read side of the stream. This
should be the same number of bytes as we told the server in step 2.
Server, on handling a new `perf` stream:
1. Read the big-endian uint64 number. This is how many bytes we'll send back in step 3.
2. Read from the stream until we get an `EOF` (client's write side was closed).
3. Send the number of bytes defined in step 1 back to the client. This MUST NOT be run
concurrently with step 2.
5. Close the stream.
# Benchmarks
The above protocol is flexible enough to run the following benchmarks and more.
The exact specifics of the benchmark (e.g. how much data to download or for how
long) are left up to the benchmark implementation. Consider these rough
guidelines for how to run one of these benchmarks.
Other benchmarks can be run with the same protocol. The following benchmarks
have immediate usefulness, but other benchmarks can be added as we find them
useful. Consult the [_QUIC Performance_
Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance#section-2.3)
for some other benchmarks (called _scenarios_ in the document).
## Single connection throughput
For an upload test, the client sets the the server response size to 0 bytes, writes
some amount of data and closes the stream.
For a download test, the client sets the server response size to N bytes, and
closes the write side of the data.
The measurements are gathered and reported by the client by measuring how many
bytes were transferred by the total time it took from stream open to stream
close.
A timer based variant is also possible where we see how much data a client can
upload or download within a specific time. For upload it's the same as before
and the client closes the stream after the timer ends. For download, the client
should request a response size of max uint64, then close the stream after the
timer ends.
## Handshakes per second
This benchmark measures connection setup efficiency. A transport that takes many
RTTs will perform worse here than one that takes fewer.
To run this benchmark:
1. Set up N clients
2. Each client opens K connections/s to a single server
3. once a connection is established, the client closes it and establishes
another one.
Handshakes per second are calculated by taking the total number of connections
successfully established and divide it by the time period of the test.
# Security Considerations
Since this protocol lets clients ask servers to do significant work, it
SHOULD NOT be enabled by default in any implementation. Users are advised not to
enable this on publicly reachable nodes.
Authentacting by Peer ID could mitigate the security concern by only allowing
trusted clients to use the protocol. Support for this is left to the implementation.
# Prior Art
As mentioned above, this document is inspired by Nick Bank's: [QUIC Performance Internet-Draft](https://datatracker.ietf.org/doc/html/draft-banks-quic-performance)
[iperf](https://iperf.fr)
[@mxinden's libp2p perf](https://github.com/mxinden/libp2p-perf)
[@marten-seemann's libp2p perf test](https://github.com/marten-seemann/libp2p-perf-test/)
[@vyzo's libp2p perf test](https://github.com/vyzo/libp2p-perf-test/)