This adds backends comparison in data extractor. It performs
comparison on a fixed list (CPU, GPU, HPU) for 64 bits precision
ciphertext as displayed in tfhe-rs public documentation.
SVG table generation is automated via the documentation benchmark
workflow.
This is done to avoid confusion for operations that might have the same between layer. For example, 'bitand' operation have the same name for shortint and integer layers
When doing regression generation, one can provide a global parameters set name pattern to filter head branch benchmark results.
This fixes the issue encountered when there are more than one parameters' set used to benchmark an operation, for example, in core_crypto or shortint tfhe-rs layer.
After running performances regression benchmarks, a performance
changes checking is executed. It will fetch results data with an
external tool then it will look for anomaly in changes.
Finally it will produce a report as an issue comment with any
anomaly display in a Markdown array. A folded section of the
report message contains all the results from the benchmark.
Note that a fully custom benchmark triggered from an issue comment
would not generate a report. In addition HPU performance
regression benchmark is not supported yet.
Regression benchmarks are meant to be run in pull-request. They
can be launched in two flavors:
* issue comment: using command like "/bench --backend cpu"
* adding a label: `bench-perfs-cpu` or `bench-perfs-gpu`
Benchmark definitions are written in TOML and located at
ci/regression.toml.
While not exhaustive, it can be easily modified by reading the
embbeded documentation.
"/bench" commands are parsed by a Python script located at
ci/perf_regression.py. This script produces output files that
contains cargo commands and a shell script generating custom
environment variables. The Python script and generated files are
meant to be used only by the workflow
benchmark_perf_regression.yml.
Switch n3-H100-SXM5x8 to US-1 as CANADA is out of stock on this
instance.
Also L40 instances fallback on n3-RTX-A6000x1 to mitigate
resource shortages issues.
Several network errors occurred while trying to install git-lfs
from within backward compatibility tests workflow. Having git-lfs
installed directly in the Amazon Machine Image fix this issue.
From SageMath point of view some tfhe-rs parameters set are
equivalent. We deduplicate those by storing their name in the tag
field. Grouping them that way we decrease analysis time
dramatically.
