- Remove all crates that are not linked with `autoprecompiles`,
`openvm`, `constraint-solver`
- Remove tests and artifacts linked with removed crates
- Adjust CI
Nightly has been failing for a while. We should fix that, but by moving
ignored the tests after the benchmarks, we should at least get nightly
benchmark results again (and test our most important guest programs).
I think this is also a good idea in general, because we often manually
run nightly to get the latest benchmarks on a particular branch. This
way, we get results earlier.
I started a manual run on this branch:
https://github.com/powdr-labs/powdr/actions/runs/17212171321
When installing `cargo-openvm` for the openvm-reth related tests, the
installation pulls fresh versions of the dependencies that are not
pinned. Some of these dependencies only work with Rust >= 1.88.
I made a [test PR in
powdr-labs/openvm](https://github.com/powdr-labs/openvm/pull/37) which
the changes here point to to get the `cargo-openvm` crate.
We can merge the PR above which would force us to change powdr as well,
but for now it may be enough to merge this PR to get the test passing,
and then change the openvm hash in powdr together with the Rust version
in a follow-up PR.
Currently we get a file not found error for input of
`effectiveness_plot.py`, because the `cargo prove` with no APC case
doesn't generate `apc_candidates.json` as input for the Python script
anymore after #3109.
This PR fixes this error and now `run_guest_benches.sh` runs on the
server, though I'm not sure if all nightly tests will be fixed that
aren't run on CI yet.
Now `JsonExport` and `plot_effectiveness.py` both live in APC crate.
Didn't move other scripts, which rely on `metrics.json` and I think it's
specific to OVM?
---------
Co-authored-by: Georg Wiese <georgwiese@gmail.com>
Larger PR, but after this basically everything that can be made agnostic
is agnostic.
`PowdrConfig` is split in two, as the `autoprecompile` crate does not
care about the arithmetization method. This could be cleaned up a bit
though.
Need to merge
https://github.com/powdr-labs/openvm-reth-benchmark/pull/24/ first and
update the hashes.
TODO:
- [x] merge #3033
- [x] fix reth
- [x] remove POWDR_OPCODE constant in apc and pass it instead
This PR runs a few benchmarks nightly, saving the results to a separate
repo: `bench-results`.
Idea is to get something going on and improve reporting later.
Currently, it runs `keccak` and `reth`.
This adds a job to our nightly run that runs all tests in the
powdr-openvm crate, including recursion.
Future ideas:
- Split the common nightly into 2. I tried to do this but couldn't get
it to work initially. TBD
- Since we don't run Halo2 anymore, we can actually run recursion for a
lot of the tests. Next I would suggest to run the guest_recursion test
enabled here in a PR test actually that also uses the self hosted runner
- Similarly to above, add a larger Keccak test to PR tests
Based on commit 1dbe4db
- Split into two crates, lib and cli
- upgrade stwo, marked one stwo test `should_panic` @ShuangWu121
- various clippy and fmt fixes linked to the rust version update
- bring all rust versions to 2025-05-14. CI still installs other
versions for openvm which uses them internally. The stable rust version
we test on is bumped to 1.85
- remove `examples` and related tests, which test the powdr crate on the
previous version of powdr (since it uses another nightly). Happy to
discuss this if it's important @leonardoalt
Instead of setting `mtime` to the committed date, which @lvella noted is
error-prone, this PR does the following so that we can utilize the build
cache:
- check out the commit the build cache was built for
- set mtime to the time the cache was built
- check out the commit we are running the tests on
This hopefully has the effect that all modified files will have the
current time as mtime and will trigger re-builds correctly
This PR increases the number of `test_slow` PR test bins from 8 to 17,
thereby strictly reducing the time spent on `test_slow`.
We have three types of runners:
- `warp-ubuntu-2404-x64-8x` for: build, run_examples, bench. I think
this is a paid server: https://www.warpbuild.com/, which @leonardoalt
last changed in #2187, so I have a question: why don't we use the GitHub
Workflow free ones? Besides, I think we can parallelize `run_examples`,
which is a bottleneck of 38 minutes currently run single threaded, so
there can be some immediate time improvement if we run it on multiple
runner instances. Is this server charged by the number of runners or the
number of minutes?
- `ubuntu-24.04` for: test_quick, test_estark_polygon, test_slow.
- `ubuntu-22.04` for: udeps.
**BEFORE Optimization**
<img width="346" alt="Screen Shot 2025-03-24 at 17 25 11"
src="https://github.com/user-attachments/assets/b197eca3-9994-4113-8f4d-1e7b106b064c"
/>
**AFTER Optimization**
See run time of this PR.
**Future Optimization**
In #2580, I'm working on creating 14 bins with 4 threads each because we
have 55 tests in total and each can run on a separate thread. This would
reduce the `test_slow` run time to that of the longest test, which is
~10 minutes.
This requires computing the bin-thread assignment because `nextest` only
supports the hash method for partitions, which isn't ideal in this case.
It seems WarpBuild removed the ability to overwrite a cache with the
same key.
But since they allow for removing a cache, this fix simply removes the
old one before saving the new one.
Please make sure the build cache test run was successful before merging.
I'm not really sure if this is the right fix. For the riscv targets, we
are using nightly-2024-08-01 - is that on purpose? Maybe we should keep
the examples on 2024-08-01 as well?
Extracts the common code in single step processor and block processor
into a unified "processor".
---------
Co-authored-by: Georg Wiese <georgwiese@gmail.com>
