github/powdr
1
1
Fork 0
mirror of https://github.com/powdr-labs/powdr.git synced 2026-01-13 21:47:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,335 Commits 439 Branches 5 Tags
seqz
Commit Graph

19 Commits

Author SHA1 Message Date
chriseth
c9d39e2a7c Revert "Merge branch 'relax_bounds' into seqz" 
This reverts commit 606018e40d, reversing
changes made to 892cdd4f6c.
 2025-09-22 12:25:13 +00:00
Georg Wiese
d1b50d1c87 Nightly: Run more benchmarks (#3286) 
Changes:
- For Keccak and Matmul, we only ran the benchmark with 0 and 100 APCs.
I changed it to run with 0, 3, 10, 30. I think more does not make sense,
because the effectiveness plots
([matmul](https://github.com/powdr-labs/bench-results/blob/gh-pages/results/2025-09-15-0630/matmul/100apc/effectiveness.png),
[keccak](https://github.com/powdr-labs/bench-results/blob/gh-pages/results/2025-09-15-0630/keccak/100apc/effectiveness.png))
do not suggest that there are more basic blocks worth accelerating.
- For reth, I also added more steps.
- Named experiments more consistently (`apc000`, `apc003`, ...)
- Added existing benchmarks for SHA, Pairing, and U256
- Increased the size of the U256 benchmark to get more meaningful
results

Manual nightly run:
https://github.com/powdr-labs/powdr/actions/runs/17772555964
Which leads to the following bench results:

https://github.com/powdr-labs/bench-results/tree/gh-pages/results/2025-09-16-2213

Note that the `test_apc` job duration goes from from 2:48h -> 4:34h.
 2025-09-17 09:40:25 +00:00
ShuangWu121
71c3c3db71 k256 updated, ecc tests with larger input (#3217) 
k256 is updated with the newer openvm-guest-hint and we can test ecc
with larger input 50

---------

Co-authored-by: Georg Wiese <georgwiese@gmail.com>
 2025-08-26 14:23:58 +00:00
ShuangWu121
6c0d481b50 Add ecrecover guest (#3160) 
Manual precompile: total_trace_cells = 15_368_081

ecrecover with PowdrAffinePoint:
APC=50: total_trace_cells = 228_239_394

---------

Co-authored-by: Georg Wiese <georgwiese@gmail.com>
 2025-08-22 23:16:30 +00:00
Georg Wiese
3422658efd Add ECC nightly test (#3207) 
Built on #3203

This PR:
- Modifies the ECC tests to continue doing MSMs for a number of times
(passed as an input)
- Adds this as a benchmark to the nightly test

I got these results:

| filename | num_segments | app_proof_cells | app_proof_cols |
total_proof_time_ms | app_proof_time_ms | leaf_proof_time_ms |
inner_recursion_proof_time_ms |

|---------------------------------|--------------|-----------------|----------------|---------------------|-------------------|--------------------|-------------------------------|
| affine-hint-apc000/metrics.json | 4 | 3852743848 | 6044 | 69327 |
35904 | 12783 | 20640 |
| affine-hint-apc003/metrics.json | 3 | 3140411518 | 6219 | 50886 |
30551 | 9791 | 10544 |
| affine-hint-apc010/metrics.json | 2 | 1618931796 | 33742 | 51152 |
16193 | 25074 | 9885 |
| affine-hint-apc030/metrics.json | 1 | 1091031082 | 40436 | 40165 |
11166 | 23824 | 5175 |
| affine-hint-apc100/metrics.json | 1 | 944216106 | 54662 | 55193 |
10174 | 39800 | 5219 |
| manual/metrics.json | 1 | 1041144324 | 5618 | 21215 | 10090 | 6236 |
4889 |
| projective-apc000/metrics.json | 9 | 9622555316 | 12991 | 156526 |
88927 | 28593 | 39006 |
| projective-apc003/metrics.json | 2 | 3195273488 | 31100 | 64314 |
31006 | 23439 | 9869 |
| projective-apc010/metrics.json | 2 | 3644260624 | 33710 | 70673 |
35933 | 24873 | 9867 |
| projective-apc030/metrics.json | 1 | 1931874516 | 56279 | 62528 |
18945 | 38336 | 5247 |
| projective-apc100/metrics.json | 1 | 1806846164 | 70086 | 65856 |
18181 | 42434 | 5241 |

It shows that:
- Using `k256` out of the box without APCs takes 156s
- Switching to affine representation + using hints for the base field
inverse takes 69s
- Accelerating the resulting code using APCs takes 40s
- Using OpenVMs manual precompile takes 21s

The Hint + APC version is similar to the manual precompile in terms of
app prove time, but due to the much larger number of columns, recursion
becomes expensive.
 2025-08-21 14:45:18 +00:00
Georg Wiese
f4345b5304 Improve basic metrics summary (#3203) 
Adds `app_proof_cols` and `total_proof_time_ms` columns to the result
CSV.
 2025-08-21 14:09:12 +00:00
Georg Wiese
81ae444b4c Refactor run_guest_benches.sh (#3198) 
I wanted to refactor the `run_guest_benches.sh` to make it a bit more
flexible. With this, it should be possible to test the ECC guest easily
(#3118), which actually has 3 guests (manual, out-of-the-box software,
hint-optimized software).

I also brought the matmul test back.

The file structure is not exactly the same as before, but similar (and
more organized):

https://github.com/powdr-labs/bench-results/tree/gh-pages/results/2025-08-20-1538

(which is from this run:
https://github.com/powdr-labs/powdr/actions/runs/17099625118/job/48492387082)

---------

Co-authored-by: chriseth <chris@ethereum.org>
 2025-08-21 10:59:04 +00:00
Steve Wang
f22419f7af Fix nightly tests (#3142) 
Currently we get a file not found error for input of
`effectiveness_plot.py`, because the `cargo prove` with no APC case
doesn't generate `apc_candidates.json` as input for the Python script
anymore after #3109.

This PR fixes this error and now `run_guest_benches.sh` runs on the
server, though I'm not sure if all nightly tests will be fixed that
aren't run on CI yet.
 2025-08-05 12:09:28 +00:00
Steve Wang
1ded39b9f6 Refactor JsonExport and effectiveness plot (#3116) 
Now `JsonExport` and `plot_effectiveness.py` both live in APC crate.

Didn't move other scripts, which rely on `metrics.json` and I think it's
specific to OVM?

---------

Co-authored-by: Georg Wiese <georgwiese@gmail.com>
 2025-07-31 14:38:41 +00:00
Georg Wiese
6329f0ea65 Remove opcode (#3091) 
We used to need the Opcode of the APC in order to hard-code it in the PC
lookup. However, for a while, we have removed PC lookups entirely,
hard-coding the PC in the execution bridge receive instead.

This removes the necessity to store the opcode. It was only used as an
ID of the APC. But for this, we can use the PC instead. This has the
added benefit that we can more easily relate it to the original program.
 2025-07-25 14:33:43 +00:00
Georg Wiese
5806f36c3a Better effectiveness plot (#3038) 
Changes the effectiveness plot to this (example: Reth benchmark):
<img width="3323" height="1768" alt="reth"
src="https://github.com/user-attachments/assets/78f4eb2c-bd9b-4d83-b1c3-e2ea20286270"
/>

I think it is more intuitive that the [previous
version](https://github.com/powdr-labs/bench-results/blob/gh-pages/results/2025-07-10-0155/reth/effectiveness.png_stacked_histogram.png).

Also prints this summary of the largest blocks:
```
Top 10 Basic Blocks by Trace Cells:
 Opcode Trace Cells  Effectiveness  Instructions  Width Before  Width After
   4372      784.9M       3.358650            12           796          237
   4361      358.3M       4.085859            13           809          198
   8079      248.9M       3.692737            10           661          179
   4391      158.6M       2.861345            21          1362          476
   4359      130.3M       3.560976             5           292           82
   4390      128.9M       2.847418            37          2426          852
   9105      105.6M       5.720029           116          7968         1393
   4368       99.4M       6.938776            24          1700          245
   9104       97.2M       4.342214           104          7334         1689
   4385       87.8M       2.671587            11           724          271
```
 2025-07-11 11:11:28 +00:00
Leandro Pacheco
6c9903d0a6 Fix effectiveness plot (#3037) 
fix the script for the new apc candidates json format
 2025-07-10 19:33:50 +00:00
Georg Wiese
f0a0e3a804 Benchmarks: Generate effectiveness plots (#3024) 
Builds on #3023 and
https://github.com/powdr-labs/openvm-reth-benchmark/pull/23

Worked on this manually triggered nightly run:

https://github.com/powdr-labs/powdr/actions/runs/16182186333/job/45680988436

See these results:

https://github.com/powdr-labs/bench-results/tree/gh-pages/results/2025-07-10-0155
 2025-07-10 09:29:52 +00:00
Georg Wiese
1d9635f709 Add plot_effectiveness.py (#3023) 
Creates this plot from an `apc_candidates.json`:
![effectiveness
png_stacked_histogram](https://github.com/user-attachments/assets/cf419b81-0609-453e-89c5-5d1207b51385)

To reproduce the above plot, see the example in #3022 and run:
`python openvm/scripts/plot_effectiveness.py
cli-openvm/apcs/apc_candidates.json`
 2025-07-09 19:52:52 +00:00
Georg Wiese
18a549c599 basic_metrics.py: Report OpenVM precompile ratio (#3004) 
Adds a heuristic to separate "normal" CPU instructions (which we can
hope to accelerate with APCs) from OpenVM precompiles.

Example:
```
python openvm/scripts/basic_metrics.py ../bench-results/results/2025-07-07-0624/reth/*.json
                                                 filename  num_segments  app_proof_cells  app_proof_time_ms  app_execute_time_ms  app_trace_gen_time_ms  leaf_proof_time_ms  inner_recursion_proof_time_ms  normal_instruction_cells  openvm_precompile_cells  powdr_cells  normal_instruction_ratio  openvm_precompile_ratio  powdr_ratio  powdr_rows
../bench-results/results/2025-07-07-0624/reth/100apc.json             3      15062260062             163682                23767                  64627               89913                          12532                4982516224               7190029662   2889714176                  0.330795                 0.477354     0.191851    16826880
 ../bench-results/results/2025-07-07-0624/reth/noapc.json             6      19633824920             217899                24907                  55865               82750                          30175               11659639296               7974185624            0                  0.593855                 0.406145     0.000000           0
```

Note that `normal_instruction_cells + openvm_precompile_cells` adds to
`non_powdr_cells`
[here](https://github.com/powdr-labs/bench-results/blob/gh-pages/results/2025-07-07-0624/reth/basic_metrics.csv).
 2025-07-07 23:00:12 +00:00
Georg Wiese
33c5b45943 plot_trace_cells.py: Only count app proofs (#3003) 
- Extracted a function to load the metrics file, which already separates
the different proof stages
- Use it in `openvm/scripts/plot_trace_cells.py`
 2025-07-07 15:23:16 +00:00
Leandro Pacheco
770e948c64 nightly benchmarks (#2982) 
This PR runs a few benchmarks nightly, saving the results to a separate
repo: `bench-results`.
Idea is to get something going on and improve reporting later.
Currently, it runs `keccak` and `reth`.
 2025-07-04 12:09:45 +00:00
Leandro Pacheco
d3b6d40c7c script for extracting basic metrics from the metrics json file (#2979) 2025-07-03 06:55:03 +00:00
Georg Wiese
7ab3a637a5 Add plot_trace_cells.py (#2943) 
Extracted from #2922
 2025-06-24 17:32:19 +00:00