Now that witgen depends on backend type, `powdr pil` requires a backend
type to run through witgen. It only runs till optimized pil if
`--prove-with` is not provided.
To match the description, this PR defaults backend to Mock if
`--prove-with` is not provided, so `powdr pil` can still run till
witgen. To prove with any backend, `--prove-with` is still required.
After discussing with @Schaeff, we formulated a plan:
1. Currently, our pipeline is `compile pilopt tune pilopt witgen0 prove0
witgen1 prove1 finalize`, where `prove0 witgen1 prove1 finalize` is
backend aware.
2. In this PR, `tune pilopt witgen0 prove0 witgen1 prove1 finalize` are
all backend aware, meaning that a backend argument is required for these
steps to run.
I made some specific implementation decisions:
3. In this PR, the `tune` step is a `BackendFactory` API, while all
other APIs remain the same places as before. For example, `witgen0` is
backend aware because it requires a `backend_tuned_pil` to run. However,
`witgen0` only runs differently because different backend tuned pil's
are provided by different backend, not *directly* because of the
different backend (it has the same algorithm regardless of the backend).
Therefore, while `witgen0` requires a backend to run, I didn't put it
under `BackendFactory`. Design choices like this should ensure maximum
backward compatibility.
Summary of all other changes on the high level, mostly fixing pipeline
test and CI incompatibilities:
CLI changes:
4. `powdr-rs` requires a backend for witgen.
5. `powdr pil` only runs witgen if a backend is provided.
Test crates changes:
6. For tests that don't require a backend but require witgen, use the
mock backend to run witgen.
7. For tests that require a backend and also require witgen but runs
witgen before setting the backend, modify the test to set the backend
before running witgen.
- Replace the rule which sets all degrees to that of main if it is
static and use a CLI flag instead: `--degrees-mode`
- Adjust tests to run mostly on vadcop
- Tweak tests to reduce wasted rows
Refactor of the linker to enable creating statements both in the source
and in the target of a link. Instead of creating a vector of statements,
we create a (name=>namespace) map.
The linker mode is exposed in the CLI, and currently defaults to the
native lookup/permutation statements.
Bus linking can be activated the following way:
```console
cargo run pil test_data/asm/vm_to_block_unique_interface.asm --force --linker-mode bus
```
TODO:
- [x] absolute names
This PR does two things:
- make `--export-csv` not export fixed columns, so it can be directly
used as external witness
- skip witness generation when all witness columns are provided
externally
Unfortunately, cargo weirdness attacks again. Even with
`estark-starky-simd = ["starky?/avx512"]` enabling the feature only if
`starky` is being used, cargo still selects `starky` for compilation.
Then, because `starky` itself doesn't check the architecture being used
at compile time to enable AVX and only checks the features, compilation
fails for https://github.com/powdr-labs/powdr-template with `simd` if
AVX is not available.
Therefore, we can't group both p3 and starky's simd features in the same
backend `simd` feature, and need one simd feature per prover. This is
what this PR does. The result has been tested with the template repo and
starky is correctly *not* selected for compilation both with and without
the `simd` feature of the template crate.
Currently the users end up always compiling starky as well. This removes
that.
I think the CI tests should not change here because we run it with
`--all-features`.
I also made p3 default in the cli and cli-rs crates.
The idea of this PR is to add the most commonly used workflow to the
outer facing `powdr` crate so that users have quick access to the prover
functionalities for Rust programs.
The changes to `powdr/src/lib.rs` wrap those common functionalities
(compiling Rust to powdr-asm, compiling powdr-asm to PIL, witgen, proof
generation with continuations) into simple function calls and hide the
Pipeline type a bit by using the new `Session` trait.
The end result is that a user can do the following:
```rust
use powdr::Session;
use std::path::Path;
fn main() {
env_logger::init();
let some_data = vec![1, 2, 3, 4, 5];
let out_path = Path::new("powdr-target");
let mut session = Session::new("./guest", out_path).write(1, &some_data);
// Dry run to test execution
session.run();
// Compute the proof
session.prove();
}
```
TODO:
- [x] Decide on the abstraction
- [x] Try to remove some clones from the continuations setup
- [x] Make sure the `prove` function re-uses artifacts if available
(like setup), and re-runs if needed
- [x] Should we have a separate `setup` call? I feel like it should be
automatic with the item above.
Depends on https://github.com/powdr-labs/powdr/pull/1923
Now that we use nightly, we can more easily enable AVX, Neon etc in
backends that support it.
Not sure why fmt added some formatting changes.
---------
Co-authored-by: Lucas Clemente Vella <lvella@powdrlabs.com>
The KoalaBear prime ($2^{31} - 2^{24} + 1$) is very similar to the
BabyBear prime ($2^{31} - 2^{27} + 1$), but allows for a more efficient
S-Box in Poseidon2 ($x^3$ instead of $x^7$). It should be slightly
faster to evaluate and significantly faster to prove recursively. For
some reason, I [saw a much more significant
advantage](https://gist.github.com/georgwiese/211d14c860c16cc4e1fbde7dc374af35)
when running the Plonky3 examples.
One downside of BabyBear is that it supports smaller traces, e.g. up to
$2^{23}$ rows with a degree bound of 3. For example,
`test_data/pil/fibonacci.pil` fails for larger instances.
This PR supports KoalaBear end-to-end:
```bash
cargo run -r --bin powdr-rs compile riscv/tests/riscv_data/keccak -o output --field kb
cargo run -r --features plonky3,halo2 pil output/keccak.asm -o output -f --field kb --prove-with plonky3
```
### **Progress on Integrating `Stwo`**
This PR makes incremental progress towards integrating the `stwo`
prover, refer to issue #1833 . It includes the following changes:
- **Add `stwo` feature to `Cargo.toml`:**
The `Cargo.toml` files for the `cli`, `pipeline`, and `backend`
components have been updated to include the `stwo` feature.
- **Implement `StwoProverFactory`:**
A skeleton structure for `StwoProverFactory` has been introduced. This
structure implements the `BackendFactory` trait.
- **Implement `StwoProver`:**
Similarly, a skeleton implementation of `StwoProver` has been created,
implementing the `Backend` trait. .
### **Commands to run:**
```bash
# Run the project with the `stwo` feature
cargo run --features stwo pil test_data/pil/fibonacci.pil -o output -f --field m31 --prove-with stwo
---------
Co-authored-by: Thibaut Schaeffer <schaeffer.thibaut@gmail.com>
Co-authored-by: Georg Wiese <georgwiese@gmail.com>
Adds Plonky3's implementation of the Mersenne-31 field.
To test:
```
cargo run pil test_data/pil/fibonacci.pil -o output -f --field m31
```
The implementation is basically the same wrapper as for BabyBear, so I
moved the code to a macro and used it for both fields.
As suggested by @lvella .
I'm not too sure about this actually. I agree that compiling Rust ->
powdr-asm should have a separate pipeline somewhere, but before we could
do Rust -> proof in a single command, and with this we can't anymore.
- Do we expose the `prove` command and everything else here as well?
That'd be a lot of duplicated code.
- Do we force the user to first run `powdr-rs compile <rust_proj>` then
`powdr pil/prove ...`? That's potentially worse UX than right now.
- Do we keep all the Rust stuff in the `cli` crate?
Similar to #1193, but in here I am just interested in having it working
end-to-end, at least for a few cases, so that everybody can try it and
build upon.
<!--
Please follow this protocol when creating or reviewing PRs in this
repository:
- Leave the PR as draft until review is required.
- When reviewing a PR, every reviewer should assign themselves as soon
as they
start, so that other reviewers know the PR is covered. You should not be
discouraged from reviewing a PR with assignees, but you will know it is
not
strictly needed.
- Unless the PR is very small, help the reviewers by not making forced
pushes, so
that GitHub properly tracks what has been changed since the last review;
use
"merge" instead of "rebase". It can be squashed after approval.
- Once the comments have been addressed, explicitly let the reviewer
know the PR
is ready again.
-->
---------
Co-authored-by: Leo <leo@powdrlabs.com>
- Refactor `CoProcessors` into a `Runtime` with submachines and syscalls
- Calls into prover functionality and submachines done via `ecall`
instead of "call symbol replacement"
- Call `poseidon_gl` machine with riscv registers by using the stack
(~50% speedup in many_chunks benchmark)
- New `riscv-syscalls` crate keeps the list of syscalls and related
constants
# Context
Now powdr can allow handle log in cli option.
Example
```bash
powdr --log DEBUG pil test_data/asm/book/hello_world.asm --field bn254 --force --inputs 0
```
--log value is LevelFilter
```rust
pub enum LevelFilter {
/// A level lower than all log levels.
Off,
/// Corresponds to the `Error` log level.
Error,
/// Corresponds to the `Warn` log level.
Warn,
/// Corresponds to the `Info` log level.
Info,
/// Corresponds to the `Debug` log level.
Debug,
/// Corresponds to the `Trace` log level.
Trace,
}
```
Fixes#164
Remove dangling function in macro
Fix variable name in schema_update.rs
Read the version number at runtime instread of compile time
Fix version_number parsing and write new schema after it
Add schemars feature `preserve_order`
Change unwrap for expect in schema_update.rs
Moved schema binary in its own crate
Graceful error management in powdr-schema instead of panic
Better graceful error handling in the pipeline and the cli
Fix clippy
Put the magic number at the start of the serialized data
Revert magic at the start of the file
Moved the SerializedAnalyzed to the schemas crate
Changed serialized.rs to analyzed.rs
Changed the magic number to be the ASCII `powdr`
Fix pipeline name from file name with suffix
Variable naming
Replace `splitted` with `split`
Co-authored-by: Leo <leo@ethereum.org>
Fix clippy
Add test for serde of PIL
Changed the optimized PIL file extension to .pilo
Changed function names to reflect the operation on pil object
