The previous version of link_merging.asm is not working correctly since
the `operation sub<1>` implementation is wrong. With this PR, `operation
sub<1>` implementation is fixed.
I started this in order to test #1603. However, due to #1604, currently
only one machine has a bus so far. This should be fixed as part of
#1604.
Then, what we would expect is that both machines have the same final
accumulator values, assuming challenges are shared across machines.
Fixes#1496
Also, a step towards #1572
This PR implements the steps needed in `CompositeBackend` to implement
dynamic VADCOP.
In summary:
- If a machines size (a.k.a. "degree") is set to `None`, fixed columns
are computed in all powers of too in some hard-coded range. This fixes
#1572. As a result, machines with a size set to `None` are available in
multiple sizes. If the size is explicitly set by the user, the machine
is only available in that one size.
- Note that the ASM linker still sets the size of machines without a
size. So, currently, this can only happen when coming from PIL directly.
- `CompositeBackend` instantiates a new backend for each machine *and
size*:
- The verification key contains a key for each machine and size.
- When proving, it it uses the backend of whatever size the witness has.
The size chosen is also stored in the proof.
- When verifying, the verification key of the reported size is used.
- Witness generation currently chooses the largest available size. This
will change in a future PR.
This is an example:
```
$ cargo run pil test_data/pil/vm_to_block_dynamic_length.pil -o output -f --field bn254 --prove-with halo2-mock-composite
...
== Proving machine: main (size 256)
==> Machine proof of 256 rows (0 bytes) computed in 209.101166ms
== Proving machine: main__rom (size 256)
==> Machine proof of 256 rows (0 bytes) computed in 226.87175ms
== Proving machine: main_arith (size 1024)
==> Machine proof of 1024 rows (0 bytes) computed in 432.807583ms
```
Related to the issue of implementing basic bus (#1497), I have
implemented basic bus together with an example
(`permutation_via_bus.asm`) as specified inside the issue.
Currently, `test_data/std/bus_permutation_via_challenges.asm` works as
intended (To make it sound, stage(1) witness columns need to be exposed
publicly and verifier needs to check such as `out_z1 + out_z2 = 0`) We
can now check using RUST_LOG=trace and adding the final z1 and z2 is
equal to 0.
However, `test_data/std/bus_permutation_via_challenges_ext.asm` is not
working correctly as intended. This will be fixed with the following
commits.
With this PR:
- Simplifies the `build_machine_pil` function by simply re-parsing a
subset if the PIL. This has the extra advantage that all IDs are
consecutive, which enables supporting STARK backends as well!
- With this approach, it is necessary to have all definitions available,
which makes it necessary to merge `std` namespaces. Those are detected
as namespaces that don't define any column.
- Also, some backends expect at least one witness column and constraint,
so I added a dummy column & constraint for now.
- With this PR, we have everything done for static VadCop! So I added a
test & removed the `should_panic` in an existing test.
This PR splits from the main Trait implementation PR #1450 to simplify
the review process.
It includes only the parsing of the traits (not impls) and some
functionality necessary for the code to compile.
---------
Co-authored-by: chriseth <chris@ethereum.org>
Implements #1055 for the Poseidon machines. Pulled out of #1508.
Specifically, this PR adds a new `PoseidonGLMemory` machine which
receives 2 memory points and then:
- Reads 24 32-Bit words and packs them into 12 field elements
- Computes the Poseidon permutation (just like `PoseidonGL`)
- For each of the 4 output field elements, it:
- Invokes the `SplitGL` machine to get the canonical `u64`
representation
- Writes the 8 32-Bit words to memory at the provided memory pointer
The read and write memory regions can even overlap! 🎉
This should simplify our RISC-V machine, as the syscall already expects
two memory pointers. We can simply pass it to the machine directly.
I started doing that in #1533, but I think it makes sense to wait until
#1443 is merged.
To test:
```
cargo run -r pil test_data/std/poseidon_gl_test.asm -o output -f --export-csv --prove-with estark-starky
```
I recommend reviewing the diff between
`std/machines/hash/poseidon_gl.asm` and
`std/machines/hash/poseidon_gl_memory.asm`
### Discussion
The overhead of the memory read / write is quite high (18 extra witness
columns, see [this
comment](40bdca4368/std/machines/hash/poseidon_gl_memory.asm (L13-L23)),
mostly because we now need to have the input available in all rows
(which previously was only the case for the outputs). If we had offsets
other than 0 and 1, this could be avoided. Doing 24 parallel memory
reads in the first row would *not* help, because we'd have to add 24
witness columns (instead of 2 now) to store the result of the memory
operation.
A few more notes:
- With Vadcop, 18 extra witness columns in a secondary machine is *a
lot* better than introducing more registers (either "regular" registers
or assignment registers) in the main machine
- As mentioned
[here](40bdca4368/std/machines/hash/poseidon_gl_memory.asm (L111-L113)),
we could get rid of two permutations if either:
- We were able to express explicitly that we want to call at most one
operation in the current row, or
- We had an optimizer that would be smart enough to batch the memory
reads and writes.
- We could also have just 1 read or write at a time (instead of 2), but
we'd have to increase the block size from 31 to 32 and the
implementation would be more complicated.
- We could also store the full final state of the Poseidon permutation,
instead of just the first 4 elements. This would need 8 more witness
columns to make the entire output available in all rows. Then, one could
use the machine to implement a Poseidon sponge, instead of.
- Looking at the bootloader, maybe it makes sense to pass 3 input
pointers instead of 1: One for the first 4 elements, one for the next 4,
and one for the capacity (often just a constant). For example, when
computing a Merkle root, you'd pass pointers for the two children hashes
and a pointer to the capacity constant.
Fixes#1494
- use cbor for witness and constant files (moving polygon serialisation
to the relevant backend)
- add `degree` field on `Symbol`, inherited from the namespace degree
- have each machine in witgen operate over its own degree
- fail in the backend if we have many degrees
Includes #1511, but also removes the declarations of the challenges in
the `permutation` and `lookup` modules.
With these changes, we never declare a column or challenge outside a
machine namespace. This greatly simplifies #1470.
---------
Co-authored-by: schaeff <thibaut@schaeff.fr>
Co-authored-by: chriseth <chris@ethereum.org>
Merge compatible `link`s into a single permutation/lookup.
We only consider merging links from different instructions, as a single
instruction can be active at a time.
Links with next references are ignored due to a limitation in witgen
(left a TODO so its easily fixed upon witgen support)
This PR builds on top of #1393.
It mainly modifies the grammar by changing the way SelectedExpressions
are declared, to allow blocks to be empty.
---------
Co-authored-by: chriseth <chris@ethereum.org>
Fixes#1493
Allow passing machines as argument when instantiating submachines, as
in:
```
use std::machines::binary::Binary;
machine Main with degree: 262144 {
reg pc[@pc];
reg X[<=];
reg Y[<=];
reg Z[<=];
reg A;
Binary binary;
WithArg sub(binary);
instr and X, Y -> Z ~ binary.and;
instr or X, Y -> Z ~ binary.or;
instr xor X, Y -> Z ~ binary.xor;
...
}
machine WithArg(bin: Binary) {
reg pc[@pc];
reg X[<=];
reg Y[<=];
reg Z[<=];
reg A;
reg B;
instr and X, Y -> Z ~ bin.and;
instr or X, Y -> Z ~ bin.or;
instr xor X, Y -> Z ~ bin.xor;
...
}
```
This PR solves #1374
There are 3 examples included in the PR:
1. `lookup_via_challenges_ext_simple.asm` is the most basic example
which implements {a} in {b} using extension field without using
selectors
2. `lookup_via_challenges_ext_no_selector.asm` implements {a1, a2} in
{b1, b2} using extension field without using selectors
3. `lookup_via_challenges_ext.asm` is a more complex example than others
which implements {a1, a2, a3} in {b1, b2, b3} using extension field
(which also handles tuples using Reed-Solomon fingerprinting). It also
use different lhs and rhs selectors.
---------
Co-authored-by: Georg Wiese <georgwiese@gmail.com>
This PR adds a new memory machine to the standard library that also
supports a `mstore_bootloader` operation: It's like a normal `mstore`,
but the first access to every memory cell *has to* come from this
operation.
As a follow-up, we'll be able to use it in the RISC-V machine (#1462).
I think this is best reviewed by reviewing the diff to the normal memory
machine and comparing it to [the code we have currently inlined in the
RISC-V
machine](abbe26618f/riscv/src/code_gen.rs (L500-L553)):
```diff
3,5c3,7
< // A read/write memory, similar to that of Polygon:
< // https://github.com/0xPolygonHermez/zkevm-proverjs/blob/main/pil/mem.pil
< machine Memory with
---
> /// This machine is a slightly extended version of std::machines::memory::Memory,
> /// where in addition to mstore, there is an mstore_bootloader operation. It behaves
> /// just like mstore, except that the first access to each memory cell must come
> /// from the mstore_bootloader operation.
> machine MemoryWithBootloaderWrite with
7c9
< operation_id: m_is_write,
---
> operation_id: operation_id,
13a16
> operation mstore_bootloader<2> m_addr, m_step, m_value ->;
29a33
> col witness m_is_bootloader_write;
30a35,36
> std::utils::force_bool(m_is_bootloader_write);
> col operation_id = m_is_write + 2 * m_is_bootloader_write;
35a42
> (1 - is_mem_op) * m_is_bootloader_write = 0;
37,39c44,45
< // If the next line is a not a write and we have an address change,
< // then the value is zero.
< (1 - m_is_write') * m_change * m_value' = 0;
---
> // The first operation of a new address has to be a bootloader write
> m_change * (1 - m_is_bootloader_write') = 0;
41,42c47,52
< // change has to be 1 in the last row, so that a first read on row zero is constrained to return 0
< (1 - m_change) * LAST = 0;
---
> // m_change has to be 1 in the last row, so that the above constraint is triggered.
> // An exception to this when the last address is -1, which is only possible if there is
> // no memory operation in the entire chunk (because addresses are 32 bit unsigned).
> // This exception is necessary so that there can be valid assignment in this case.
> pol m_change_or_no_memory_operations = (1 - m_change) * (m_addr + 1);
> LAST * m_change_or_no_memory_operations = 0;
46c56
< (1 - m_is_write') * (1 - m_change) * (m_value' - m_value) = 0;
---
> (1 - m_is_write' - m_is_bootloader_write') * (1 - m_change) * (m_value' - m_value) = 0;
➜ powdr git:(memory-with-bootloader-write) ✗
```
Allow VM instructions to use the `link` notation, unifying the way
machines are linked from VMs and block machines.
Previous syntax for "external instructions" not allowed anymore, and
should use the new `link` syntax.
Makes the permutation argument sound on the Goldilocks field by
evaluating polynomials on the extension field introduced in #1310.
I also used the new `Constr::Permutation` variant!
A few test cases (also tested in CI):
#### No extension field
`cargo run pil test_data/std/permutation_via_challenges.asm -o output -f
--field bn254 --prove-with halo2-mock`
This still works and produces the same output as before, thanks to the
PIL evaluator removing multiplications by 0 etc:
```
col witness stage(1) z;
(std::protocols::permutation::is_first * (main.z - 1)) = 0;
((((1 - main.first_four) * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.b1) + main.b2)) - 1)) + 1) * main.z') = (((main.first_four * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.a1) + main.a2)) - 1)) + 1) * main.z);
```
#### With extension field
`cargo run pil test_data/std/permutation_via_challenges_ext.asm -o
output -f --field bn254 --prove-with halo2-mock`
The constraints are significantly more complex but seem correct to me:
```
col witness stage(1) z1;
col witness stage(1) z2;
(std::protocols::permutation::is_first * (main.z1 - 1)) = 0;
(std::protocols::permutation::is_first * main.z2) = 0;
(((((1 - main.first_four) * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.b1) + main.b2)) - 1)) + 1) * main.z1') + ((7 * ((1 - main.first_four) * (std::protocols::permutation::beta2 - (std::protocols::permutation::alpha2 * main.b1)))) * main.z2')) = ((((main.first_four * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.a1) + main.a2)) - 1)) + 1) * main.z1) + ((7 * (main.first_four * (std::protocols::permutation::beta2 - (std::protocols::permutation::alpha2 * main.a1)))) * main.z2));
((((1 - main.first_four) * (std::protocols::permutation::beta2 - (std::protocols::permutation::alpha2 * main.b1))) * main.z1') + ((((1 - main.first_four) * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.b1) + main.b2)) - 1)) + 1) * main.z2')) = (((main.first_four * (std::protocols::permutation::beta2 - (std::protocols::permutation::alpha2 * main.a1))) * main.z1) + (((main.first_four * ((std::protocols::permutation::beta1 - ((std::protocols::permutation::alpha1 * main.a1) + main.a2)) - 1)) + 1) * main.z2));
```
#### On Goldilocks
Running the first example on GL fails, because using the permutation
argument without the extension field would not be sound. The second
example works, but because we don't support challenges on GL yet, it
doesn't actually run the second-phase witness generation.
---------
Co-authored-by: chriseth <chris@ethereum.org>
This simulates one approach we could go for when moving registers to
memory. The memory machine remains completely unchanged, but the step is
increased by more than 1 in each step of the main machine. This way,
from the point of view of memory, all the memory operations happen at
different time steps, which allows for:
- Reading from the same address twice
- Writing to the same address that we read from (which from the point of
view of memory should happen *after* the read)
The only downside I see with this approach is that this makes the
differences of time steps between memory accesses bigger: Before it was
at most the degree, now it is some small constant times the degree (in
this example 3). The way the memory machine is currently built, the
difference can be at most $2^{32} - 1$, so I think this is fine in
practice. E.g., for a degree $2^{30}$ machine we could do up to 4
parallel reads / writes.
*Cherry-picked b1a07bd9a7 from #1380, and
extended on it.*
Fixes#1382.
With this PR, a lookup like `selector { byte_lower + 256 * byte_upper }
in { <some other machine> }` works, even if the range constraints on
`byte_lower` and `byte_upper` are not "global". For example, they could
be implemented as `selector { byte_lower } in { BYTES }` (i.e.,
`byte_lower` is only range constrained when the machine call is active).
To make this work, I changed the `Machine::process_plookup` interface
like this:
```diff
fn process_plookup<'b, Q: QueryCallback<T>>(
&mut self,
mutable_state: &'b mut MutableState<'a, 'b, T, Q>,
identity_id: u64,
- args: &[AffineExpression<&'a AlgebraicReference, T>],
+ caller_rows: &'b RowPair<'b, 'a, T>,
) -> EvalResult<'a, T>;
```
The `RowPair` passed by the caller contains all range constraints known
at runtime. The LHS of the lookup (or permutation) is no longer
evaluated by the caller but by the callee. For this, the callee needs to
remember the identity associated with the `identity_id` (before this PR,
most machines just remembered the RHS, not the full identity). I don't
expect there to be any performance implications, because we only invoke
one machine (since #1154).
### Benchmark results
```
executor-benchmark/keccak
time: [14.609 s 14.645 s 14.678 s]
change: [-2.5984% -2.3127% -2.0090%] (p = 0.00 < 0.05)
Performance has improved.
executor-benchmark/many_chunks_chunk_0
time: [39.299 s 39.380 s 39.452 s]
change: [-3.9505% -3.6909% -3.4063%] (p = 0.00 < 0.05)
Performance has improved.
```
---------
Co-authored-by: Leo <leo@powdrlabs.com>
Cherry-picked ef6a72fcfa from #1380.
With this PR, we track whether a call to a machine led to some side
effect (e.g. added a block). In that case, the processed identity should
count has having led to some progress, even if no values were returned
to the calling machine. An example would be writing values to memory,
which does not return any values and hence does not change the state of
the caller.
Fixes an issue that @leonardoalt had on his `binary-mux2` branch.
There are two ways to have a block machine that is connected via a
permutation:
1. Use permutations `<sel> { ... } is (sub.sel * sub.LATCH) { ... }`.
This makes sure only rows where `sub.LATCH` is `1` can be selected. This
is what we do when we compile from ASM to PIL.
2. Use permutations `<sel> { ... } is sub.sel { ... }`, but also a
constraint `(1 - sub.LATCH) * sub.sel = 0`. This achieves something
similar.
The problem is that in the second case, detecting the block size is
harder, because the latch doesn't appear anywhere in the selector. So we
used to look into *all* fixed columns to detect the period. But this
includes fixed columns that might have a larger period (as is the case
for the multiplexer machine).
This PR simply removes support for the second approach. I think this is
fine in practice, as I don't see a disadvantage of the first approach
and when you come from ASM everything works as expected. I did need to
adjust `test_data/pil/block_lookup_or_permutation.pil`, which used the
second approach.
This PR attempts various issues around using challenges in hints, which
is blocking #1306:
1. Hints of later-phase witness columns are now removed in witgen, as
these columns don't need to be computed yet anyway and the hint might be
accessing a challenge that does not exist.
2. The query callback is now cloned for each phase of witness generation
(because otherwise it was only available in the first phase).
3. `SymbolicEvaluator` no longer panics when encountering challenges,
but returns an error. This evaluator is used to detect patterns in
identities, like `A' - A = 0`. This means that we can't detect patterns
in identities that involve challenges, but at least it doesn't panic.
4. `witgen::query_processor::Symbols` can now evaluate challenges.
5. `witgen::query_processor::Symbols` now also looks up intermediate
"polynomials" (which includes challenges). This is necessary because we
don't currently inline intermediate polynomials in hints (which we do
for identities).
I added a test that demonstrates that challenges can now be used in
hints.
This PR adds witness generation support for copy constraints: Whenever a
cell value is determined, this value is copied to all cells in its
equivalence class. This allows us to do witgen for arbitrary Plonkish
circuits (which would be detected as block machines) *as long as the
circuit is topologically sorted* (because otherwise, our row-by-row
solving strategy does not work.
Copy constraints are currently only supported in the language as
`connect` identities, as opposed to lists of cell pairs that belong to
the same equivalence class. Connecting this to the PIL input should be
part of another PR.
Fixes#844
This PR adds a new machine to the STD: `WriteOnceMemory`. This can be
used in our RISC-V machine for bootloader inputs (#1203).
Most of the issues mentioned in the issue were fixed in the meantime or
had a simple workaround (like defining `let LATCH = 1`). The only
remaining issues were in the machine detection, which I fixed here.
I also re-factor two existing tests.
