- Adds `memory.max_var_bytes` to the manifest to limit the number of
bytes allowed to be stored in Extism vars - if `max_var_bytes` is set to
0 then vars are disabled.
- Adds some builder functions to `MemoryOptions` struct
- Sets the default var store size to 1mb
- Includes a test to make sure `var_set` returns an error when the limit
is reached
It looks like when a module is added to the linker, all of its imports
must already be present. This PR updates the linking process to take
that into consideration and adds a test with a reproduction of the issue
@chrisdickinson shared with me.
- Adds quickcheck tests for alloc/free/load/store from the kernel, I
wasn't able to include these in the new wasm-bindgen tests because
`getrandom` isn't available on wasm32-unknown-unknown
- Fixes a bug in the calculation of how much memory is needed to
allocate the next block in the kernel. This bug is triggered when
allocating at the end of a page, the size of the MemoryBlock value
wasn't being taken in consideration when determining whether or not to
call memory.grow
base64 support is retained. Now `data` can instead be `{"ptr":
ptr_value, "len": len_value}` where `ptr_value` points to a wasm module
and `len_value` is the size of bytes of it.
Merged the other one too quickly, this PR is updated with some
additional feedback from @chrisdickinson:
(Ooh, this could also start with `(;`)
https://github.com/extism/extism/pull/650#discussion_r1443419249
It's also updated to accommodate for modules that start with an open
paren followed by some whitespace. I doubt this covers all of the
permitted syntax but it should be good enough.
- Adds `length_unsafe` function to the extism kernel, a more performant
`length` for known-valid memory handles
After this is merged I will update go-sdk and js-sdk too.
Closes#643
This fixes a test failure on the python-sdk [1]. (See also [2]). In
particular, while maturin creates bindings for `extism_log_drain` on
clang platforms, it seems that MSVC cannot generate the required binding
information for `ffi.py`. This results in an `extism_sys` wheel whose
shared object (in this case, a DLL) contains the `extism_log_drain`, but
whose Python FFI bindings don't contain a definition for that function.
Naming the function pointer type parameter resolves the issue. What a
strange issue!
[1]:
https://github.com/extism/python-sdk/actions/runs/7172934060/job/19669775001#step:9:35
[2]:
https://github.com/extism/python-sdk/pull/18#issuecomment-1850892835
Fixes#634
- Updates `extism_length` to walks the allocation list to determine
valid offsets instead of assuming the provided offset is valid
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
See https://github.com/extism/cpp-sdk/issues/15
- Limits a call to memset in the kernel to the size of the current
memory offset instead of the total size of memory.
- Adds `extism_plugin_reset` to the C API and `extism::Plugin::reset` to
Rust
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
Before `extern "C"` was needed to include the header in C++ code. Now
that's included inside, so it's easier and cleaner to include in c++
projects like the `cpp-sdk`.
Fixes#619
- Also updates `host_fn`, `encoding`, and `typed_plugin` macros to allow
for visibility specifiers to manage the visibility of the defined types.
This means that `pub` will need to be added to existing invocations that
should remain public
Add an example of dynamically linking plugins and a benchmark that does
an apples-to-apples comparison of `reflect` using host functions vs.
`reflect` using a linked wasm module. (To my surprise, the host
functions are a _little bit faster_!)
This PR simplifies the resolution of the `main` module when multiple
modules are provided. Before we would try to look at the path/URL when
the wasm was coming from disk or via HTTP, now any module missing a name
will be used as `main`. This is much nicer and more consistent since
this is what was being done when no filename was available (i.e. raw
data modules). It also make sense because non-main modules will need to
be named for the functions to be linked correctly.
Alternate to: #596 without support for manually compiling/loading native
code
- Enables wasmtime caching: https://docs.wasmtime.dev/cli-cache.html
- Adds `EXTISM_CACHE_CONFIG` and `PluginBuilder::with_cache_config` to
determine where to load a custom cache configuration from
- Adds `PluginBuilder::with_cache_disabled` to disable the cache when
initializing a plugin
- Setting `EXTISM_CACHE_CONFIG=""` will also disable caching
## Performance
With caching:
```
create/create_plugin time: [2.9079 ms 2.9139 ms 2.9200 ms]
change: [+3.2677% +3.6399% +3.9766%] (p = 0.00 < 0.20)
Change within noise threshold.
```
Compared to `main`:
```
create/create_plugin time: [26.089 ms 26.498 ms 26.923 ms]
change: [+0.1729% +2.0868% +4.1337%] (p = 0.04 < 0.20)
Change within noise threshold.
```
Further flesh out our runtime benchmarks: add benchmarks for sending
bytes one-way into Wasm linear memory, as well as testing copy-in then
copy-out. This compliments our `reflect` benchmark; we can get a sense
of how expensive the sub-operations are via `echo` and `consume`.
(Notably, we're missing a `produce` or `emit` that purely generates
output and sends it to the host!)
The source of these plugins is available in `extism/plugins` as
`echo.wat` and `consume.wat` [1].
[1]: https://github.com/extism/plugins/pull/4/files
This parameterizes the `reflect` test and calls the `g.throughput` on
each data set to get an idea of our roundtrip throughput. (The changes
follow the "Throughput Measurements" [1] part of the Criterion guide
pretty much to the letter.)
Example output:
```
reflect/reflect 1 time: [208.57 µs 209.05 µs 209.57 µs]
thrpt: [298.23 MiB/s 298.97 MiB/s 299.66 MiB/s]
```
[1]:
https://bheisler.github.io/criterion.rs/book/user_guide/advanced_configuration.html#throughput-measurements
- Uses `tracing` instead of `log` crate
- Uses `tracing-subscriber` instead of `fern`
- This allows us to automatically capture `log` events using
`tracing-subscriber`
- Breaking: Makes `extism::set_log_file` private and only used through
the C API, Rust users should use `tracing-subscriber` to determine which
filters/levels to log.
- Adds `extism::set_log_callback` function to set a callback that can be
used for custom logging from Rust.
- Adds `bool extism_log_custom(const char *level)` and
`extism_log_drain(void (*fn)(const char *s, size_t length)` to the C API
to enable custom sinks in other SDKs
My initial goal was to make logging configurable for each plugin instead
of global but wasn't able to accomplish that in this PR (still looking
into it)
- Switches from `log4rs` to `fern` - this significantly simplifies the
logging code
- Also considered `simplelog`
- Adds `plugin.id` to the logs whenever available
- Uses `extism::plugin::$id` target for functions logged from the PDK
- Removes `Plugin::new_with_manifest` and updates `Plugin::new` to take
wasm bytes or `Manifest` using the new `WasmInput` trait
- Removes `PluginBuilder::new_with_module` in favor of a
`PluginBuilder::new` with a `WasmInput` argument
This will help simplify writing the types for host functions, `PTR` can
be used for any arguments that will point to Extism memory instead of
`ValType::I64`
