Okay, phew.
The main bug from the [last
PR](https://github.com/extism/extism/pull/610) was that the rust
releases scramble to publish, but we have to publish them in a
particular order for the release to work since they depend on each
other.
There's a secondary bug: `cargo publish` is prone to 502'ing on publish.
I lean towards seeing how painful this is in practice; we should be able
to safely re-run the release flow on failure.
Add an example of dynamically linking plugins and a benchmark that does
an apples-to-apples comparison of `reflect` using host functions vs.
`reflect` using a linked wasm module. (To my surprise, the host
functions are a _little bit faster_!)
This PR simplifies the resolution of the `main` module when multiple
modules are provided. Before we would try to look at the path/URL when
the wasm was coming from disk or via HTTP, now any module missing a name
will be used as `main`. This is much nicer and more consistent since
this is what was being done when no filename was available (i.e. raw
data modules). It also make sense because non-main modules will need to
be named for the functions to be linked correctly.
Alternate to: #596 without support for manually compiling/loading native
code
- Enables wasmtime caching: https://docs.wasmtime.dev/cli-cache.html
- Adds `EXTISM_CACHE_CONFIG` and `PluginBuilder::with_cache_config` to
determine where to load a custom cache configuration from
- Adds `PluginBuilder::with_cache_disabled` to disable the cache when
initializing a plugin
- Setting `EXTISM_CACHE_CONFIG=""` will also disable caching
## Performance
With caching:
```
create/create_plugin time: [2.9079 ms 2.9139 ms 2.9200 ms]
change: [+3.2677% +3.6399% +3.9766%] (p = 0.00 < 0.20)
Change within noise threshold.
```
Compared to `main`:
```
create/create_plugin time: [26.089 ms 26.498 ms 26.923 ms]
change: [+0.1729% +2.0868% +4.1337%] (p = 0.04 < 0.20)
Change within noise threshold.
```
Follow-up to f7d297f98f.
Update the release workflows for the dependent crates. The Cargo
workflows run on GitHub release publish to match the Python package
release workflow. This means all of our crates are versioned in lockstep
by the Git tag.
There are four changes:
1. Trigger the workflows on GitHub release publish
2. Add the `set version` step from `release.yml` to modify the version
tags in `Cargo.toml`.
3. Publish with `--allow-dirty` (to account for the edit in step 2)
4. In `extism-maturin`, do not inherit `authors` from the workspace
since [PyPI rejects the value we have
set](https://github.com/extism/extism/actions/runs/7012534645/job/19077216730#step:7:23).
Further flesh out our runtime benchmarks: add benchmarks for sending
bytes one-way into Wasm linear memory, as well as testing copy-in then
copy-out. This compliments our `reflect` benchmark; we can get a sense
of how expensive the sub-operations are via `echo` and `consume`.
(Notably, we're missing a `produce` or `emit` that purely generates
output and sends it to the host!)
The source of these plugins is available in `extism/plugins` as
`echo.wat` and `consume.wat` [1].
[1]: https://github.com/extism/plugins/pull/4/files
This parameterizes the `reflect` test and calls the `g.throughput` on
each data set to get an idea of our roundtrip throughput. (The changes
follow the "Throughput Measurements" [1] part of the Criterion guide
pretty much to the letter.)
Example output:
```
reflect/reflect 1 time: [208.57 µs 209.05 µs 209.57 µs]
thrpt: [298.23 MiB/s 298.97 MiB/s 299.66 MiB/s]
```
[1]:
https://bheisler.github.io/criterion.rs/book/user_guide/advanced_configuration.html#throughput-measurements
- Uses `tracing` instead of `log` crate
- Uses `tracing-subscriber` instead of `fern`
- This allows us to automatically capture `log` events using
`tracing-subscriber`
- Breaking: Makes `extism::set_log_file` private and only used through
the C API, Rust users should use `tracing-subscriber` to determine which
filters/levels to log.
- Adds `extism::set_log_callback` function to set a callback that can be
used for custom logging from Rust.
- Adds `bool extism_log_custom(const char *level)` and
`extism_log_drain(void (*fn)(const char *s, size_t length)` to the C API
to enable custom sinks in other SDKs
Attempts to address an error seen when upgrading both Rust PDK & SDK to
latest:
```
Updating git repository `https://github.com/extism/extism.git`
Updating git repository `https://github.com/extism/rust-pdk.git`
Updating crates.io index
error: failed to select a version for `base64`.
... required by package `extism-convert v0.2.0 (https://github.com/extism/extism.git?branch=main#7636c873)`
... which satisfies git dependency `extism-convert` of package `extism v1.0.0-alpha.0 (https://github.com/extism/extism.git?branch=main#7636c873)`
... which satisfies git dependency `extism` of package `proto_core v0.22.4 (/Users/miles/Projects/proto/crates/core)`
... which satisfies path dependency `proto_core` (locked to 0.22.4) of package `proto_cli v0.22.0 (/Users/miles/Projects/proto/crates/cli)`
versions that meet the requirements `^0.21.3` are: 0.21.5, 0.21.4, 0.21.3
all possible versions conflict with previously selected packages.
previously selected package `base64 v0.21.0`
... which satisfies dependency `base64 = "^0.21.0"` (locked to 0.21.0) of package `extism-manifest v0.5.0`
... which satisfies dependency `extism-manifest = "^0.5.0"` (locked to 0.5.0) of package `extism-pdk v1.0.0-beta.0 (https://github.com/extism/rust-pdk.git?branch=main#009bf808)`
... which satisfies git dependency `extism-pdk` of package `proto_pdk v0.10.2 (/Users/miles/Projects/proto/crates/pdk)`
failed to select a version for `base64` which could resolve this conflict
```
My initial goal was to make logging configurable for each plugin instead
of global but wasn't able to accomplish that in this PR (still looking
into it)
- Switches from `log4rs` to `fern` - this significantly simplifies the
logging code
- Also considered `simplelog`
- Adds `plugin.id` to the logs whenever available
- Uses `extism::plugin::$id` target for functions logged from the PDK
I realized the glibc static build does actually work (though prints
warnings at compile time), so I made it no longer hard-coded to the
compiler. It's still a bad idea, so I added notes for why you shouldn't
do it with glibc.
The `x86_64-unknown-linux-musl` target does not build dynamic libraries
for now:
```
warning: dropping unsupported crate type `cdylib` for target `x86_64-unknown-linux-musl`
```
A target may now be optionally specified to the root `Makefile`,
example:
`make RUST_TARGET=x86_64-unknown-linux-musl && sudo make
RUST_TARGET=x86_64-unknown-linux-musl install`
The new musl targets may be used to create entirely static binaries with
libextism. See the new `musl-static` target in `libextism/Makefile`
- Removes `Plugin::new_with_manifest` and updates `Plugin::new` to take
wasm bytes or `Manifest` using the new `WasmInput` trait
- Removes `PluginBuilder::new_with_module` in favor of a
`PluginBuilder::new` with a `WasmInput` argument
```
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
```
Adds `pkg-config` configs for dynamic and static builds of libextism.
Example usage is shown in `libextism/Makefile`.
Added the `.in` versions of the `.pc` files to non-MSVC releases.
This will help simplify writing the types for host functions, `PTR` can
be used for any arguments that will point to Extism memory instead of
`ValType::I64`
We were updating the epoch deadline callback after the call has ended to
ignore any timeouts that might happen when reading output, which would
allow for the plugin to be cancelled successfully the first time a
plugin is called but would fail after the first call. This PR fixes
cancellation by resetting the epoch deadline and callback before each
plugin call.
I tested this against the dotnet sdk tests, where this was discovered,
and added a similar test to the runtime test suite,
Fixes#556
See #504
Removes CI for the in-repo SDKs, we can remove the actual code closer to
the 1.0 release.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
* libextism is now built to crate-type `staticlib` as recommend by
@zshipko
* updated `Makefile` to install and uninstall the static library.
* updated c example to have `static` target to link libextism statically
* Compiler is no longer hardcoded to `clang`. Set `CC` to use a compiler
other than the system compiler. Tested with `gcc` and `clang`.
* added `static-artifact` to actions release. Untested as it's only ran
on push with tag.
---------
Co-authored-by: zach <zach@dylibso.com>
Fixes#545
- Adds a type parameter to `UserData` type to avoid dynamic typing
issues
- Adds KV example from README to `examples/readme.rs` to make sure it
stays in-sync
- Implement `Default` for `UserData<T>` when `T` implements `Default`
- Make `UserData` argument non-optional in `Function::new`,
`UserData::default()` can be used instead.
Point everything to to new repos. We will eventually remove these
subrepos to clean everything up but leaving them for the next month at
least.
---------
Co-authored-by: zach <zach@dylibso.com>
Fixes https://github.com/extism/extism/issues/537
- Requires wasmtime 14.0.0 or greater for coredump serialization
- Adds `EXTISM_COREDUMP` environment variable to write wasmtime coredump
to a file when an error occurs
- This will create a coredump from the main wasm module, which means it
doesn't give us too much insight into the kernel
- Adds `EXTISM_MEMDUMP` environment variable to write extism linear
memory to a file when an error occurs
- This gives us access to the kernel memory
- Adds some missing profiling options
- Converts timeouts to a Trap instead of a plain error, this helps us
get better information about where the timeout occured
- Some small improvements to error handling after a plugin call
- Adds a test for coredump and memdump generation
- Adds the ability to configure debug options using `PluginBuilder`
- Fixes memory layout and a wasted page of memory in the kernel, found
while debugging a memory dump
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
