- Breaking: No longer copies Extism config values into WASI environment
variables because the new interface doesn't allow for the environment to
be updated - these should be accessed using the Extism config functions
instead
- Requires wasmtime 20 or greater
- Enables wasm-gc
- Similar to https://github.com/extism/extism/pull/697 without sockets
or additional support for command modules
- Adds quickcheck tests for alloc/free/load/store from the kernel, I
wasn't able to include these in the new wasm-bindgen tests because
`getrandom` isn't available on wasm32-unknown-unknown
- Fixes a bug in the calculation of how much memory is needed to
allocate the next block in the kernel. This bug is triggered when
allocating at the end of a page, the size of the MemoryBlock value
wasn't being taken in consideration when determining whether or not to
call memory.grow
- Uses `tracing` instead of `log` crate
- Uses `tracing-subscriber` instead of `fern`
- This allows us to automatically capture `log` events using
`tracing-subscriber`
- Breaking: Makes `extism::set_log_file` private and only used through
the C API, Rust users should use `tracing-subscriber` to determine which
filters/levels to log.
- Adds `extism::set_log_callback` function to set a callback that can be
used for custom logging from Rust.
- Adds `bool extism_log_custom(const char *level)` and
`extism_log_drain(void (*fn)(const char *s, size_t length)` to the C API
to enable custom sinks in other SDKs
My initial goal was to make logging configurable for each plugin instead
of global but wasn't able to accomplish that in this PR (still looking
into it)
- Switches from `log4rs` to `fern` - this significantly simplifies the
logging code
- Also considered `simplelog`
- Adds `plugin.id` to the logs whenever available
- Uses `extism::plugin::$id` target for functions logged from the PDK
Fixes https://github.com/extism/extism/issues/537
- Requires wasmtime 14.0.0 or greater for coredump serialization
- Adds `EXTISM_COREDUMP` environment variable to write wasmtime coredump
to a file when an error occurs
- This will create a coredump from the main wasm module, which means it
doesn't give us too much insight into the kernel
- Adds `EXTISM_MEMDUMP` environment variable to write extism linear
memory to a file when an error occurs
- This gives us access to the kernel memory
- Adds some missing profiling options
- Converts timeouts to a Trap instead of a plain error, this helps us
get better information about where the timeout occured
- Some small improvements to error handling after a plugin call
- Adds a test for coredump and memdump generation
- Adds the ability to configure debug options using `PluginBuilder`
- Fixes memory layout and a wasted page of memory in the kernel, found
while debugging a memory dump
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
- Adds benchmarking, run with `cargo bench` or `cargo criterion`
- Adds `MemoryRoot::pointer_in_bounds_fast` to do less precise bounds
checking in loads/stores
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
- Updates to wasmtime 13.0.0 as the lowest supported version
- Removes `wasmtime-wasi-nn` support because it now takes parameters to
specify particular backends and registries during initialization. Since
I don't think anyone is using the `nn` feature I chose to remove it for
now, but we could also expand the manifest to add these options.
Updates the requirements on [toml](https://github.com/toml-rs/toml) to
permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="310f6ee9c5"><code>310f6ee</code></a>
chore: Release</li>
<li><a
href="90da8bc425"><code>90da8bc</code></a>
docs: Update changelog</li>
<li><a
href="f3e120f1a0"><code>f3e120f</code></a>
Merge pull request <a
href="https://redirect.github.com/toml-rs/toml/issues/608">#608</a> from
epage/enum</li>
<li><a
href="58a7101f68"><code>58a7101</code></a>
fix(serde): Support struct variants as table of a table</li>
<li><a
href="88a4dba312"><code>88a4dba</code></a>
fix(serde): Support tuple variants as table of an array</li>
<li><a
href="cf06b83424"><code>cf06b83</code></a>
test(serde): Verify both Table and Value serializers</li>
<li><a
href="4ffa44ec16"><code>4ffa44e</code></a>
test(serde): Make parameter order more consistent</li>
<li><a
href="2b7c34c900"><code>2b7c34c</code></a>
test(serde): Focus on string serialization first</li>
<li><a
href="e2a6a1cece"><code>e2a6a1c</code></a>
test(serde): Verify existing variant behavior</li>
<li><a
href="3f3e8329bb"><code>3f3e832</code></a>
chore: Release</li>
<li>Additional commits viewable in <a
href="https://github.com/toml-rs/toml/compare/toml-v0.7.0...toml-v0.8.0">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on
[cbindgen](https://github.com/mozilla/cbindgen) to permit the latest
version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/mozilla/cbindgen/releases">cbindgen's
releases</a>.</em></p>
<blockquote>
<h2>0.26.0</h2>
<ul>
<li>Fix swapping of <code>>>=</code> and <code><<=</code> in
constants.</li>
<li>Add support for #[deprecated] (<a
href="https://redirect.github.com/mozilla/cbindgen/issues/860">#860</a>).</li>
<li>Built-in support for bitflags 2.0.</li>
<li>Support for "C-unwind" ABI.</li>
<li>Generate bindings for non-public extern items if they are
#[no_mangle].</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/mozilla/cbindgen/blob/master/CHANGES">cbindgen's
changelog</a>.</em></p>
<blockquote>
<h1>0.26.0</h1>
<pre><code> * Fix swapping of `>>=` and `<<=` in constants.
* Add support for #[deprecated]
([#860](https://github.com/mozilla/cbindgen/issues/860)).
* Built-in support for bitflags 2.0.
* Support for "C-unwind" ABI.
* Generate bindings for non-public extern items if they are
#[no_mangle].
</code></pre>
<h2>0.25.0</h2>
<pre><code> * Re-release of yanked 0.24.6 as a major release
* Update MSRV to 1.57
* Support variadic arguments (`...`)
([#805](https://github.com/mozilla/cbindgen/issues/805))
* Add --depfile option
([#820](https://github.com/mozilla/cbindgen/issues/820))
* Breaking changes: The `Config` struct now has a private member.
</code></pre>
<h2>0.24.6 (YANKED: depfile option was breaking, see <a
href="https://redirect.github.com/mozilla/cbindgen/issues/841">#841</a>)</h2>
<pre><code> * Update MSRV to 1.57
* Support variadic arguments (`...`)
([#805](https://github.com/mozilla/cbindgen/issues/805))
* Add --depfile option
([#820](https://github.com/mozilla/cbindgen/issues/820))
</code></pre>
<h2>0.24.5</h2>
<pre><code> * Don't enforce tempfile version.
</code></pre>
<h2>0.24.4</h2>
<pre><code> * Move expand infinite recursion fix
([#799](https://github.com/mozilla/cbindgen/issues/799))
* Add with_cpp_compat to the builder
([#796](https://github.com/mozilla/cbindgen/issues/796))
* Handle never type in return position consistently
([#780](https://github.com/mozilla/cbindgen/issues/780))
* Fix warnings ([#816](https://github.com/mozilla/cbindgen/issues/816),
[#819](https://github.com/mozilla/cbindgen/issues/819))
* Updated documentation
([#788](https://github.com/mozilla/cbindgen/issues/788),
[#791](https://github.com/mozilla/cbindgen/issues/791),
[#792](https://github.com/mozilla/cbindgen/issues/792),
[#810](https://github.com/mozilla/cbindgen/issues/810),
[#823](https://github.com/mozilla/cbindgen/issues/823))
</code></pre>
<h2>0.24.3</h2>
<pre><code> * Make struct expressions correctly generated through
typedefs ([#768](https://github.com/mozilla/cbindgen/issues/768)).
</code></pre>
<h2>0.24.2</h2>
<pre><code> * Make bitfield operators use explicit constructors.
</code></pre>
<h2>0.24.1</h2>
<pre><code> * Add support for unary negation
([#765](https://github.com/mozilla/cbindgen/issues/765)).
* Make more bitfield operators constexpr
([#765](https://github.com/mozilla/cbindgen/issues/765)).
</code></pre>
<h2>0.24.0</h2>
<pre><code> * Basic const generic support
([#759](https://github.com/mozilla/cbindgen/issues/759),
[#760](https://github.com/mozilla/cbindgen/issues/760)
[#762](https://github.com/mozilla/cbindgen/issues/762)).
</code></pre>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="703b53c06f"><code>703b53c</code></a>
v0.26.0</li>
<li><a
href="56f0febc9b"><code>56f0feb</code></a>
Update MSRV in Readme</li>
<li><a
href="9b4a14958e"><code>9b4a149</code></a>
Add support for out-of-line bitfields declarations</li>
<li><a
href="35f2e44ef2"><code>35f2e44</code></a>
Update URLs</li>
<li><a
href="85eb0f4436"><code>85eb0f4</code></a>
Bump clippy msrv to 1.64</li>
<li><a
href="43af1ebe6e"><code>43af1eb</code></a>
Handle bitflags bits method calls</li>
<li><a
href="f72e447156"><code>f72e447</code></a>
CHANGES: Note #[deprecated] support.</li>
<li><a
href="1473070230"><code>1473070</code></a>
utilities: annotation: Clean-up deprecated parsing and getter.</li>
<li><a
href="0fb5d07ae4"><code>0fb5d07</code></a>
Add support for #[deprecated].</li>
<li><a
href="d8355da466"><code>d8355da</code></a>
Support "C-unwind" ABI</li>
<li>Additional commits viewable in <a
href="https://github.com/mozilla/cbindgen/compare/v0.25.0...0.26.0">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- Adds `extism-convert` crate with `ToBytes`, `FromBytes` and
`FromBytesOwned` traits
- This serves as a single interface for reading/writing rich types from
WebAssembly linear memory.
- Supports `Json` and `Msgpack` and `Protobuf` encodings out-of-the-box
- Updates `Plugin::call` to take `ToBytes` as the input argument and
return a `FromBytes` value
- Adds `host_fn!` macro to simplify host function creation
- Cleans up generated documentation a little
- PR for the Rust PDK: https://github.com/extism/rust-pdk/pull/31
- Adds a `typed_plugin!` macro to implement type-safe wrappers around
`Plugin`
- After this we should focus on adding similar type-conversion helpers
to the SDKs and other PDKs to make it easier to use across languages.
For example, a Python host running a Rust plugin using Msgpack encoded
types.
## Examples
### Calling a function
Instead of the untyped, bytes-only `call` function:
```rust
let output = plugin.call("func_name", "my data").unwrap();
let output: MyType = serde_json::from_slice(&output).unwrap();
```
We can now use richer types to encode/decode our values directly when
using `call`:
```rust
let Json(output) = plugin.call::<_, Json<MyType>>("func_name", "my data").unwrap();
```
### Allocating inside of a host function
The same interface works for host functions, so instead of:
```rust
fn hello_world(
plugin: &mut CurrentPlugin,
inputs: &[Val],
outputs: &mut [Val],
_user_data: UserData,
) -> Result<(), Error> {
let handle = plugin.memory_handle_val(&inputs[0])?;
let input = plugin.memory_read_str(handle)?;
let output = plugin.memory_alloc_bytes(&input).unwrap();
outputs[0] = output.into();
Ok(())
}
```
Becomes:
```rust
fn hello_world(
plugin: &mut CurrentPlugin,
inputs: &[Val],
outputs: &mut [Val],
_user_data: UserData,
) -> Result<(), Error> {
let my_value: String = plugin.memory_get_val(&inputs[0])?;
let output = plugin.memory_new(&my_value)?;
outputs[0] = plugin.memory_to_val(output);
Ok(())
}
```
Although this isn't much of an improvement, using the `host_fn` macro,
we can really begin to see how the above function is really just an
identity function:
```rust
host_fn!(hello_world(a: String) -> String {
a
});
```
### typed_plugin!
`typed_plugin!` is used to make a typed wrapper around a Plugin:
```rust
/// Create the typed plugin
typed_plugin!(Testing {
count_vowels(&str) -> Json<Count>
});
/// Create the `Plugin` and convert it to `Testing` wrapper
let mut plugin: Testing = Plugin::new(WASM, [f], true).unwrap().into();
/// Call the `count_vowels` function:
let Json(output0): Json<Count> = plugin.count_vowels("abc123")?;
```
It could make sense to convert `host_fn` and/or `typed_plugin` to
proc-macros at some point, but for now they work and provide some
flexibility in experimenting with the interfaces. Another future update
could be to figure out a nice way to make it so input can be written in
multiple chunks, so the entire input doesn't have to get copied into
memory at once.
- Removes the `ExtismContext` type from runtime and all SDKs
- Removed SDK functions: `extism_context_new`, `extism_context_reset`,
`extism_context_free`
- All SDKs have been updated, but there are still some TODOs below
- Removes `extism_plugin_update`
- Plugins can no longer be updated - a new plugin should be created
instead
- Adds `extism_plugin_id` to uniquely identify plugins
- Merges the `extism-runtime` and `extism` crates (there is no longer an
`extism-runtime` crate)
- Makes `extism::Manifest` an alias for `extism_manifest::Manifest`
instead of a distinct type
- Adds `MemoryHandle` type to SDKs to refer to blocks of Extism memory
that can be accessed in host functions
- Improves thread-safety of Plugins, adds C++ test to call a single
plugin from multiple threads.
- Expands wasmtime bounds to include 12.0
- Delays instantiation to right before a call by using `instantiate_pre`
instead. This fixes an issue with the assemblyscript PDK.
- Makes timeouts only apply to calls
- Also bumps the wasmtime lower-bounds to 10.0, because the return type
of the epoch callback has changed
This PR adds the `kernel` directory which contains a port of the Extism
memory allocator compiled to WebAssembly and removes
`runtime/src/memory.rs` completely.
Being able to re-use memory functions as a WASM module allows us to
begin to experiment with porting Extism to new runtimes!
This is in a draft state while I'm verifying some of these changes.
Currently wasmtime `8.0.0` - `10.0.0` are compatible with
`extism-runtime`, in the interest of remaining compatible with as many
applications as possible, this PR updates the wasmtime version
requirement from a single version to a range of acceptable versions.
## Breaking Changes
* https://github.com/extism/extism/pull/315
HTTP calls will be disallowed by default now. If you want to enable HTTP
you need to specify the hosts that the plug-in is allowed to communicate
with. If you want to allow all hosts you can set it to `{allowed_hosts:
["*"]}` in the manifest. However, this isn't recommended unless you have
some trust in the plug-in or are controlling the networking by some
other means.
* https://github.com/extism/extism/pull/335
In this PR we are creating an implicit context so people don't need to
know about it if they don't care. In some languages function signatures
have changed to make context an optional argument when creating a
plug-in.
Unsure if now is the best time to do the `extism` crate version bumps,
but I figured they'd need to happen at some point in the near future
anyways. Happy to revert if there is any opposition. Also, I added back
the local `path` property to the Elixir NIF crate manifest. I want to
see if this breaks again in CI, or if we can leave it.
---------
Co-authored-by: zach <zach@dylib.so>
Let's get the last changes in this week for a release. Will hold this PR
until all changes we want are in.
Release checklist:
- [x] test: updates across CI to test for Host Function output /
integration (#219)
- this should probably look something like a grep for some kind of
output proving guest/host interop?
- [x] Fix for userData pointer issue in Go host functions (#220)
- [x] docs: Host Functions in SDKs
- [ ] sdk: C
- [ ] sdk: C++
- [ ] sdk: Python
- [ ] sdk: Node
- [ ] sdk: Go
- [ ] sdk: Rust
- [x] docs: Manifest property names (http `headers` & memory
`max_pages`)
- [ ] blog: announcing v0.2.0, including host functions, Zig SDK/PDK,
Java SDK, .NET SDK, + ...
- Adds `Manifest::timeout_ms` field to specify the plugin timeout in
milliseconds
- Sets a 30 second default timeout
Co-authored-by: Steve Manuel <steve@dylib.so>
- Updates codebase to use the latest version of wasmtime, wasmtime-wasi
and wasmtime-wasi-nn
- Allows functions with no return values to be called, this provides
basic support for `_start` functions
- For now `_start` functions called by extism still need to use the
extism input/output functions instead of the command line arguments
- Adds `allowed_hosts` to the manifest
- If there are any `allowed_hosts` then `extism_http_request` checks to
make sure a URL's host is listed before performing the reques
