This is a rough POC for allowing people to whitelist a dir as readonly.
When the source path is prefixed with `ro:`, the dir is considered as
readonly. This preserved backward compatibility. This suggestion came up
in https://github.com/extism/go-sdk/pull/1#discussion_r1276700587
Readonly:
```rs
let manifest = Manifest::new([url])
.with_allowed_path("ro:D:/x/rust/fs/data".to_string(), "/data")
.with_config_key("path", "/data/data.txt");
```
```
trying to read file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n"
-----------------------------------------------------
trying to write file:
thread '<unnamed>' panicked at src\lib.rs:24:34:
called `Result::unwrap()` on an `Err` value: Os { code: 58, kind: Unsupported, message: "Not supported" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at runtime\examples\fs.rs:27:6:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
0: 0x234d2 - fs.wasm!__rust_start_panic
1: 0x232a1 - fs.wasm!rust_panic
2: 0x231da - fs.wasm!std::panicking::rust_panic_with_hook::hd3fb69bc0aea298a
3: 0x22467 - fs.wasm!std::panicking::begin_panic_handler::{{closure}}::h4d99b90b43f79472
4: 0x223ca - fs.wasm!std::sys_common::backtrace::__rust_end_short_backtrace::h5691573a73161cb1
5: 0x22bca - fs.wasm!rust_begin_unwind
6: 0x303e9 - fs.wasm!core::panicking::panic_fmt::hdb62f5cdb45533e4
7: 0x3234d - fs.wasm!core::result::unwrap_failed::h30d23efcc9e41efc
8: 0x36c2 - fs.wasm!fs::try_write::inner::h0b3b0df8e129f5cc
9: 0x29cd - fs.wasm!try_write
10: 0x35e4a - fs.wasm!try_write.command_export
note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
Caused by:
wasm trap: wasm `unreachable` instruction executed
Stack backtrace:
0: std::backtrace_rs::backtrace::dbghelp64::trace
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:99
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
2: std::backtrace::Backtrace::create
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:331
3: std::backtrace::Backtrace::capture
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:296
4: anyhow::error::impl$1::from<wasmtime_environ::trap_encoding::Trap>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\anyhow-1.0.86\src\error.rs:565
5: core::convert::impl$3::into<wasmtime_environ::trap_encoding::Trap,anyhow::Error>
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\convert\mod.rs:759
6: wasmtime_environ::impl$1::into_anyhow<wasmtime_environ::trap_encoding::Trap>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-environ-22.0.0\src\lib.rs:90
7: wasmtime::runtime::trap::from_runtime_box
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\trap.rs:118
8: wasmtime::runtime::func::invoke_wasm_and_catch_traps::closure$0<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597
9: enum2$<core::result::Result<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global> > >::map_err<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global>,anyhow::Error,wasmtime::runtime:
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:829
10: wasmtime::runtime::func::invoke_wasm_and_catch_traps<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597
11: wasmtime::runtime::func::Func::call_unchecked_raw<extism::current_plugin::CurrentPlugin>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1063
12: wasmtime::runtime::func::Func::call_unchecked<ref_mut$<wasmtime::runtime::store::context::StoreContextMut<extism::current_plugin::CurrentPlugin> > >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1049
13: wasmtime::runtime::func::Func::call_impl_do_call<extism::current_plugin::CurrentPlugin>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1243
14: wasmtime::runtime::func::Func::call<ref_mut$<wasmtime::runtime::store::Store<extism::current_plugin::CurrentPlugin> > >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1002
15: extism::plugin::Plugin::raw_call<ref$<str$>,ref$<str$> >
at .\src\plugin.rs:753
16: extism::plugin::Plugin::call<ref$<str$>,ref$<str$>,ref$<str$> >
at .\src\plugin.rs:900
17: fs::main
at .\examples\fs.rs:25
18: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250
19: core::hint::black_box
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337
20: std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\sys_common\backtrace.rs:155
21: std::rt::lang_start::closure$0<tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:166
22: std::rt::lang_start_internal
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\rt.rs:148
23: std::rt::lang_start<tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:165
24: main
25: invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
26: __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
27: BaseThreadInitThunk
28: RtlUserThreadStart
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\panicking.rs:645
1: core::panicking::panic_fmt
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\panicking.rs:72
2: core::result::unwrap_failed
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\result.rs:1654
3: enum2$<core::result::Result<ref$<str$>,anyhow::Error> >::unwrap
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:1077
4: fs::main
at .\examples\fs.rs:25
5: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250
6: core::hint::black_box
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `D:\dylibso\extism\target\debug\examples\fs.exe` (exit code: 101)
```
Writable:
```rs
let manifest = Manifest::new([url])
.with_allowed_path("D:/x/rust/fs/data".to_string(), "/data")
.with_config_key("path", "/data/data.txt");
```
```
trying to read file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n"
-----------------------------------------------------
trying to write file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\nHello World at 1719851500.7803263s\n"
done!
```
- Breaking: No longer copies Extism config values into WASI environment
variables because the new interface doesn't allow for the environment to
be updated - these should be accessed using the Extism config functions
instead
- Requires wasmtime 20 or greater
- Enables wasm-gc
- Similar to https://github.com/extism/extism/pull/697 without sockets
or additional support for command modules
Currently HTTP requests can extend beyond the configured timeout for a
plugin - this PR sets the timeout of the HTTP request to the remaining
time left if a timeout is set in the manifest.
Add `plugin.call_with_host_context` and `current_plugin.host_context`
methods, enabling per-call context to be looped from the guest invocation
to any host functions it calls. In an HTTP server environment, this enables
re-using a plugin across multiple requests while switching out backing
connections and user information in host functions. (Imagine a host
function, `update_user` -- previously the plugin would have to have been
aware of the user to pass to the host function. Now that information is
ambient.)
This is a backwards-compatible change and requires no changes to
existing plugins.
Implement by adding a global, mutable externref to the extism kernel.
Since most programming languages, including Rust, don't let you define
these natively, we accomplish this by using `wasm-merge` to combine the
kernel Wasm with Wasm generated by a WAT file containing only the
global.
(This pattern might be useful for other Wasm constructs we can't use
directly from Rust, like `v128` in argument parameters.)
Wasmtime requires extern refs to be `Any + Send + Sync + 'static`; we
additionally add `Clone`. I haven't tried this with an `Arc` directly,
but it should work at least for container structs that hold `Arc`'s
themselves.
Some of the old constants conflicted with Perl headers used for build
Perl extensions such the
[perl-sdk](002e4437ac/Extism/lib/Extism/XS.xs).
This fix would allow removing the copy of extism.h (inlined in that file
linked) out of the perl-sdk.
These constants especially `I32` likely have conflicts with many other
large C codebases.
A new extism release with this fix should not be made until the affected
sdks (cpp-sdk) have an update ready.
- Adds `memory.max_var_bytes` to the manifest to limit the number of
bytes allowed to be stored in Extism vars - if `max_var_bytes` is set to
0 then vars are disabled.
- Adds some builder functions to `MemoryOptions` struct
- Sets the default var store size to 1mb
- Includes a test to make sure `var_set` returns an error when the limit
is reached
It looks like when a module is added to the linker, all of its imports
must already be present. This PR updates the linking process to take
that into consideration and adds a test with a reproduction of the issue
@chrisdickinson shared with me.
- Adds quickcheck tests for alloc/free/load/store from the kernel, I
wasn't able to include these in the new wasm-bindgen tests because
`getrandom` isn't available on wasm32-unknown-unknown
- Fixes a bug in the calculation of how much memory is needed to
allocate the next block in the kernel. This bug is triggered when
allocating at the end of a page, the size of the MemoryBlock value
wasn't being taken in consideration when determining whether or not to
call memory.grow
base64 support is retained. Now `data` can instead be `{"ptr":
ptr_value, "len": len_value}` where `ptr_value` points to a wasm module
and `len_value` is the size of bytes of it.
Merged the other one too quickly, this PR is updated with some
additional feedback from @chrisdickinson:
(Ooh, this could also start with `(;`)
https://github.com/extism/extism/pull/650#discussion_r1443419249
It's also updated to accommodate for modules that start with an open
paren followed by some whitespace. I doubt this covers all of the
permitted syntax but it should be good enough.
- Adds `length_unsafe` function to the extism kernel, a more performant
`length` for known-valid memory handles
After this is merged I will update go-sdk and js-sdk too.
Closes#643
This fixes a test failure on the python-sdk [1]. (See also [2]). In
particular, while maturin creates bindings for `extism_log_drain` on
clang platforms, it seems that MSVC cannot generate the required binding
information for `ffi.py`. This results in an `extism_sys` wheel whose
shared object (in this case, a DLL) contains the `extism_log_drain`, but
whose Python FFI bindings don't contain a definition for that function.
Naming the function pointer type parameter resolves the issue. What a
strange issue!
[1]:
https://github.com/extism/python-sdk/actions/runs/7172934060/job/19669775001#step:9:35
[2]:
https://github.com/extism/python-sdk/pull/18#issuecomment-1850892835
Fixes#634
- Updates `extism_length` to walks the allocation list to determine
valid offsets instead of assuming the provided offset is valid
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
See https://github.com/extism/cpp-sdk/issues/15
- Limits a call to memset in the kernel to the size of the current
memory offset instead of the total size of memory.
- Adds `extism_plugin_reset` to the C API and `extism::Plugin::reset` to
Rust
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
Before `extern "C"` was needed to include the header in C++ code. Now
that's included inside, so it's easier and cleaner to include in c++
projects like the `cpp-sdk`.
Fixes#619
- Also updates `host_fn`, `encoding`, and `typed_plugin` macros to allow
for visibility specifiers to manage the visibility of the defined types.
This means that `pub` will need to be added to existing invocations that
should remain public
Add an example of dynamically linking plugins and a benchmark that does
an apples-to-apples comparison of `reflect` using host functions vs.
`reflect` using a linked wasm module. (To my surprise, the host
functions are a _little bit faster_!)
This PR simplifies the resolution of the `main` module when multiple
modules are provided. Before we would try to look at the path/URL when
the wasm was coming from disk or via HTTP, now any module missing a name
will be used as `main`. This is much nicer and more consistent since
this is what was being done when no filename was available (i.e. raw
data modules). It also make sense because non-main modules will need to
be named for the functions to be linked correctly.
Alternate to: #596 without support for manually compiling/loading native
code
- Enables wasmtime caching: https://docs.wasmtime.dev/cli-cache.html
- Adds `EXTISM_CACHE_CONFIG` and `PluginBuilder::with_cache_config` to
determine where to load a custom cache configuration from
- Adds `PluginBuilder::with_cache_disabled` to disable the cache when
initializing a plugin
- Setting `EXTISM_CACHE_CONFIG=""` will also disable caching
## Performance
With caching:
```
create/create_plugin time: [2.9079 ms 2.9139 ms 2.9200 ms]
change: [+3.2677% +3.6399% +3.9766%] (p = 0.00 < 0.20)
Change within noise threshold.
```
Compared to `main`:
```
create/create_plugin time: [26.089 ms 26.498 ms 26.923 ms]
change: [+0.1729% +2.0868% +4.1337%] (p = 0.04 < 0.20)
Change within noise threshold.
```
