- Adds `extism:host/env::http_headers` to access HTTP response headers
- Adds `PluginBuilder::with_http_response_headers` to enable response
headers from Rust
- Adds `extism_plugin_allow_http_response_headers` to enable response
headers using the C API
TODO:
- [x] Update a PDK to use `extism:host/env::http_headers` so I can test
this
Updates the kernel to only scan for free blocks before a `memory.grow`
operation, this should improve performance of `alloc` by not iterating
over every block for each allocation.
This is a rough POC for allowing people to whitelist a dir as readonly.
When the source path is prefixed with `ro:`, the dir is considered as
readonly. This preserved backward compatibility. This suggestion came up
in https://github.com/extism/go-sdk/pull/1#discussion_r1276700587
Readonly:
```rs
let manifest = Manifest::new([url])
.with_allowed_path("ro:D:/x/rust/fs/data".to_string(), "/data")
.with_config_key("path", "/data/data.txt");
```
```
trying to read file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n"
-----------------------------------------------------
trying to write file:
thread '<unnamed>' panicked at src\lib.rs:24:34:
called `Result::unwrap()` on an `Err` value: Os { code: 58, kind: Unsupported, message: "Not supported" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at runtime\examples\fs.rs:27:6:
called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace:
0: 0x234d2 - fs.wasm!__rust_start_panic
1: 0x232a1 - fs.wasm!rust_panic
2: 0x231da - fs.wasm!std::panicking::rust_panic_with_hook::hd3fb69bc0aea298a
3: 0x22467 - fs.wasm!std::panicking::begin_panic_handler::{{closure}}::h4d99b90b43f79472
4: 0x223ca - fs.wasm!std::sys_common::backtrace::__rust_end_short_backtrace::h5691573a73161cb1
5: 0x22bca - fs.wasm!rust_begin_unwind
6: 0x303e9 - fs.wasm!core::panicking::panic_fmt::hdb62f5cdb45533e4
7: 0x3234d - fs.wasm!core::result::unwrap_failed::h30d23efcc9e41efc
8: 0x36c2 - fs.wasm!fs::try_write::inner::h0b3b0df8e129f5cc
9: 0x29cd - fs.wasm!try_write
10: 0x35e4a - fs.wasm!try_write.command_export
note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
Caused by:
wasm trap: wasm `unreachable` instruction executed
Stack backtrace:
0: std::backtrace_rs::backtrace::dbghelp64::trace
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:99
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
2: std::backtrace::Backtrace::create
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:331
3: std::backtrace::Backtrace::capture
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:296
4: anyhow::error::impl$1::from<wasmtime_environ::trap_encoding::Trap>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\anyhow-1.0.86\src\error.rs:565
5: core::convert::impl$3::into<wasmtime_environ::trap_encoding::Trap,anyhow::Error>
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\convert\mod.rs:759
6: wasmtime_environ::impl$1::into_anyhow<wasmtime_environ::trap_encoding::Trap>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-environ-22.0.0\src\lib.rs:90
7: wasmtime::runtime::trap::from_runtime_box
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\trap.rs:118
8: wasmtime::runtime::func::invoke_wasm_and_catch_traps::closure$0<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597
9: enum2$<core::result::Result<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global> > >::map_err<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global>,anyhow::Error,wasmtime::runtime:
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:829
10: wasmtime::runtime::func::invoke_wasm_and_catch_traps<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597
11: wasmtime::runtime::func::Func::call_unchecked_raw<extism::current_plugin::CurrentPlugin>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1063
12: wasmtime::runtime::func::Func::call_unchecked<ref_mut$<wasmtime::runtime::store::context::StoreContextMut<extism::current_plugin::CurrentPlugin> > >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1049
13: wasmtime::runtime::func::Func::call_impl_do_call<extism::current_plugin::CurrentPlugin>
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1243
14: wasmtime::runtime::func::Func::call<ref_mut$<wasmtime::runtime::store::Store<extism::current_plugin::CurrentPlugin> > >
at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1002
15: extism::plugin::Plugin::raw_call<ref$<str$>,ref$<str$> >
at .\src\plugin.rs:753
16: extism::plugin::Plugin::call<ref$<str$>,ref$<str$>,ref$<str$> >
at .\src\plugin.rs:900
17: fs::main
at .\examples\fs.rs:25
18: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250
19: core::hint::black_box
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337
20: std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\sys_common\backtrace.rs:155
21: std::rt::lang_start::closure$0<tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:166
22: std::rt::lang_start_internal
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\rt.rs:148
23: std::rt::lang_start<tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:165
24: main
25: invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
26: __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
27: BaseThreadInitThunk
28: RtlUserThreadStart
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\panicking.rs:645
1: core::panicking::panic_fmt
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\panicking.rs:72
2: core::result::unwrap_failed
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\result.rs:1654
3: enum2$<core::result::Result<ref$<str$>,anyhow::Error> >::unwrap
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:1077
4: fs::main
at .\examples\fs.rs:25
5: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250
6: core::hint::black_box
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `D:\dylibso\extism\target\debug\examples\fs.exe` (exit code: 101)
```
Writable:
```rs
let manifest = Manifest::new([url])
.with_allowed_path("D:/x/rust/fs/data".to_string(), "/data")
.with_config_key("path", "/data/data.txt");
```
```
trying to read file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n"
-----------------------------------------------------
trying to write file:
"Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\nHello World at 1719851500.7803263s\n"
done!
```
Add an example of dynamically linking plugins and a benchmark that does
an apples-to-apples comparison of `reflect` using host functions vs.
`reflect` using a linked wasm module. (To my surprise, the host
functions are a _little bit faster_!)
Further flesh out our runtime benchmarks: add benchmarks for sending
bytes one-way into Wasm linear memory, as well as testing copy-in then
copy-out. This compliments our `reflect` benchmark; we can get a sense
of how expensive the sub-operations are via `echo` and `consume`.
(Notably, we're missing a `produce` or `emit` that purely generates
output and sends it to the host!)
The source of these plugins is available in `extism/plugins` as
`echo.wat` and `consume.wat` [1].
[1]: https://github.com/extism/plugins/pull/4/files
See #504
Removes CI for the in-repo SDKs, we can remove the actual code closer to
the 1.0 release.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
Same as #471 for `main` branch
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: zshipko <zshipko@users.noreply.github.com>
Adds support for host functions and cleans up some of the FFI code.
## API
To make a host function, you can pass a proc to `Function::new`:
```ruby
func = proc do |current_plugin, inputs, outputs, user_data|
input = current_plugin.input_as_bytes(inputs.first)
current_plugin.return_string(outputs.first, "#{input} #{user_data}")
end
f = Extism::Function.new('transform_string', [Extism::ValType::I64], [Extism::ValType::I64], func, 'My User Data')
plugin = Extism::Plugin.new(host_manifest, [f], true)
result = plugin.call('reflect_string', 'Hello, World!')
assert_equal result, 'Hello, World! My User Data'
```
If your function is in a module or a class, you can use
`method(name).to_proc`. Example:
```ruby
module Test
def self.my_function(current_plugin, inputs, outputs, user_data)
input = current_plugin.input_as_bytes(inputs.first)
current_plugin.return_string(outputs.first, "#{input} #{user_data}")
end
end
func = Test.method(:my_function).to_proc
f = Extism::Function.new('my_function', [Extism::ValType::I64], [Extism::ValType::I64], func, 'My User Data')
```
`current_plugin` is of the type CurrentPlugin which has some helpful
methods:
* `CurrentPlugin#memory_at_offset(int)` returns a `Memory` object given
a memory pointer
* `CurrentPlugin#free(Memory)` frees the memory
* `CurrentPlugin#alloc(int)` allocates new memory and returns a `Memory`
* `CurrentPlugin#input_as_bytes(Value)` returns the bytes for the given
input param
* `CurrentPlugin#return_bytes(Value, Array)` Sets the array of bytes to
the return for the given output value
* `CurrentPlugin#input_as_bytes(Value, String)` Sets the string to the
return for the given output value
It seems like our runtime initialization process is a little too
aggressive, this PR scales it back to initialize the runtime once if it
exists and only reinitializes when `_start` is called. This prevents
globals from being wiped out between plugin calls.
- Removes Runtime::cleanup
- Only initializes runtime once, or if `_start` is called
- Improves Haskell reactor support
See https://github.com/extism/go-sdk/pull/11 for the go-sdk PR.
This PR adds the `kernel` directory which contains a port of the Extism
memory allocator compiled to WebAssembly and removes
`runtime/src/memory.rs` completely.
Being able to re-use memory functions as a WASM module allows us to
begin to experiment with porting Extism to new runtimes!
This is in a draft state while I'm verifying some of these changes.
- New types:
- `ExtismValType` - Enum of WebAssembly types
- `ExtismValUnion` - A union of the possible WebAssembly types
- `ExtismVal` - A struct with `ExtismValType` and `ExtismValUnion`
- `ExtismFunction` - The host function wrapper type
- `ExtismFunctionType` - The type of the host function callback
- `ExtismCurrentPlugin` - Provides access to the currently running
plugin from inside a host function
- New functions:
- `extism_function_new` - Create a new `ExtismFunction`
- `extism_function_free` - Free an `ExtismFunction`
- `extism_current_plugin_memory`, `extism_current_plugin_memory_alloc`,
`extism_current_plugin_memory_free`,
`extism_current_plugin_memory_length` - Manage plugin memory from inside
a host functions
- Updated functions
- `extism_plugin_new` and `extsim_plugin_update` - now accept two extra
parameters for `ExtismFunction*` array and length of that array
## Notes
- Host functions take a user-data argument, which is owned by the
resulting `ExtismFunction` and will be cleaned up when
`extism_function_free` is called (if a cleanup function was passed in
with the user data)
- Host functions in every SDK require working with `ExtismVal` arguments
directly, this is pretty low-level for what is kind of a high-level
feature. We could work on adding some types to the SDKs that make
working with pointers to plugin data more accessible, maybe something
similar to how the Rust PDK handes input/output data.
- In each language the host functions more-or-less share a signature:
`(CurrentPlugin plugin, Val inputs[], Val outputs[], userData)`
- C, C++, OCaml and Go take a single userData argument but Python and
Node take a "rest" argument which allows passing any number of user-data
values
- Go requires the host function to be exported:
f9eb5ed839/go/main.go (L13-L26)
- Zig and Ruby should be relatively simple to add host functions to next
but I haven't really looked into Elixir, .NET or Java yet.
- Also closes#20
This PR updates `extism_output_get` to return an actual pointer to the
output value (`const uint8_t* extism_output_get(PluginIndex plugin)`
instead of `void extism_output_get(PluginIndex plugin, uint8_t *buffer,
uint64_t length)`), this pointer will only be valid until the next call,
but it makes it possible to access the output data without copying.
The input buffer is also not copied and the same issue applies:
the input buffer must not change during `call`.
Co-authored-by: Steve Manuel <steve@dylib.so>
