Compare commits

..

10 Commits

Author SHA1 Message Date
Benjamin Eckel
3d08c1e2e8 center the image 2024-01-09 08:14:39 -06:00
Steve Manuel
65464e5e13 feat: update README 2024-01-09 02:03:51 -07:00
Gavin Hayes
4012dd22de chore: kernel add Handle and Offset types to differentiate from Pointer (#652)
This cosmetic change to the kernel adds two `u64` types for usage where
`Pointer` was used and it is actually something else or is more
restrictive (a pointer to the start of a data block (handle) or a
relative offset).

Instead of adding `Offset` to match `Length` I'd prefer to use `u64`
directly when referring to those scalars, but not sure if the those
types are preferred?
2024-01-08 12:31:24 -05:00
Steve Manuel
211d55337d feat: add rust-protobuf support to extism-convert (#653)
(code from @zshipko, thank you!)

---------

Co-authored-by: Zach Shipko <zach@dylibso.com>
2024-01-05 16:36:40 -07:00
zach
431bc4d8af cleanup: improve wat detection (again) (#651)
Merged the other one too quickly, this PR is updated with some
additional feedback from @chrisdickinson:

              (Ooh, this could also start with `(;`)
https://github.com/extism/extism/pull/650#discussion_r1443419249

It's also updated to accommodate for modules that start with an open
paren followed by some whitespace. I doubt this covers all of the
permitted syntax but it should be good enough.
2024-01-05 14:58:47 -08:00
zach
cd4fc39655 cleanup: improve wat detection (#650)
- Update to ensure `is_wat` is only true when the input is a valid
string
- Update to allow leading whitespace in a wat file
2024-01-05 14:00:41 -08:00
zach
03e761908c feat: add Plugin::call_get_error_code to get Extism error code along with the error (#649)
- Adds `Plugin::call_get_error_code` to get the Extism error code when a
call fails
2024-01-05 12:52:31 -08:00
zach
26542d5740 feat(kernel): add extism_length_unsafe (#648)
- Adds `length_unsafe` function to the extism kernel, a more performant
`length` for known-valid memory handles

After this is merged I will update go-sdk and js-sdk too.

Closes #643
2024-01-03 09:29:05 -08:00
CosmicHorror
950a0f449f Toggle off default clap feature for cbindgen (#644)
The only feature for `cbindgen` is the `clap` feature for using it as a
standalone binary and isn't needed when using it as a library
2023-12-26 19:40:32 -08:00
zach
c8868c37d8 chore: update wasmtime bounds to include 16.0.0 (#642) 2023-12-20 14:37:43 -08:00
20 changed files with 263 additions and 330 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 465 KiB

BIN
.github/assets/logo-horizontal.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 465 KiB

126
README.md
View File

@@ -1,65 +1,111 @@
# [Extism](https://extism.org)
<div align="center">
<a href="https://extism.org">
<picture>
<source media="(prefers-color-scheme: dark)" srcset=".github/assets/logo-horizontal-darkmode.png">
<img alt="Extism - the WebAssembly framework" width="75%" style="max-width: 600px" src=".github/assets/logo-horizontal.png">
</picture>
</a>
[![Discord](https://img.shields.io/discord/1011124058408112148?color=%23404eed&label=Community%20Chat&logo=Discord&logoColor=%23404eed)](https://discord.gg/cx3usBCWnc)
[![Discord](https://img.shields.io/discord/1011124058408112148?color=%23404eed&label=Community%20Chat&logo=Discord&logoColor=%23404eed)](https://extism.org/discord)
![GitHub Org's stars](https://img.shields.io/github/stars/extism)
![GitHub all releases](https://img.shields.io/github/downloads/extism/extism/total)
![GitHub License](https://img.shields.io/github/license/extism/extism)
![GitHub release (with filter)](https://img.shields.io/github/v/release/extism/extism)
The universal plug-in system. Run WebAssembly extensions inside your app. Use idiomatic Host SDKs for [Go](https://github.com/extism/go-sdk#readme),
[Ruby](https://github.com/extism/ruby-sdk#readme),
[Python](https://github.com/extism/python-sdk#readme),
[JavaScript](https://github.com/extism/js-sdk#readme),
[Rust](/runtime/#readme),
[C](libextism/#readme),
[C++](https://github.com/extism/cpp-sdk/#readme),
[OCaml](https://github.com/extism/ocaml-sdk#readme),
[Haskell](https://github.com/extism/haskell-sdk#readme),
[PHP](https://github.com/extism/php-sdk#readme),
[Elixir](https://github.com/extism/elixir-sdk#readme),
[.NET](https://github.com/extism/dotnet-sdk#readme),
[Java](https://github.com/extism/java-sdk#readme),
[Zig](https://github.com/extism/zig-sdk#readme),
[D](https://github.com/extism/d-sdk#readme),
&amp; more (others coming soon).
</div>
Plug-in development kits (PDK) for plug-in authors supported in [Rust](https://github.com/extism/rust-pdk#readme), [AssemblyScript](https://github.com/extism/assemblyscript-pdk#readme), [Go](https://github.com/extism/go-pdk#readme), [C/C++](https://github.com/extism/c-pdk#readme), [Haskell](https://github.com/extism/haskell-pdk#readme), [JavaScript](https://github.com/extism/js-pdk#readme), [C#](https://github.com/extism/dotnet-pdk#readme), [F#](https://github.com/extism/dotnet-pdk#readme) and [Zig](https://github.com/extism/zig-pdk#readme).
# Overview
<p align="center">
<img style="width: 70%;" src="https://user-images.githubusercontent.com/7517515/210286900-39b144fd-1b26-4dd0-b7a9-2b5755bc174d.png" alt="Extism embedded SDK language support"/>
</p>
Extism is a lightweight framework for building with WebAssembly (Wasm). It
supports running Wasm code on servers, the edge, CLIs, IoT, browsers and
everything in between. Extism is designed to be "universal" in that it supports
a common interface, no matter where it runs.
Add a flexible, secure, and _bLaZiNg FaSt_ plug-in system to your project. Server, desktop, mobile, web, database -- you name it. Enable users to write and execute safe extensions to your software in **3 easy steps:**
> **Note:** One of the primary use cases for Extism is **building extensible
> software & plugins**. You want to be able to execute arbitrary, untrusted code
> from your users? Extism makes this safe and practical to do.
### 1. Import
Additionally, Extism adds some extra utilities on top of standard Wasm runtimes.
For example, we support persistent memory/module-scope variables, secure &
host-controlled HTTP without WASI, runtime limiters & timers, simpler host
function linking, and more. Extism users build:
Import an Extism Host SDK into your code as a library dependency.
- plug-in systems
- FaaS platforms
- code generators
- web applications
- & much more...
### 2. Integrate
# Run WebAssembly In Your App
Identify the place(s) in your code where some arbitrary logic should run (the plug-in!), returning your code some results.
Pick a SDK to import into your program, and refer to the documentation to get
started:
### 3. Execute
| Type | Language | Source Code | Package |
| ----------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| Rust SDK | <img alt="Rust SDK" src="https://extism.org/img/sdk-languages/rust.svg" width="50px"/> | https://github.com/extism/extism/tree/main/runtime | [Crates.io](https://crates.io/crates/extism) |
| JS SDK | <img alt="JS SDK" src="https://extism.org/img/sdk-languages/js.svg" width="50px"/> | https://github.com/extism/js-sdk <br/>(supports Web, Node, Deno & Bun!) | [NPM](https://www.npmjs.com/package/@extism/extism) |
| Elixir SDK | <img alt="Elixir SDK" src="https://extism.org/img/sdk-languages/elixir.svg" width="50px"/> | https://github.com/extism/elixir-sdk | [Hex](https://hex.pm/packages/extism) |
| Go SDK | <img alt="Go SDK" src="https://extism.org/img/sdk-languages/go.svg" width="50px"/> | https://github.com/extism/go-sdk | [Go mod](https://pkg.go.dev/github.com/extism/go-sdk) |
| Haskell SDK | <img alt="Haskell SDK" src="https://extism.org/img/sdk-languages/haskell.svg" width="50px"/> | https://github.com/extism/haskell-sdk | [Hackage](https://hackage.haskell.org/package/extism) |
| Java SDK | <img alt="Java SDK" src="https://extism.org/img/sdk-languages/java-android.svg" width="50px"/> | https://github.com/extism/java-sdk | [Sonatype](https://central.sonatype.com/artifact/org.extism.sdk/extism) |
| .NET SDK | <img alt=".NET SDK" src="https://extism.org/img/sdk-languages/dotnet.svg" width="50px"/> | https://github.com/extism/dotnet-sdk <br/>(supports C# & F#!) | [Nuget](https://www.nuget.org/packages/Extism.Sdk) |
| OCaml SDK | <img alt="OCaml SDK" src="https://extism.org/img/sdk-languages/ocaml.svg" width="50px"/> | https://github.com/extism/ocaml-sdk | [opam](https://opam.ocaml.org/packages/extism/) |
| PHP SDK | <img alt="PHP SDK" src="https://extism.org/img/sdk-languages/php.svg" width="50px"/> | https://github.com/extism/php-sdk | [Packagist](https://packagist.org/packages/extism/extism) |
| Python SDK | <img alt="Python SDK" src="https://extism.org/img/sdk-languages/python.svg" width="50px"/> | https://github.com/extism/python-sdk | [PyPi](https://pypi.org/project/extism/) |
| Ruby SDK | <img alt="Ruby SDK" src="https://extism.org/img/sdk-languages/ruby.svg" width="50px"/> | https://github.com/extism/ruby-sdk | [RubyGems](https://rubygems.org/gems/extism) |
| Zig SDK | <img alt="Zig SDK" src="https://extism.org/img/sdk-languages/zig.svg" width="50px"/> | https://github.com/extism/zig-sdk | N/A |
| C SDK | <img alt="C SDK" src="https://extism.org/img/sdk-languages/c.svg" width="50px"/> | https://github.com/extism/extism/tree/main/libextism | N/A |
| C++ SDK | <img alt="C++ SDK" src="https://extism.org/img/sdk-languages/cpp.svg" width="50px"/> | https://github.com/extism/cpp-sdk | N/A |
Load WebAssembly modules at any time in your app's lifetime and Extism will execute them in a secure sandbox, fully isolated from your program's memory.
# Compile WebAssembly to run in Extism Hosts
# API Status
Extism Hosts (running the SDK) must execute WebAssembly code that has a PDK
library compiled in to the `.wasm` binary. PDKs make it easy for plug-in /
extension code authors to read input from the host and return data back, read
provided configuration, set/get variables, make outbound HTTP calls if allowed,
and more.
**Please note:** This project still under active development and APIs are still changing. We are aiming for a stable 1.0 release in January, 2024.
The main branch may have breaking changes until that point, but if you starting today, a 1.0.0-rcx release is the best place to start.
Pick a PDK to import into your Wasm program, and refer to the documentation to
get started:
If you experience any problems or have any questions, please join our [Discord](https://discord.gg/cx3usBCWnc) and let us know.
Our community is very responsive and happy to help get you started.
| Type | Language | Source Code | Package |
| ------------------ | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------- |
| Rust PDK | <img alt="Rust PDK" src="https://extism.org/img/sdk-languages/rust.svg" width="50px"/> | https://github.com/extism/rust-pdk | [Crates.io](https://crates.io/crates/extism-pdk) |
| JS PDK | <img alt="JS PDK" src="https://extism.org/img/sdk-languages/js.svg" width="50px"/> | https://github.com/extism/js-pdk | N/A |
| Go PDK | <img alt="Go PDK" src="https://extism.org/img/sdk-languages/go.svg" width="50px"/> | https://github.com/extism/go-pdk | [Go mod](https://pkg.go.dev/github.com/extism/go-pdk) |
| Haskell PDK | <img alt="Haskell PDK" src="https://extism.org/img/sdk-languages/haskell.svg" width="50px"/> | https://github.com/extism/haskell-pdk | [Hackage](https://hackage.haskell.org/package/extism-pdk) |
| AssemblyScript PDK | <img alt="AssemblyScript PDK" src="https://extism.org/img/sdk-languages/assemblyscript.svg" width="50px"/> | https://github.com/extism/assemblyscript-pdk | [NPM](https://www.npmjs.com/package/@extism/as-pdk) |
| C PDK | <img alt="C PDK" src="https://extism.org/img/sdk-languages/c.svg" width="50px"/> | https://github.com/extism/c-pdk | N/A |
| Zig PDK | <img alt="Zig PDK" src="https://extism.org/img/sdk-languages/zig.svg" width="50px"/> | https://github.com/extism/zig-pdk | N/A |
| .NET PDK | <img alt=".NET PDK" src="https://extism.org/img/sdk-languages/dotnet.svg" width="50px"/> | https://github.com/extism/dotnet-pdk <br/>(supports C# & F#!) | N/A |
# Support
## Discord
If you experience any problems or have any questions, please join our
[Discord](https://extism.org/discord) and let us know. Our community is very
responsive and happy to help get you started.
## Usage
Head to the [project website](https://extism.org) for more information and docs. Also, consider reading an [overview](https://extism.org/docs/overview) of Extism and its goals & approach.
Head to the [project website](https://extism.org) for more information and docs.
Also, consider reading an [overview](https://extism.org/docs/overview) of Extism
and its goals & approach.
## Contribution
Thank you for considering a contribution to Extism, we are happy to help you make a PR or find something to work on!
Thank you for considering a contribution to Extism, we are happy to help you
make a PR or find something to work on!
The easiest way to start would be to join the [Discord](https://discord.gg/cx3usBCWnc) or open an issue on the [`extism/proposals`](https://github.com/extism/proposals) issue tracker, which can eventually become an Extism Improvement Proposal (EIP).
The easiest way to start would be to join the
[Discord](https://extism.org/discord) or open an issue on the
[`extism/proposals`](https://github.com/extism/proposals) issue tracker, which
can eventually become an Extism Improvement Proposal (EIP).
For more information, please read the
[Contributing](https://extism.org/docs/concepts/contributing) guide.
---
@@ -68,8 +114,8 @@ The easiest way to start would be to join the [Discord](https://discord.gg/cx3us
Extism is an open-source product from the team at:
<p align="left">
<a href="https://dylib.so" _target="blanks"><img width="200px" src="https://user-images.githubusercontent.com/7517515/198204119-5afdebb9-a5d8-4322-bd2a-46179c8d7b24.svg"/></a>
<a href="https://dylibso.com" _target="blanks"><img width="200px" src="https://user-images.githubusercontent.com/7517515/198204119-5afdebb9-a5d8-4322-bd2a-46179c8d7b24.svg"/></a>
</p>
_Reach out and tell us what you're building! We'd love to help._
_Reach out and tell us what you're building! We'd love to help:_
<a href="mailto:hello@dylibso.com">hello@dylibso.com</a>

View File

@@ -14,6 +14,7 @@ anyhow = "1.0.75"
base64 = "~0.21"
bytemuck = {version = "1.14.0", optional = true }
prost = { version = "0.12.0", optional = true }
protobuf = { version = "3.2.0", optional = true }
rmp-serde = { version = "1.1.2", optional = true }
serde = "1.0.186"
serde_json = "1.0.105"
@@ -22,7 +23,6 @@ serde_json = "1.0.105"
serde = { version = "1.0.186", features = ["derive"] }
[features]
default = ["msgpack", "protobuf", "raw"]
default = ["msgpack", "prost", "raw"]
msgpack = ["rmp-serde"]
protobuf = ["prost"]
raw = ["bytemuck"]

View File

@@ -112,19 +112,19 @@ impl FromBytesOwned for Base64<String> {
/// Protobuf encoding
///
/// Allows for `prost` Protobuf messages to be used as arguments to Extism plugin calls
#[cfg(feature = "protobuf")]
#[cfg(feature = "prost")]
#[derive(Debug)]
pub struct Protobuf<T: prost::Message>(pub T);
pub struct Prost<T: prost::Message>(pub T);
#[cfg(feature = "protobuf")]
impl<T: prost::Message> From<T> for Protobuf<T> {
#[cfg(feature = "prost")]
impl<T: prost::Message> From<T> for Prost<T> {
fn from(data: T) -> Self {
Self(data)
}
}
#[cfg(feature = "protobuf")]
impl<'a, T: prost::Message> ToBytes<'a> for Protobuf<T> {
#[cfg(feature = "prost")]
impl<'a, T: prost::Message> ToBytes<'a> for Prost<T> {
type Bytes = Vec<u8>;
fn to_bytes(&self) -> Result<Self::Bytes, Error> {
@@ -132,10 +132,32 @@ impl<'a, T: prost::Message> ToBytes<'a> for Protobuf<T> {
}
}
#[cfg(feature = "protobuf")]
impl<T: Default + prost::Message> FromBytesOwned for Protobuf<T> {
#[cfg(feature = "prost")]
impl<T: Default + prost::Message> FromBytesOwned for Prost<T> {
fn from_bytes_owned(data: &[u8]) -> Result<Self, Error> {
Ok(Protobuf(T::decode(data)?))
Ok(Prost(T::decode(data)?))
}
}
/// Protobuf encoding
///
/// Allows for `rust-protobuf` Protobuf messages to be used as arguments to Extism plugin calls
#[cfg(feature = "protobuf")]
pub struct Protobuf<T: protobuf::Message>(pub T);
#[cfg(feature = "protobuf")]
impl<'a, T: protobuf::Message> ToBytes<'a> for Protobuf<T> {
type Bytes = Vec<u8>;
fn to_bytes(&self) -> Result<Self::Bytes, Error> {
Ok(self.0.write_to_bytes()?)
}
}
#[cfg(feature = "protobuf")]
impl<T: Default + protobuf::Message> FromBytesOwned for Protobuf<T> {
fn from_bytes_owned(data: &[u8]) -> Result<Self, Error> {
Ok(Protobuf(T::parse_from_bytes(data)?))
}
}

View File

@@ -18,6 +18,9 @@ pub use encoding::{Base64, Json};
#[cfg(feature = "msgpack")]
pub use encoding::Msgpack;
#[cfg(feature = "prost")]
pub use encoding::Prost;
#[cfg(feature = "protobuf")]
pub use encoding::Protobuf;

View File

@@ -39,7 +39,7 @@
use core::sync::atomic::*;
pub type Pointer = u64;
pub type Length = u64;
pub type Handle = u64;
/// WebAssembly page size
const PAGE_SIZE: usize = 65536;
@@ -81,13 +81,13 @@ pub struct MemoryRoot {
/// Offset of error block
pub error: AtomicU64,
/// Input position in memory
pub input_offset: Pointer,
pub input_offset: Handle,
/// Input length
pub input_length: Length,
pub input_length: u64,
/// Output position in memory
pub output_offset: Pointer,
/// Output length
pub output_length: Length,
pub output_length: u64,
/// A pointer to the start of the first block
pub blocks: [MemoryBlock; 0],
}
@@ -206,7 +206,7 @@ impl MemoryRoot {
// is used to avoid loading the allocators position more than once when performing an allocation.
unsafe fn find_free_block(
&mut self,
length: Length,
length: u64,
self_position: u64,
) -> Option<&'static mut MemoryBlock> {
// Get the first block
@@ -252,7 +252,7 @@ impl MemoryRoot {
/// Create a new `MemoryBlock`, when `Some(block)` is returned, `block` will contain at least enough room for `length` bytes
/// but may be as large as `length` + `BLOCK_SPLIT_SIZE` bytes. When `None` is returned the allocation has failed.
pub unsafe fn alloc(&mut self, length: Length) -> Option<&'static mut MemoryBlock> {
pub unsafe fn alloc(&mut self, length: u64) -> Option<&'static mut MemoryBlock> {
let self_position = self.position.load(Ordering::Acquire);
let self_length = self.length.load(Ordering::Acquire);
let b = self.find_free_block(length, self_position);
@@ -350,21 +350,21 @@ impl MemoryBlock {
/// Allocate a block of memory and return the offset
#[no_mangle]
pub unsafe fn alloc(n: Length) -> Pointer {
pub unsafe fn alloc(n: u64) -> Handle {
if n == 0 {
return 0;
}
let region = MemoryRoot::new();
let block = region.alloc(n);
match block {
Some(block) => block.data.as_mut_ptr() as Pointer,
Some(block) => block.data.as_mut_ptr() as Handle,
None => 0,
}
}
/// Free allocated memory
#[no_mangle]
pub unsafe fn free(p: Pointer) {
pub unsafe fn free(p: Handle) {
if p == 0 {
return;
}
@@ -382,13 +382,42 @@ pub unsafe fn free(p: Pointer) {
}
/// Get the length of an allocated memory block
///
/// Note: this should only be called on memory handles returned
/// by a call to `alloc` - it will return garbage on invalid offsets
#[no_mangle]
pub unsafe fn length(p: Pointer) -> Length {
pub unsafe fn length_unsafe(p: Handle) -> u64 {
if p == 0 {
return 0;
}
if !MemoryRoot::pointer_in_bounds_fast(p) {
return 0;
}
let ptr = p - core::mem::size_of::<MemoryBlock>() as u64;
let block = &mut *(ptr as *mut MemoryBlock);
// Simplest sanity check to verify the pointer is a block
if block.status.load(Ordering::Acquire) != MemoryStatus::Active as u8 {
return 0;
}
block.used as u64
}
/// Get the length but returns 0 if the offset is not a valid handle.
///
/// Note: this function walks each node in the allocations list, which ensures correctness, but is also
/// slow
#[no_mangle]
pub unsafe fn length(p: Pointer) -> u64 {
if p == 0 {
return 0;
}
if let Some(block) = MemoryRoot::new().find_block(p) {
block.used as Length
block.used as u64
} else {
0
}
@@ -416,24 +445,24 @@ pub unsafe fn load_u64(p: Pointer) -> u64 {
/// Load a byte from the input data
#[no_mangle]
pub unsafe fn input_load_u8(p: Pointer) -> u8 {
pub unsafe fn input_load_u8(offset: u64) -> u8 {
let root = MemoryRoot::new();
#[cfg(feature = "bounds-checking")]
if p >= root.input_length {
if offset >= root.input_length {
return 0;
}
*((root.input_offset + p) as *mut u8)
*((root.input_offset + offset) as *mut u8)
}
/// Load a u64 from the input data
#[no_mangle]
pub unsafe fn input_load_u64(p: Pointer) -> u64 {
pub unsafe fn input_load_u64(offset: u64) -> u64 {
let root = MemoryRoot::new();
#[cfg(feature = "bounds-checking")]
if p + core::mem::size_of::<u64>() as Pointer > root.input_length {
if offset + core::mem::size_of::<u64>() as u64 > root.input_length {
return 0;
}
*((root.input_offset + p) as *mut u64)
*((root.input_offset + offset) as *mut u64)
}
/// Write a byte in Extism-managed memory
@@ -457,22 +486,24 @@ pub unsafe fn store_u64(p: Pointer, x: u64) {
}
/// Set the range of the input data in memory
/// h must always be a handle so that length works on it
/// len must match length(handle)
#[no_mangle]
pub unsafe fn input_set(p: Pointer, len: Length) {
pub unsafe fn input_set(h: Handle, len: u64) {
let root = MemoryRoot::new();
#[cfg(feature = "bounds-checking")]
{
if !root.pointer_in_bounds(p) || !root.pointer_in_bounds(p + len - 1) {
if !root.pointer_in_bounds(h) || !root.pointer_in_bounds(h + len - 1) {
return;
}
}
root.input_offset = p;
root.input_offset = h;
root.input_length = len;
}
/// Set the range of the output data in memory
#[no_mangle]
pub unsafe fn output_set(p: Pointer, len: Length) {
pub unsafe fn output_set(p: Pointer, len: u64) {
let root = MemoryRoot::new();
#[cfg(feature = "bounds-checking")]
{
@@ -486,25 +517,25 @@ pub unsafe fn output_set(p: Pointer, len: Length) {
/// Get the input length
#[no_mangle]
pub fn input_length() -> Length {
pub fn input_length() -> u64 {
unsafe { MemoryRoot::new().input_length }
}
/// Get the input offset in Exitsm-managed memory
#[no_mangle]
pub fn input_offset() -> Length {
pub fn input_offset() -> Handle {
unsafe { MemoryRoot::new().input_offset }
}
/// Get the output length
#[no_mangle]
pub fn output_length() -> Length {
pub fn output_length() -> u64 {
unsafe { MemoryRoot::new().output_length }
}
/// Get the output offset in Extism-managed memory
#[no_mangle]
pub unsafe fn output_offset() -> Length {
pub unsafe fn output_offset() -> Pointer {
MemoryRoot::new().output_offset
}
@@ -516,30 +547,30 @@ pub unsafe fn reset() {
/// Set the error message offset
#[no_mangle]
pub unsafe fn error_set(ptr: Pointer) {
pub unsafe fn error_set(h: Handle) {
let root = MemoryRoot::new();
// Allow ERROR to be set to 0
if ptr == 0 {
root.error.store(ptr, Ordering::SeqCst);
if h == 0 {
root.error.store(h, Ordering::SeqCst);
return;
}
#[cfg(feature = "bounds-checking")]
if !root.pointer_in_bounds(ptr) {
if !root.pointer_in_bounds(h) {
return;
}
root.error.store(ptr, Ordering::SeqCst);
root.error.store(h, Ordering::SeqCst);
}
/// Get the error message offset, if it's `0` then no error has been set
#[no_mangle]
pub unsafe fn error_get() -> Pointer {
pub unsafe fn error_get() -> Handle {
MemoryRoot::new().error.load(Ordering::SeqCst)
}
/// Get the position of the allocator, this can be used as an indication of how many bytes are currently in-use
#[no_mangle]
pub unsafe fn memory_bytes() -> Length {
pub unsafe fn memory_bytes() -> u64 {
MemoryRoot::new().length.load(Ordering::Acquire)
}

View File

@@ -9,8 +9,8 @@ repository.workspace = true
version.workspace = true
[dependencies]
wasmtime = ">= 14.0.0, < 16.0.0"
wasmtime-wasi = ">= 14.0.0, < 16.0.0"
wasmtime = ">= 14.0.0, < 17.0.0"
wasmtime-wasi = ">= 14.0.0, < 17.0.0"
anyhow = "1"
serde = {version = "1", features = ["derive"]}
serde_json = "1"
@@ -33,7 +33,7 @@ register-filesystem = [] # enables wasm to be loaded from disk
http = ["ureq"] # enables extism_http_request
[build-dependencies]
cbindgen = "0.26"
cbindgen = { version = "0.26", default-features = false }
[dev-dependencies]
criterion = "0.5.1"

View File

@@ -173,13 +173,6 @@ void extism_function_free(ExtismFunction *f);
*/
void extism_function_set_namespace(ExtismFunction *ptr, const char *namespace_);
/**
* Set the cost of an `ExtismFunction`, when set to 0 this has no effect, when set to `1` this will add
* the runtime of the function back to the plugin timer.
*/
void extism_function_set_cost(ExtismFunction *ptr,
double cost);
/**
* Create a new plugin with host functions, the functions passed to this function no longer need to be manually freed using
*

View File

@@ -246,6 +246,26 @@ impl CurrentPlugin {
Ok(len)
}
pub fn memory_length_unsafe(&mut self, offs: u64) -> Result<u64, Error> {
let (linker, mut store) = self.linker_and_store();
let output = &mut [Val::I64(0)];
if let Some(f) = linker.get(&mut store, EXTISM_ENV_MODULE, "length_unsafe") {
f.into_func()
.unwrap()
.call(&mut store, &[Val::I64(offs as i64)], output)?;
} else {
anyhow::bail!("unable to locate an extism kernel function: length_unsafe",)
}
let len = output[0].unwrap_i64() as u64;
trace!(
plugin = self.id.to_string(),
"memory_length_unsafe({}) = {}",
offs,
len
);
Ok(len)
}
/// Access a plugin's variables
pub fn vars(&self) -> &std::collections::BTreeMap<String, Vec<u8>> {
&self.vars

Binary file not shown.

View File

@@ -155,7 +155,7 @@ unsafe impl<T> Sync for UserData<T> {}
unsafe impl Send for CPtr {}
unsafe impl Sync for CPtr {}
pub(crate) type FunctionInner = dyn Fn(wasmtime::Caller<CurrentPlugin>, &[wasmtime::Val], &mut [wasmtime::Val]) -> Result<(), Error>
type FunctionInner = dyn Fn(wasmtime::Caller<CurrentPlugin>, &[wasmtime::Val], &mut [wasmtime::Val]) -> Result<(), Error>
+ Sync
+ Send;
@@ -174,9 +174,6 @@ pub struct Function {
/// Function handle
pub(crate) f: Arc<FunctionInner>,
/// Function cost (in terms of time)
pub(crate) cost: f64,
/// UserData
pub(crate) _user_data: UserDataHandle,
}
@@ -207,12 +204,10 @@ impl Function {
ty,
f: Arc::new(
move |mut caller: Caller<_>, inp: &[Val], outp: &mut [Val]| {
let plugin = caller.data_mut();
let x = data.clone();
f(plugin, inp, outp, x)
f(caller.data_mut(), inp, outp, x)
},
),
cost: 0.0,
namespace: None,
_user_data: match &user_data {
UserData::C(ptr) => UserDataHandle::C(ptr.clone()),
@@ -244,23 +239,6 @@ impl Function {
self
}
/// Function cost
pub fn cost(&self) -> f64 {
self.cost
}
/// Set host function cost
pub fn set_cost(&mut self, cost: f64) {
trace!("Setting cost for {} to {cost}", self.name);
self.cost = cost;
}
/// Update host function cost
pub fn with_cost(mut self, cost: f64) -> Self {
self.set_cost(cost);
self
}
/// Get function type
pub fn ty(&self) -> &wasmtime::FuncType {
&self.ty

View File

@@ -14,7 +14,6 @@ pub(crate) mod manifest;
pub(crate) mod pdk;
mod plugin;
mod plugin_builder;
mod timeout_manager;
mod timer;
/// Extism C API
@@ -28,7 +27,6 @@ pub use plugin::{CancelHandle, Plugin, WasmInput, EXTISM_ENV_MODULE, EXTISM_USER
pub use plugin_builder::{DebugOptions, PluginBuilder};
pub(crate) use internal::{Internal, Wasi};
pub(crate) use timeout_manager::TimeoutManager;
pub(crate) use timer::{Timer, TimerAction};
pub(crate) use tracing::{debug, error, trace, warn};

View File

@@ -117,10 +117,17 @@ pub(crate) fn load(
match input {
WasmInput::Data(data) => {
let has_magic = data.len() >= 4 && data[0..4] == WASM_MAGIC;
let is_wat = data.starts_with(b"(module") || data.starts_with(b";;");
let s = std::str::from_utf8(&data);
let is_wat = s.is_ok_and(|s| {
let s = s.trim_start();
let starts_with_module = s.len() > 2
&& data[0] == b'(' // First character is `(`
&& s[1..].trim_start().starts_with("module"); // Then `module` (after any whitespace)
starts_with_module || s.starts_with(";;") || s.starts_with("(;")
});
if !has_magic && !is_wat {
trace!("Loading manifest");
if let Ok(s) = std::str::from_utf8(&data) {
if let Ok(s) = s {
let t = if let Ok(t) = toml::from_str::<extism_manifest::Manifest>(s) {
trace!("Manifest is TOML");
modules(engine, &t, &mut mods)?;

View File

@@ -238,20 +238,6 @@ impl Plugin {
CurrentPlugin::new(manifest, with_wasi, available_pages, id)?,
);
store.set_epoch_deadline(1);
store.call_hook(|data, hook| {
if hook.entering_host() {
let tx = Timer::tx();
tx.send(TimerAction::EnterHost {
id: data.id.clone(),
})?;
} else if hook.exiting_host() {
let tx = Timer::tx();
tx.send(TimerAction::ExitHost {
id: data.id.clone(),
})?;
}
Ok(())
});
let mut linker = Linker::new(&engine);
linker.allow_shadowing(true);
@@ -298,13 +284,9 @@ impl Plugin {
for f in &mut imports {
let name = f.name().to_string();
let ns = f.namespace().unwrap_or(EXTISM_USER_MODULE);
let cost = f.cost();
let inner: &function::FunctionInner = unsafe { &*(f.f.as_ref() as *const _) };
linker.func_new(ns, &name, f.ty().clone(), move |caller, params, results| {
let _timeout =
TimeoutManager::new(caller.data()).map(|x| x.with_cost(cost.clone()));
inner(caller, params, results)
})?;
unsafe {
linker.func_new(ns, &name, f.ty().clone(), &*(f.f.as_ref() as *const _))?;
}
}
let instance_pre = linker.instantiate_pre(main)?;
@@ -842,6 +824,24 @@ impl Plugin {
.and_then(move |_| self.output())
}
/// Similar to `Plugin::call`, but returns the Extism error code along with the
/// `Error`. It is assumed if `Ok(_)` is returned that the error code was `0`.
///
/// All Extism plugin calls return an error code, `Plugin::call` consumes the error code,
/// while `Plugin::call_get_error_code` preserves it - this function should only be used
/// when you need to inspect the actual return value of a plugin function when it fails.
pub fn call_get_error_code<'a, 'b, T: ToBytes<'a>, U: FromBytes<'b>>(
&'b mut self,
name: impl AsRef<str>,
input: T,
) -> Result<U, (Error, i32)> {
let lock = self.instance.clone();
let mut lock = lock.lock().unwrap();
let data = input.to_bytes().map_err(|e| (e, -1))?;
self.raw_call(&mut lock, name, data)
.and_then(move |_| self.output().map_err(|e| (e, -1)))
}
/// Get a `CancelHandle`, which can be used from another thread to cancel a running plugin
pub fn cancel_handle(&self) -> CancelHandle {
self.cancel_handle.clone()

View File

@@ -258,18 +258,6 @@ pub unsafe extern "C" fn extism_function_set_namespace(
}
}
/// Set the cost of an `ExtismFunction`, when set to 0 this has no effect, when set to `1` this will add
/// the runtime of the function back to the plugin timer.
#[no_mangle]
pub unsafe extern "C" fn extism_function_set_cost(ptr: *mut ExtismFunction, cost: f64) {
let f = &mut *ptr;
if let Some(x) = f.0.get_mut() {
x.set_cost(cost);
} else {
debug!("Trying to set the cost of already registered function")
}
}
/// Create a new plugin with host functions, the functions passed to this function no longer need to be manually freed using
///
/// `wasm`: is a WASM module (wat or wasm) or a JSON encoded manifest

View File

@@ -22,6 +22,20 @@ fn extism_length<T>(mut store: &mut wasmtime::Store<T>, instance: &mut Instance,
out[0].unwrap_i64() as u64
}
fn extism_length_unsafe<T>(
mut store: &mut wasmtime::Store<T>,
instance: &mut Instance,
p: u64,
) -> u64 {
let out = &mut [Val::I64(0)];
instance
.get_func(&mut store, "length_unsafe")
.unwrap()
.call(&mut store, &[Val::I64(p as i64)], out)
.unwrap();
out[0].unwrap_i64() as u64
}
fn extism_load_u8<T>(mut store: &mut wasmtime::Store<T>, instance: &mut Instance, p: u64) -> u8 {
let out = &mut [Val::I32(0)];
instance
@@ -173,6 +187,7 @@ fn test_kernel_allocations() {
let first_alloc = p;
assert!(p > 0);
assert_eq!(extism_length(&mut store, instance, p), 1);
assert_eq!(extism_length_unsafe(&mut store, instance, p), 1);
extism_free(&mut store, instance, p);
// 2 bytes
@@ -180,18 +195,21 @@ fn test_kernel_allocations() {
assert!(x > 0);
assert!(x != p);
assert_eq!(extism_length(&mut store, instance, x), 2);
assert_eq!(extism_length_unsafe(&mut store, instance, x), 2);
extism_free(&mut store, instance, x);
for i in 0..64 {
let p = extism_alloc(&mut store, instance, 64 - i);
assert!(p > 0);
assert_eq!(extism_length(&mut store, instance, p), 64 - i);
assert_eq!(extism_length_unsafe(&mut store, instance, p), 64 - i);
extism_free(&mut store, instance, p);
// should re-use the last allocation
let q = extism_alloc(&mut store, instance, 64 - i);
assert_eq!(p, q);
assert_eq!(extism_length(&mut store, instance, q), 64 - i);
assert_eq!(extism_length_unsafe(&mut store, instance, q), 64 - i);
extism_free(&mut store, instance, q);
}

View File

@@ -235,43 +235,6 @@ fn test_timeout() {
assert!(err == "timeout");
}
fn hello_world_timeout_manager(
plugin: &mut CurrentPlugin,
inputs: &[Val],
outputs: &mut [Val],
_user_data: UserData<()>,
) -> Result<(), Error> {
let mgr = plugin.timeout_manager().map(|x| x.add_on_drop());
std::thread::sleep(std::time::Duration::from_secs(3));
outputs[0] = inputs[0].clone();
drop(mgr);
Ok(())
}
#[test]
fn test_timeout_manager() {
let f = Function::new(
"hello_world",
[PTR],
[PTR],
UserData::default(),
hello_world_timeout_manager,
);
let manifest = Manifest::new([extism_manifest::Wasm::data(WASM)])
.with_timeout(std::time::Duration::from_secs(1));
let mut plugin = Plugin::new(manifest, [f], true).unwrap();
let start = std::time::Instant::now();
let output: Result<&[u8], Error> = plugin.call("count_vowels", "testing");
println!("Result {:?}", output);
assert!(output.is_ok());
let end = std::time::Instant::now();
let time = end - start;
println!("Plugin ran for {:?}", time);
assert!(time.as_secs() >= 3);
}
typed_plugin!(pub TestTypedPluginGenerics {
count_vowels<T: FromBytes<'a>>(&str) -> T
});

View File

@@ -1,59 +0,0 @@
use crate::*;
/// `TimeoutManager` is used to control `Plugin` timeouts from within host functions
///
/// It can be used to add or subtract time from a plug-in's timeout. If a plugin is not
/// configured to have a timeout then this will have no effect.
pub(crate) struct TimeoutManager {
start_time: std::time::Instant,
id: uuid::Uuid,
tx: std::sync::mpsc::Sender<TimerAction>,
cost: f64,
}
impl TimeoutManager {
/// Create a new `TimeoutManager` from the `CurrentPlugin`, this will return `None` if no timeout
/// is configured
pub fn new(plugin: &CurrentPlugin) -> Option<TimeoutManager> {
plugin.manifest.timeout_ms.map(|_| TimeoutManager {
start_time: std::time::Instant::now(),
id: plugin.id.clone(),
tx: Timer::tx(),
cost: 0.0,
})
}
/// Add the amount of time this value has existed to the configured timeout
pub fn add_elapsed(&mut self) -> Result<(), Error> {
let cost = self.cost.abs();
let d = self.start_time.elapsed().mul_f64(cost);
let mut d: timer::ExtendTimeout = d.into();
if self.cost.is_sign_negative() {
d = -d;
}
self.tx.send(TimerAction::Extend {
id: self.id.clone(),
duration: d,
})?;
self.start_time = std::time::Instant::now();
Ok(())
}
pub fn with_cost(mut self, cost: f64) -> Self {
self.cost = cost;
self
}
}
impl Drop for TimeoutManager {
fn drop(&mut self) {
let x = self.add_elapsed();
if let Err(e) = x {
error!(
plugin = self.id.to_string(),
"unable to extend timeout: {}",
e.to_string()
);
}
}
}

View File

@@ -1,51 +1,17 @@
use crate::*;
#[derive(Debug)]
pub(crate) struct ExtendTimeout {
duration: std::time::Duration,
negative: bool,
}
impl From<std::time::Duration> for ExtendTimeout {
fn from(value: std::time::Duration) -> Self {
ExtendTimeout {
duration: value,
negative: false,
}
}
}
impl std::ops::Neg for ExtendTimeout {
type Output = ExtendTimeout;
fn neg(mut self) -> Self::Output {
self.negative = !self.negative;
self
}
}
pub(crate) enum TimerAction {
Start {
id: uuid::Uuid,
engine: Engine,
duration: Option<std::time::Duration>,
},
Extend {
id: uuid::Uuid,
duration: ExtendTimeout,
},
Stop {
id: uuid::Uuid,
},
Cancel {
id: uuid::Uuid,
},
EnterHost {
id: uuid::Uuid,
},
ExitHost {
id: uuid::Uuid,
},
Shutdown,
}
@@ -65,8 +31,6 @@ extern "C" fn cleanup_timer() {
static mut TIMER: std::sync::Mutex<Option<Timer>> = std::sync::Mutex::new(None);
type TimerMap = std::collections::BTreeMap<uuid::Uuid, (Engine, Option<std::time::Instant>)>;
impl Timer {
pub(crate) fn tx() -> std::sync::mpsc::Sender<TimerAction> {
let mut timer = match unsafe { TIMER.lock() } {
@@ -85,8 +49,7 @@ impl Timer {
pub fn init(timer: &mut Option<Timer>) -> std::sync::mpsc::Sender<TimerAction> {
let (tx, rx) = std::sync::mpsc::channel();
let thread = std::thread::spawn(move || {
let mut plugins = TimerMap::new();
let mut in_host = TimerMap::new();
let mut plugins = std::collections::BTreeMap::new();
macro_rules! handle {
($x:expr) => {
@@ -107,16 +70,12 @@ impl Timer {
TimerAction::Stop { id } => {
trace!(plugin = id.to_string(), "handling stop event");
plugins.remove(&id);
in_host.remove(&id);
}
TimerAction::Cancel { id } => {
trace!(plugin = id.to_string(), "handling cancel event");
if let Some((engine, _)) = plugins.remove(&id) {
engine.increment_epoch();
}
if let Some((engine, _)) = in_host.remove(&id) {
engine.increment_epoch();
}
}
TimerAction::Shutdown => {
trace!("Shutting down timer");
@@ -124,42 +83,8 @@ impl Timer {
trace!(plugin = id.to_string(), "handling shutdown event");
engine.increment_epoch();
}
for (id, (engine, _)) in in_host.iter() {
trace!(plugin = id.to_string(), "handling shutdown event");
engine.increment_epoch();
}
return;
}
TimerAction::Extend { id, duration } => {
if let Some((_engine, Some(timeout))) = plugins.get_mut(&id) {
let x = if duration.negative {
timeout.checked_sub(duration.duration)
} else {
timeout.checked_add(duration.duration)
};
if let Some(t) = x {
*timeout = t;
} else {
error!(
plugin = id.to_string(),
"unable to extend timeout by {:?}", duration.duration
);
}
}
}
TimerAction::EnterHost { id } => {
trace!(plugin = id.to_string(), "enter host function");
if let Some(x) = plugins.remove(&id) {
in_host.insert(id, x);
}
}
TimerAction::ExitHost { id } => {
trace!(plugin = id.to_string(), "exit host function");
if let Some(x) = in_host.remove(&id) {
plugins.insert(id, x);
}
}
}
};
}
@@ -171,10 +96,6 @@ impl Timer {
}
}
for x in rx.try_iter() {
handle!(x)
}
plugins = plugins
.into_iter()
.filter(|(_k, (engine, end))| {
@@ -188,6 +109,10 @@ impl Timer {
true
})
.collect();
for x in rx.try_iter() {
handle!(x)
}
}
});
*timer = Some(Timer {