* improvement(repo): restructuring to make realtime image narrower scoped
* improvements
* chore(repo): rebase fixes and quality improvements for realtime split
Addresses merge-time issues and gaps from the realtime app split:
- Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks
- Restore README branding, fix AGENTS.md script reference
- Restore TSDoc on workflow-persistence subblocks helpers
- Use toError() from @sim/utils/errors in save.ts
- Add vitest config + local mocks so @sim/audit tests run standalone
- Move socket.io-client to devDependencies in apps/realtime
- Add missing package COPY steps to docker/app.Dockerfile
- Add check:boundaries/check:realtime-prune scripts and wire into CI
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(security): consolidate crypto primitives into @sim/security
Move general-purpose crypto primitives out of apps/sim into the
@sim/security package so both apps/sim and apps/realtime can share them.
@sim/security exports (all pure, dependency-free):
./compare safeCompare (constant-time HMAC-wrapped equality)
./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format)
./hash sha256Hex
./tokens generateSecureToken (base64url)
Migrate apps/sim call sites to use these + @sim/utils helpers:
crypto.randomUUID() -> generateId() from @sim/utils/id
createHash('sha256').digest -> sha256Hex
timingSafeEqual on hashed hex -> safeCompare
new Promise(setTimeout) -> sleep from @sim/utils/helpers
No behavior change: encryption format, digest output, and token
length are preserved exactly.
* refactor(copilot): use toError in remaining otel/finalize sites
Replace the last two `error instanceof Error ? error : new Error(String(error))`
patterns with toError from @sim/utils/errors. Completes the sweep of clean
candidates — no behavior change.
* refactor(security): consolidate HMAC-SHA256 primitives into @sim/security
Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates
15 webhook providers plus 5 other hot paths (deployment token signing,
outbound webhook requests, workspace notification delivery, notification
test route, Shopify OAuth callback) off bare `createHmac` calls. Secret
parameter accepts `string | Buffer` to cover base64-decoded Svix-style
secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and
Textract tools intentionally retains direct `createHmac` usage — its
multi-step key derivation chain doesn't fit a generic helper.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(packages): post-audit test + packaging polish
- Add safeCompare unit tests (identity, length mismatch, hex-nibble diff).
- Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract.
- Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports.
- Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained.
- Document the data-field invariant in load.ts for loop/parallel subflow patching.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(realtime): address PR review feedback
- Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage
(env.PORT already defaults to 3002 via zod schema).
- Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over
the schema default for PORT; keeps SOCKET_PORT functional as an
override instead of dead code.
- Add dedicated type-check CI step for @sim/realtime so TS errors
surface pre-deploy (the Dockerfile runs source TS via Bun and has
no implicit build-time type check).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(realtime): remove unused SOCKET_PORT env var
SOCKET_PORT has lived in the socket server since the June 2025 refactor
but was never actually set in any deploy config — docker-compose.prod,
helm values/templates, .env.example, and docs all use PORT or the 3002
default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so
removing it is safe.
Simplifies realtime port resolution to `env.PORT` (zod-validated with a
3002 default) and drops the orphaned sim-side schema entry.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Waleed Latif <walif6@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(execution): run pptx/docx/pdf generation inside isolated-vm sandbox
Retires the legacy doc-worker.cjs / pptx-worker.cjs pipeline that ran user
DSL via node:vm + full require() in the same UID/PID namespace as the main
Next.js process. User code now runs inside the existing isolated-vm pool
(V8 isolate, no process / require / fs, no /proc/1/environ reachability).
Introduces a first-class SandboxTask abstraction under apps/sim/sandbox-tasks/
that mirrors apps/sim/background/ — one file per task, central typed
registry, kebab-case ids. Adding a new thing that runs in the isolate is
one file plus one registry entry.
Runtime additions in lib/execution/:
- task-mode execution in isolated-vm-worker.cjs: load pre-built library
bundles, run task bootstrap, run user code, run finalize, transfer
Uint8Array result as base64 via IPC
- named broker IPC bridge (generalizes the existing fetch bridge) with
args size, result size, and per-execution call caps
- cooperative AbortSignal support: cancel IPC disposes the isolate, pool
slot is freed, pending broker-call timers are swept
- compiled scripts + references explicitly released per execution
- isolate.isDisposed used for cancellation detection (no error-string
substring matching)
Library bundles (pptxgenjs, docx, pdf-lib) are built into isolate-safe
IIFE bundles by apps/sim/lib/execution/sandbox/bundles/build.ts and
committed; next.config.ts / trigger.config.ts / Dockerfile updated to
ship them instead of the deleted dist/*-worker.cjs artifacts.
Call sites migrated:
- app/api/workspaces/[id]/pptx/preview/route.ts
- app/api/files/serve/[...path]/route.ts (+ test mock)
- lib/copilot/tools/server/files/{workspace-file,edit-content}.ts
All pass owner key user:<userId> for per-user pool fairness + distributed
lease accounting.
Made-with: Cursor
* improvement(sandbox): delegate timers to Node, add phase timings + saturation logs
Follow-ups on top of the isolated-vm migration (da14027b2):
Timer delegation (laverdet/isolated-vm#136 recommended pattern):
- setTimeout / setInterval / clearTimeout / clearImmediate delegate to
Node's real timer heap via ivm.Reference. Real delays are honored;
clearTimeout actually cancels; ms is clamped to the script timeout
so callbacks can't fire after the isolate is disposed.
- Per-execution timer tracking + dispose-sweep in finally. Zero stale
callbacks post-dispose.
- unwrapPrimitive helper normalizes ivm.Reference-wrapped primitives
(arguments: { reference: true } applies uniformly to all args).
- _polyfills.ts shrinks from ~130 lines to the global->globalThis alias.
Timers / TextEncoder / TextDecoder / console all install per-execution
from the worker via ivm bridges.
AbortSignal race fix (pre-existing bug surfaced by the timer smoke):
- Listener is registered after await tryAcquireDistributedLease. If the
signal aborted during that ~200ms window (Redis down), AbortSignal
doesn't fire listeners registered after the fact — the abort was
silently missed. Now re-checks signal.aborted synchronously after
addEventListener.
Observability:
- executeTask returns IsolatedVMTaskTimings (setup, runtimeBootstrap,
bundles, brokerInstall, taskBootstrap, harden, userCode, finalize,
total) in every success + error path. run-task.ts logs these with
workspaceId + queueMs so 'which tenant is slow' is queryable.
- Pool saturation events now emit structured logger.warn with reason
codes: queue_full_global, queue_full_owner, queue_wait_timeout,
distributed_lease_limit. Matches the existing broker reject pattern.
Security policy:
- New .cursor/rules/sim-sandbox.mdc codifies the hard rules for the
worker process: no app credentials, all credentialed work goes
through host-side brokers, every broker scopes by workspaceId.
Pre-merge checklist for future changes to isolated-vm-worker.cjs.
Measured phase breakdown (local smoke, Redis down): pptx wall=~310ms
with bundles=~16ms, finalize=~83ms; docx ~290ms / 17ms / 70ms; pdf
~235ms / 17ms / 5ms. Bundle compilation is not the bottleneck —
library finalize is.
Made-with: Cursor
* fix(sandbox): thread AbortSignal into runSandboxTask at every call site
Three remaining callers of runSandboxTask were not threading a
cancellation signal, so a client disconnect mid-compile left the pool
slot occupied for the full 60s task timeout. Matching the pattern the
pptx-preview route already uses.
- apps/sim/app/api/files/serve/[...path]/route.ts — GET forwards
`request.signal` into handleLocalFile / handleCloudProxy, which
forward into compileDocumentIfNeeded, which forwards into
runSandboxTask.
- apps/sim/lib/copilot/tools/server/files/workspace-file.ts — passes
`context.abortSignal` (transport/user stop) into runSandboxTask.
- apps/sim/lib/copilot/tools/server/files/edit-content.ts — same.
Smoke: simulated client disconnect at t=1000ms during a task that would
otherwise have waited 10s. The pool slot unwinds at t=1002ms with
AbortError; previously would have sat 60s until the task-level timeout.
Made-with: Cursor
* chore(build): raise node heap to 8GB for next build type-check
Next.js's type-check worker OOMs at the default 4GB heap on Node 23 for
this project's type graph size. Bumps the heap to 8GB only for the
`next build` invocation inside `bun run build`.
Docker builds are unaffected — `next.config.ts` sets
`typescript.ignoreBuildErrors: true` when DOCKER_BUILD=1, which skips
the type-check pass entirely. This only fixes local `bun run build`.
No functional code changes.
Made-with: Cursor
* fix lint
* refactor(copilot): dedup getDocumentFormatInfo across copilot file tools
The same extension -> { formatName, sourceMime, taskId } mapping was
duplicated in workspace-file.ts and edit-content.ts. Any future format
or task-id change had to happen in two places.
Exports getDocumentFormatInfo + DocumentFormatInfo from workspace-file.ts
(which already owned the PPTX/DOCX/PDF source MIME constants) and
imports it in edit-content.ts. Same source-of-truth pattern the file
already uses for inferContentType.
Made-with: Cursor
* fix(sandbox): propagate empty-message broker/fetch errors
Both bridges in the isolate used truthiness to detect host-side errors:
if (response.error) throw new Error(response.error); // broker
if (result.error) throw new Error(result.error); // fetch
If a host handler ever threw `new Error('')`, err.message would be ''
(falsy), so { error: '' } was silently swallowed and the isolate saw
a successful null result. Existing call sites don't throw empty-message
errors, but the pattern was structurally unsafe.
Switch both to typeof check === 'string' and fall back to a default
message if the string is empty, so all host-reported errors propagate
into the isolate regardless of message content.
Made-with: Cursor
* improvement(ci): parallelize Docker builds with tests and remove duplicate turbo install
* fix(test): use SecureFetchResponse shape in mock instead of standard Response
* feat(tools): added speech to text with openai whisper, elevenlabs, and deepgram
* added new file icons, implemented ffmpeg
* updated docs
* revert environment
* added blacksmith optimizations to workflows and dockerfiles to enhance performance. please review before pushing to production
* remove cache from and cache to directives from docker based actions, per blacksmith docs
---------
Co-authored-by: Connor Mulholland <connormul@Connors-MacBook-Pro.local>
* fix(webhooks): use next public app url instead of request origin for webhook registration
* ack PR comments
* ci: pin Bun to v1.2.22 to avoid Bun 1.3 breaking changes
* feat: implement native ARM64 Docker builds with CDN support
- Replace QEMU emulation with native ARM64/AMD64 runners (linux-arm64-8-core, linux-x64-8-core)
- Fix manifest creation with proper error handling and image existence checks
- Add CDN video support with getVideoUrl function and Video component
- Update all docs MDX files to use Video component instead of raw video tags
- Update GitHub Actions workflow to use architecture-specific builds
- Remove QEMU setup to eliminate emulation timeout issues
- Maintain multi-arch Docker image support through manifests
* Update .github/workflows/build.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* migrate to bun
* added envvars to drizzle
* upgrade bun devcontainer feature to a valid one
* added bun, docker not working
* updated envvars, updated to bunder and esnext modules
* fixed build, reinstated otel
* feat: optimized multi-stage docker images
* add coerce for boolean envvar
* feat: add docker-compose configuration for local LLM services and remove legacy Dockerfile and entrypoint script
* feat: add docker-compose files for local and production environments, and implement GitHub Actions for Docker image build and publish
* refactor: remove unused generateStaticParams function from various API routes and maintain dynamic rendering
* cleanup
* upgraded bun
* updated ci
* fixed build
---------
Co-authored-by: Aditya Tripathi <aditya@climactic.co>
