205 Commits

Author SHA1 Message Date
Waleed
60b80ec172 improvement(tables): race-free row-count trigger + scoped tx timeouts (#4289)
* improvement(tables): race-free row-count trigger + scoped tx timeouts

* fix(tables): close upsert race + serialize replaceTableRows

Two concurrency bugs flagged by review:

1. `upsertRow` insert path: removing FOR UPDATE broke serialization between
   the initial existing-row SELECT and the INSERT. Two concurrent upserts
   on the same conflict target could both find no match, then both insert,
   producing a duplicate that bypasses the app-level unique check. Fixed
   by re-checking for the matching row *after* acquiring the per-table
   advisory lock; if a racing tx already inserted, switch to UPDATE.

2. `replaceTableRows`: under READ COMMITTED each tx's DELETE uses its own
   MVCC snapshot, so two concurrent replaces could each DELETE only the
   rows visible at their start, then both INSERT, leaving the table with
   the union of both row sets. Fixed by acquiring the per-table advisory
   lock at the start of the tx to serialize replaces against each other
   and against auto-position inserts.

Also updated a stale docstring on `upsertRow` that still referenced the
removed FOR UPDATE.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(tables): serialize explicit-position inserts with advisory lock

The `(table_id, position)` index is non-unique. Concurrent explicit-
position inserts at the same slot can both observe the slot empty, both
skip the shift, then each INSERT — producing duplicate `(table_id,
position)` rows.

Take the per-table advisory lock in the explicit-position branches of
`insertRow` and `batchInsertRows` (the auto-position branches already do
this). Updated the test that asserted the explicit path skipped the lock,
and added coverage for `batchInsertRows` with explicit positions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(tables): dedupe upsert UPDATE path + extract nextAutoPosition

Two pure cleanups on the round-1 changes:

1. Extract `nextAutoPosition(trx, tableId)` — the `SELECT coalesce(max(
   position), -1) + 1` pattern was repeated in three call sites
   (`insertRow` auto branch, `batchInsertRows` auto branch, `upsertRow`
   insert branch). One helper, same behavior.

2. Consolidate `upsertRow` update path. The initial-SELECT match and the
   post-lock re-select match previously had two literal duplicates of the
   same UPDATE + return block. Resolve `matchedRowId` first, then run one
   UPDATE branch. Lock is still only acquired when we don't find a match
   on the first pass.

No behavior change. 98/98 table unit tests unchanged.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 15:33:17 -07:00
Theodore Li
04f1d015f3 fix(mothership): Use heartbeat mechanism for chat locks (#4286) 2026-04-24 14:36:50 -04:00
Waleed
ccb5f1e690 fix(db): revert statement_timeout startup options breaking pooled connections (#4284) 2026-04-23 23:31:52 -07:00
Theodore Li
c22ac38ab0 Set statement timeout of 90 seconds (#4276) 2026-04-23 18:27:15 -04:00
Waleed
0ae19dab85 feat(files): default sort by updated and add updated sort option (#4279)
* feat(files): default sort by updated and add updated sort option

* feat(files): show Last Updated column

Matches the visible-column pattern already used on Knowledge and Tables
so users can see the value they're sorting by.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-23 15:02:07 -07:00
Theodore Li
65972f2fa3 fix(retention): switch data retention to be org-level (#4270)
* fix(retention): switch data retention to be org-level

* fix lint

* cleanup mothership ran logs

* fix cleanup dispatcher

* fix ui flash for data retention settings

* fix lint

* remove raw sql string interprolation
2026-04-23 02:41:49 -04:00
Vikhyath Mondreti
5f0f0edd63 improvement(repo): separate realtime into separate app (#4262)
* improvement(repo): restructuring to make realtime image narrower scoped

* improvements

* chore(repo): rebase fixes and quality improvements for realtime split

Addresses merge-time issues and gaps from the realtime app split:
- Retarget stale vi.mock paths to @sim/workflow-persistence/subblocks
- Restore README branding, fix AGENTS.md script reference
- Restore TSDoc on workflow-persistence subblocks helpers
- Use toError() from @sim/utils/errors in save.ts
- Add vitest config + local mocks so @sim/audit tests run standalone
- Move socket.io-client to devDependencies in apps/realtime
- Add missing package COPY steps to docker/app.Dockerfile
- Add check:boundaries/check:realtime-prune scripts and wire into CI

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(security): consolidate crypto primitives into @sim/security

Move general-purpose crypto primitives out of apps/sim into the
@sim/security package so both apps/sim and apps/realtime can share them.

@sim/security exports (all pure, dependency-free):
  ./compare    safeCompare (constant-time HMAC-wrapped equality)
  ./encryption encrypt/decrypt (AES-256-GCM, iv:cipher:tag format)
  ./hash       sha256Hex
  ./tokens     generateSecureToken (base64url)

Migrate apps/sim call sites to use these + @sim/utils helpers:
  crypto.randomUUID()            -> generateId() from @sim/utils/id
  createHash('sha256').digest    -> sha256Hex
  timingSafeEqual on hashed hex  -> safeCompare
  new Promise(setTimeout)        -> sleep from @sim/utils/helpers

No behavior change: encryption format, digest output, and token
length are preserved exactly.

* refactor(copilot): use toError in remaining otel/finalize sites

Replace the last two `error instanceof Error ? error : new Error(String(error))`
patterns with toError from @sim/utils/errors. Completes the sweep of clean
candidates — no behavior change.

* refactor(security): consolidate HMAC-SHA256 primitives into @sim/security

Adds hmacSha256Hex and hmacSha256Base64 to @sim/security/hmac and migrates
15 webhook providers plus 5 other hot paths (deployment token signing,
outbound webhook requests, workspace notification delivery, notification
test route, Shopify OAuth callback) off bare `createHmac` calls. Secret
parameter accepts `string | Buffer` to cover base64-decoded Svix-style
secrets (Resend) and MS Teams' HMAC scheme. AWS SigV4 signing in S3 and
Textract tools intentionally retains direct `createHmac` usage — its
multi-step key derivation chain doesn't fit a generic helper.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(packages): post-audit test + packaging polish

- Add safeCompare unit tests (identity, length mismatch, hex-nibble diff).
- Add Buffer-secret cases to hmac tests to lock in Svix/MS-Teams contract.
- Declare `reactflow` as a peerDependency on @sim/workflow-types — only used for type imports.
- Add a barrel export to @sim/workflow-persistence for consumers that prefer package-level imports; subpath exports retained.
- Document the data-field invariant in load.ts for loop/parallel subflow patching.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(realtime): address PR review feedback

- Remove redundant SOCKET_PORT=3002 env from Dockerfile runner stage
  (env.PORT already defaults to 3002 via zod schema).
- Reorder PORT fallback so an explicitly-set SOCKET_PORT wins over
  the schema default for PORT; keeps SOCKET_PORT functional as an
  override instead of dead code.
- Add dedicated type-check CI step for @sim/realtime so TS errors
  surface pre-deploy (the Dockerfile runs source TS via Bun and has
  no implicit build-time type check).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(realtime): remove unused SOCKET_PORT env var

SOCKET_PORT has lived in the socket server since the June 2025 refactor
but was never actually set in any deploy config — docker-compose.prod,
helm values/templates, .env.example, and docs all use PORT or the 3002
default exclusively. No self-hoster was ever pointed at SOCKET_PORT, so
removing it is safe.

Simplifies realtime port resolution to `env.PORT` (zod-validated with a
3002 default) and drops the orphaned sim-side schema entry.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Waleed Latif <walif6@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-22 23:06:16 -07:00
Theodore Li
8ce56fe1f2 fix(auth): add api key auth via sha256 hash lookup (#4266)
* fix(auth): add api key auth via sha256 hash lookup

* Remove promise all logic

* Restore feature flag

* fix feature flag

* Combine auth and hash gate
2026-04-22 18:30:37 -04:00
Theodore Li
d927d8bdff fix(db): raise db pool size (#4263)
* fix(db): raise db pool size

* Raise socket connections

* bump up connection size even more
2026-04-22 14:25:13 -04:00
Vikhyath Mondreti
41a1b50ace improvement(migrations): log better errors (#4260) 2026-04-21 22:06:05 -07:00
Vikhyath Mondreti
51ace655e4 fix(migration): permission group migration error (#4258) 2026-04-21 21:10:50 -07:00
Waleed
45bf396968 fix(deps): bump drizzle-orm 0.45.2 + adopt MCP SDK 1.25.3 native types (#4252)
* fix(deps): bump drizzle-orm to 0.45.2 (GHSA-gpj5-g38j-94v9)

Resolves Dependabot alert #98. Drizzle ORM <0.45.2 improperly escaped
quoted SQL identifiers, allowing SQL injection via untrusted input
passed to APIs like sql.identifier() or .as().

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(mcp): adopt native SDK types after @modelcontextprotocol/sdk 1.25.3 bump

Replace hand-written schema/annotation shapes with the SDK's exported
Tool, JSONRPCResultResponse, and Tool['annotations'] types so changes
upstream flow through automatically.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(types): use drizzle $inferSelect for row types

Replace hand-written interfaces that duplicated schema shape with
typeof table.$inferSelect aliases for webhook, workflow, and
workspaceFiles rows. Also simplify metadata insert/update to use
.returning() instead of field-by-field copies.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(uploads): fall through to INSERT if restore-deleted row races a hard delete

If a hard delete races between the initial SELECT and the restore UPDATE,
.returning() yields no row. Previously the function would return undefined
and silently violate the Promise<FileMetadataRecord> contract. Now the
function falls through to the INSERT path, which already handles
uniqueness races via the 23505 catch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(uploads): align metadata.ts with global standards

Replace dynamic uuid import with generateId() per @sim/utils/id
convention, narrow the error catch off `any`, and convert the inline
comment to TSDoc.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-21 19:52:15 -07:00
Vikhyath Mondreti
aee6189d14 improvement(access-control): migrate to workspace scope (#4244)
* improvement(access-control): migrate to workspace scope

* fix edge cases

* update docs

* prep merge

* regen migrations

* address comments

* add ws id, user constraint

* address more comments

* address ui comments

* address more comments
2026-04-21 15:53:17 -07:00
Theodore Li
699bbfd16f feat(log): Add wrapper function for standardized logging (#4061)
* feat(log): Add wrapper function for standardized logging

* Add all routes to wrapper, handle background execution

* fix lint

* fix test

* fix test missing url

* fix lint

* fix tests

* fix build

* fix(build): unmangle generic in admin outbox requeue route

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 13:21:13 -04:00
Waleed
0e1ff0a1ac improvement(enterprise): slack wizard UI, enterprise docs, data retention updates (#4241)
* improvement(enterprise): slack wizard UI, enterprise docs, data retention updates

* improvement(docs): add enterprise screenshots to sso, access-control, whitelabeling pages

* form

* fix(enterprise): address PR review — h-full for recently-deleted, shared SettingRow, toast UX, stale form fix, emcn tokens

* fix(whitelabeling): scope drop zone to thumbnail only, not full upload row

* fix(whitelabeling): remove drop image text from drag overlay

* fix(config): add DATA_RETENTION_ENABLED to env schema to fix build type error

* fix(testing): add isDataRetentionEnabled to feature flags mock

* improvement(docs): remove redundant requirements section from data-retention page

* improvement(docs): remove requirements sections from all enterprise doc pages

* improvement(docs): add screenshot to audit-logs page

* fix(data-retention): bypass enterprise gate when billing is disabled for self-hosted
2026-04-20 23:24:14 -07:00
Theodore Li
802f4cf0fc feat(jobs): Add data retention jobs (#4128)
* feat(jobs): Add data retention jobs

Add 3 cron-triggered cleanup jobs dispatched via Trigger.dev (or inline fallback):
- cleanup-soft-deletes: hard-deletes soft-deleted workspace resources past retention
- cleanup-logs: deletes expired workflow execution logs + S3 files
- cleanup-tasks: deletes expired copilot chats, runs, feedback, inbox tasks

Enterprise admins can configure per-workspace retention via Settings > Data Retention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

# Conflicts:
#	packages/db/migrations/meta/0192_snapshot.json
#	packages/db/migrations/meta/_journal.json
#	packages/db/schema.ts

* Cleanup orphaned using ids, not timestamp sorting

* fix lint
2026-04-20 20:24:39 -04:00
Waleed
ac4ccfcac8 fix(billing): close TOCTOU race in subscription transfer, centralize stripe test mocks (#4239)
* fix(billing): close TOCTOU race in subscription transfer, centralize stripe test mocks

* more mocks

* fix(testing): provide complete Stripe.Event defaults in createMockStripeEvent

* fix(testing): make dbChainMock .for('update') chainable with .limit()

* fix(billing): gate subscription transfer noop behind membership check

Previously the 'already belongs to this organization' early return fired
before the org/member lookups, letting any authenticated caller probe
sub-to-org pairings without being a member of the target org. Move the
noop check after the admin/owner verification so unauthorized callers
hit the 403 first.
2026-04-20 16:46:28 -07:00
Waleed
0cd14f4ac9 improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs (#4238)
* improvement(sso): fix provider lookup, migrate UI to emcn, add enterprise SSO docs

* fix(sso): add org membership guard on providers route, fix idpMetadata round-trip

* fix(sso): add org membership guard on register route, fix SP entityID, remove fullError leak

* fix(sso): fix SAML script callbackUrl and SP entityID to use app base URL

* fix(sso): correct SAML callback URL path in script header comment

* fix(sso): restrict SSO provider read/write to org owners and admins

* docs(sso): restructure page, fix provider guide accuracy, add external doc links

* fix(sso): correct SAML callback path and generate idpMetadata from cert+entryPoint

* fix(sso): always require NEXT_PUBLIC_APP_URL for SAML SP metadata entityID

* fix(sso): scope provider query to org only when organizationId is provided

* fix(sso): escape XML special chars in script idpMetadata generation

* fix(sso): final audit corrections — saml mapping, xml escaping, self-hosted org guard

* fix(sso): redact oidc client secret in providers response, add self-hosted org admin guard

* fix(sso): scope redacted-secret lookup to caller's org or userId

* fix(sso): null out oidcConfig on parse failure to prevent unredacted secret leak

* fix(sso): use issuer as entityID in auto-generated idp metadata xml
2026-04-20 16:45:37 -07:00
Vikhyath Mondreti
d9209f9588 improvement(governance): workspace-org invitation system consolidation (#4230)
* workspace re-org checkpoint

* admin route reconciliation

* checkpoint consistency fixes

* prep merge

* regen migration

* checkpoint

* code cleanup

* update docs

* add feature for owner to leave + admin route

* address comments

* fix new account race

* address comments
2026-04-20 14:45:07 -07:00
Waleed
5cf7e8d546 improvement(codebase): migrate tests to dbChainMock, extract react-query hooks (#4235)
* improvement(codebase): migrate tests to dbChainMock, extract react-query hooks

Migrate 97 test files to centralized dbChainMock/dbChainMockFns helpers from
@sim/testing — removes hoisted chain-wiring boilerplate.

Extend dbChainMock to cover insert/update/delete/transaction/execute patterns.
Extract useGitHubStars and useVoiceSettings react-query hooks from inline fetches.
Centralize additional mocks (authMockFns, hybridAuthMockFns) and update docs.

* fix(github-stars): centralize fallback via initialData, remove stale constants

Move the placeholder star count into useGitHubStars as initialData with
initialDataUpdatedAt: 0 so `data` is always a narrowed string while still
refetching on mount. Fixes two Bugbot issues: stale '25.8k' in chat.tsx
(vs '27.8k' in navbar) and empty-string return in fetchGitHubStars that
bypassed `??` fallbacks in consumers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(testing): wire dbChainMock.db to shared transaction and execute fns

dbChainMock.db.transaction was an inline vi.fn() separate from the exported
dbChainMockFns.transaction, so dbChainMockFns.transaction.mockResolvedValueOnce
and assertions silently targeted the wrong instance. dbChainMock.db also
omitted execute, so tests for any module that calls db.execute (logging-session,
table service, billing balance) would throw TypeError. Both mocks now reference
the module-level constants so overrides and resetDbChainMock affect the same fn.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(chat,testing): memoize welcome message and add selectDistinct to dbChainMock.db

Why:
- Welcome ChatMessage was rebuilt inline each render, producing a fresh
  timestamp and new array identity — cascading to ChatMessageContainer
  and VoiceInterface props on every tick.
- dbChainMockFns exports selectDistinct/selectDistinctOn but the
  dbChainMock.db object omitted them, so tests that stub those builders
  hit undefined on the mocked module.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(chat): re-attach scroll listener once container mounts

The scroll effect's empty dep array meant it ran only on the first
render, when `chatConfig` is still loading and the component returns
`<ChatLoadingState />` — so `messagesContainerRef.current` was null and
the listener was never attached. Depend on the gating conditions that
control which tree renders, so the effect re-runs once the real
container is in the DOM (and re-attaches when toggling in/out of voice
mode).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(chat): reset chat state on identifier change via key prop

Keying `<ChatClient>` on `identifier` guarantees a full remount on
route transitions between chats, so `conversationId`, `messages`, and
every other piece of local state start fresh — no reset effect
required.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 23:05:06 -07:00
Waleed
b5674d9ed4 improvement(codebase): centralize test mocks, extract @sim/utils, remove dead code (#4228)
* improvement(codebase): centralize test mocks, extract @sim/utils, remove dead code

* improvement(codebase): apply @sim/utils conventions to staging-introduced files
2026-04-18 14:39:03 -07:00
Vikhyath Mondreti
c246f5c660 improvement(billing): route scope by subscription referenceId, sync plan from Stripe, transfer storage on org join, outbox service (#4219)
* fix(billing): route scope by subscription referenceId, sync plan from Stripe, transfer storage on org join

Route every billing decision (usage limits, credits, storage, rate
limit, threshold billing, webhooks, UI permissions) through the
subscription's `referenceId` instead of plan-name heuristics. Fixes
the production state where a `pro_6000` subscription attached to an
organization was treated as personal Pro by display/edit code while
execution correctly enforced the org cap.

Scope
- Add `isOrgScopedSubscription(sub, userId)` (pure) and
  `isSubscriptionOrgScoped(sub)` (async DB-backed) helpers. One is
  used wherever a user perspective is available; the other in webhook
  handlers that only have a subscription row.
- Replace plan-name scope checks in ~20 files: usage/limit readers,
  credits balance + purchase, threshold billing, storage limits +
  tracking, rate limiter, invoice + subscription webhooks, seat
  management, membership join/leave, `switch-plan` admin gate,
  admin credits/billing routes, copilot 402 handler, UI subscription
  settings + permissions + sidebar indicator, React Query types.

Plan sync
- Add `syncSubscriptionPlan(subscriptionId, currentPlan, planFromStripe)`
  called from `onSubscriptionComplete` and `onSubscriptionUpdate` so
  the DB `plan` column heals on every Stripe event. Pro->Team upgrades
  previously updated price, seats, and referenceId but left `plan`
  stale — this is what produced the `pro_6000`-on-org row.

Priority + grace period
- `getHighestPrioritySubscription` now prefers org over personal
  within each tier (Enterprise > Team > Pro, org > personal at each).
  A user with a `cancelAtPeriodEnd` personal Pro who joins a paid org
  routes pooled resources to the org through the grace window.
- `calculateSubscriptionOverage` personal-Pro branch reads user_stats
  directly (bypassing priority) and bills only `proPeriodCostSnapshot`
  when the user joined a paid org mid-cycle, so post-join org usage
  isn't double-charged on the personal Pro's final invoice.
  `resetUsageForSubscription` mirrors this: preserves
  `currentPeriodCost` / `currentPeriodCopilotCost` when
  `proPeriodCostSnapshot > 0` so the org's next cycle-close captures
  post-join usage correctly.

Uniform base-price formula
- `basePrice × (seats ?? 1)` everywhere: `getOrgUsageLimit`,
  `updateOrganizationUsageLimit`, `setUsageLimitForCredits`,
  `calculateSubscriptionOverage`, threshold billing,
  `syncSubscriptionUsageLimits`, `getOrganizationBillingData`.
  Admin dashboard math now agrees with enforcement math.

Storage transfer on join
- Invitation-accept flow moves `user_stats.storageUsedBytes` into
  `organization.storageUsedBytes` inside the same transaction when
  the org is paid.
- `syncSubscriptionUsageLimits` runs a bulk-backfill version so
  members who joined before this fix, or orgs that upgraded from
  free to paid after members joined, get pulled into the org pool
  on the next subscription event. Idempotent.

UX polish
- Copilot 402 handler differentiates personal-scoped ("increase your
  usage limit") from org-scoped ("ask an owner or admin to raise the
  limit") while keeping the `increase_limit` action code the parser
  already understands.
- Duplicate-subscription error on team upgrade names the existing
  plan via `getDisplayPlanName`.
- Invitation-accept invalidates subscription + organization React
  Query caches before redirect so settings doesn't flash the user's
  pre-join personal view.

Dead code removal
- Remove unused `calculateUserOverage`, and the following fields on
  `SubscriptionBillingData` / `getSimplifiedBillingSummary` that no
  consumer in the monorepo read: `basePrice`, `overageAmount`,
  `totalProjected`, `tierCredits`, `basePriceCredits`,
  `currentUsageCredits`, `overageAmountCredits`, `totalProjectedCredits`,
  `usageLimitCredits`, `currentCredits`, `limitCredits`,
  `lastPeriodCostCredits`, `lastPeriodCopilotCostCredits`,
  `copilotCostCredits`, and the `organizationData` subobject. Add
  `metadata: unknown` to match what the server returns.

Notes for the triggering customer
- The `pro_6000`-on-org row self-heals on the next Stripe event via
  `syncSubscriptionPlan`. For the one known customer, a direct
  UPDATE is sufficient:
  `UPDATE subscription SET plan='team_6000' WHERE id='aq2...' AND plan='pro_6000'`.

Made-with: Cursor

* fix tests

* address more comments

* progress

* harden further

* outbox service

* address comments

* address comment on check

* simplify

* cleanup code

* minor improvement
2026-04-18 10:46:14 -07:00
Siddharth Ganesan
0abcc6e813 improvement(mothership): restructured stream, tool structures, code typing, file write/patch/append tools, timing issues (#4090)
* fix build error

* improvement(mothership): new agent loop (#3920)

* feat(transport): replace shared chat transport with mothership-stream module

* improvement(contracts): regenerate contracts from go

* feat(tools): add tool catalog codegen from go tool contracts

* feat(tools): add tool-executor dispatch framework for sim side tool routing

* feat(orchestrator): rewrite tool dispatch with catalog-driven executor and simplified resume loop

* feat(orchestrator): checkpoint resume flow

* refactor(copilot): consolidate orchestrator into request/ layer

* refactor(mothership): reorganize lib/copilot into structured subdirectories

* refactor(mothership): canonical transcript layer, dead code cleanup, type consolidation

* refactor(mothership): rebase onto latest staging

* refactor(mothership): rename request continue to lifecycle

* feat(trace): add initial version of request traces

* improvement(stream): batch stream from redis

* fix(resume): fix the resume checkpoint

* fix(resume): fix resume client tool

* fix(subagents): subagent resume should join on existing subagent text block

* improvement(reconnect): harden reconnect logic

* fix(superagent): fix superagent integration tools

* improvement(stream): improve stream perf

* Rebase with origin dev

* fix(tests): fix failing test

* fix(build): fix type errors

* fix(build): fix build errors

* fix(build): fix type errors

* feat(mothership): add cli execution

* fix(mothership): fix function execute tests

* Force redeploy

* feat(motheship): add docx support

* feat(mothership): append

* Add deps

* improvement(mothership): docs

* File types

* Add client retry logic

* Fix stream reconnect

* Eager tool streaming

* Fix client side tools

* Security

* Fix shell var injection

* Remove auto injected tasks

* Fix 10mb tool response limit

* Fix trailing leak

* Remove dead tools

* file/folder tools

* Folder tools

* Hide function code inline

* Dont show internal tool result reads

* Fix spacing

* Auth vfs

* Empty folders should show in vfs

* Fix run workflow

* change to node runtime

* revert back to bun runtime

* Fix

* Appends

* Remove debug logs

* Patch

* Fix patch tool

* Temp

* Checkpoint

* File writes

* Fix

* Remove tool truncation limits

* Bad hook

* replace react markdown with streamdown

* Checkpoitn

* fix code block

* fix stream persistence

* temp

* Fix file tools

* tool joining

* cleanup subagent + streaming issues

* streamed text change

* Tool display intetns

* Fix dev

* Fix tests

* Fix dev

* Speed up dev ci

* Add req id

* Fix persistence

* Tool call names

* fix payload accesses

* Fix name

* fix snapshot crash bug

* fix

* Fix

* remove worker code

* Clickable resources

* Options ordering

* Folder vfs

* Restore and mass delete tools

* Fix

* lint

* Update request tracing and skills and handlers

* Fix editable

* fix type error

* Html code

* fix(chat): make inline code inherit parent font size in markdown headers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* improved autolayout

* durable stream for files

* one more fix

* POSSIBLE BREAKAGE: SCROLLING

* Fixes

* Fixes

* Lint fix

* fix(resource): fix resource view disappearing on ats (#4103)

Co-authored-by: Theodore Li <theo@sim.ai>

* Fixes

* feat(mothership): add execution logs as a resource type

Adds `log` as a first-class mothership resource type so copilot can open
and display workflow execution logs as tabs alongside workflows, tables,
files, and knowledge bases.

- Add `log` to MothershipResourceType, all Zod enums, and VALID_RESOURCE_TYPES
- Register log in RESOURCE_REGISTRY (Library icon) and RESOURCE_INVALIDATORS
- Add EmbeddedLog and EmbeddedLogActions components in resource-content
- Export WorkflowOutputSection from log-details for reuse in EmbeddedLog
- Add log resolution branch in open_resource handler via new getLogById service
- Include log id in get_workflow_logs response and extract resources from output
- Exclude log from manual add-resource dropdown (enters via copilot tools only)
- Regenerate copilot contracts after adding log to open_resource Go enum

* Fix perf and message queueing

* Fix abort

* fix(ui): dont delete resource on clearing from context, set resource closed on new task (#4113)

Co-authored-by: Theodore Li <theo@sim.ai>

* improvement(mothership): structure sim side typing

* address comments

* reactive text editor tweaks

* Fix file read and tool call name persistence bug

* Fix code stream + create file opening resource

* fix use chat race + headless trace issues

* Fix type issue

* Fix mothership block req lifecycle

* Fix build

* Move copy reqid

* Fix

* fix(ui): fix resource tag transition from home to task (#4132)

Co-authored-by: Theodore Li <theo@sim.ai>

* Fix persistence

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Waleed Latif <walif6@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Theodore Li <theo@sim.ai>
Co-authored-by: Theodore Li <theodoreqili@gmail.com>
2026-04-13 16:46:35 -07:00
Waleed
7491d70a67 feat(workspaces): add workspace logo upload (#4136)
* feat(workspaces): add workspace logo upload

* feat(workspaces): add workspace logo upload

* fix(workspaces): validate logoUrl accepts only paths or HTTPS URLs

* fix(workspaces): add admin authorization, audit log, and posthog event for workspace logo uploads

* lint

* fix: add WebP support and use refs pattern in useProfilePictureUpload

- Add image/webp to ACCEPTED_IMAGE_TYPES in useProfilePictureUpload
- Add image/webp to file input accept attributes in whitelabeling settings
- Refactor useProfilePictureUpload to use refs for onUpload, onError, and
  currentImage callbacks, matching the established codebase pattern

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore cloudwatch/cloudformation files from staging

These files were accidentally regressed during rebase conflict resolution,
reverting changes from #4027. Restoring to staging versions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add workspace_logo_uploaded to PostHogEventMap

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: separate workspaceId ref sync to prevent overwrite on re-render

Split the ref sync useEffect so workspaceIdRef only updates when the
workspaceId prop changes, not when onUpload/onError callbacks get new
references. Prevents setTargetWorkspaceId from being overwritten by
a re-render before the file upload completes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use Pick type for workspace dropdown in knowledge header

The shared Workspace type requires ownerId and other fields that aren't
available from the workspaces API response mapping. Use a Pick type to
accurately represent the subset of fields actually constructed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: replace raw fetch with useWorkspacesQuery in knowledge header

Remove useState + useEffect + fetch anti-pattern for loading workspaces.
Use useWorkspacesQuery from React Query with inline filter for write/admin
permissions. Eliminates ~30 lines of manual state management, any casts,
and the Pick type workaround.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:54:21 -07:00
Waleed
5ab85c6930 feat(workspaces): add recency-based workspace switching and redirect (#4131)
* feat(workspaces): add recency-based workspace switching and redirect

* fix(workspaces): skip prune when workspace list is empty on mount
2026-04-13 14:10:29 -07:00
Waleed
30c5e82ab0 feat(ee): add enterprise audit logs settings page (#4111)
* feat(ee): add enterprise audit logs settings page with server-side search

Add a new audit logs page under enterprise settings that displays all
actions captured via recordAudit. Includes server-side search, resource
type filtering, date range selection, and cursor-based pagination.

- Add internal API route (app/api/audit-logs) with session auth
- Extract shared query logic (buildFilterConditions, buildOrgScopeCondition,
  queryAuditLogs) into app/api/v1/audit-logs/query.ts
- Refactor v1 and admin audit log routes to use shared query module
- Add React Query hook with useInfiniteQuery and cursor pagination
- Add audit logs UI with debounced search, combobox filters, expandable rows
- Gate behind requiresHosted + requiresEnterprise navigation flags
- Place all enterprise audit log code in ee/audit-logs/

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* fix(ee): fix build error and address PR review comments

- Fix import path: @/lib/utils → @/lib/core/utils/cn
- Guard against empty orgMemberIds array in buildOrgScopeCondition
- Skip debounce effect on mount when search is already synced

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* fix(ee): fix type error with unknown metadata in JSX expression

Use ternary instead of && chain to prevent unknown type from being
returned as ReactNode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ee): align skeleton filter width with actual component layout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* feat(audit): add audit logging for passwords, credentials, and schedules

- Add PASSWORD_RESET_REQUESTED audit on forget-password with user lookup
- Add CREDENTIAL_CREATED/UPDATED/DELETED audit on credential CRUD routes
  with metadata (credentialType, providerId, updatedFields, envKey)
- Add SCHEDULE_CREATED audit on schedule creation with cron/timezone metadata
- Fix SCHEDULE_DELETED (was incorrectly using SCHEDULE_UPDATED for deletes)
- Enhance existing schedule update/disable/reactivate audit with structured
  metadata (operation, updatedFields, sourceType, previousStatus)
- Add CREDENTIAL resource type and Credential filter option to audit logs UI
- Enhance password reset completed description with user email

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit): align metadata with established recordAudit patterns

- Add actorName/actorEmail to all new credential and schedule audit calls
  to match the established pattern (e.g., api-keys, byok-keys, knowledge)
- Add resourceId and resourceName to forget-password audit call
- Enhance forget-password description with user email

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(testing): sync audit mock with new AuditAction and AuditResourceType entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(audit-logs): derive resource type filter from AuditResourceType

Instead of maintaining a separate hardcoded list, the filter dropdown
now derives its options directly from the AuditResourceType const object.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(audit): enrich all recordAudit calls with structured metadata

- Move resource type filter options to ee/audit-logs/constants.ts
  (derived from AuditResourceType, no separate list to maintain)
- Remove export from internal cursor helpers in query.ts
- Add 5 new AuditAction entries: BYOK_KEY_UPDATED, ENVIRONMENT_DELETED,
  INVITATION_RESENT, WORKSPACE_UPDATED, ORG_INVITATION_RESENT
- Enrich ~80 recordAudit calls across the codebase with structured
  metadata (knowledge bases, connectors, documents, workspaces, members,
  invitations, workflows, deployments, templates, MCP servers, credential
  sets, organizations, permission groups, files, tables, notifications,
  copilot operations)
- Sync audit mock with all new entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit): remove redundant metadata fields duplicating top-level audit fields

Remove metadata entries that duplicate resourceName, workspaceId, or
other top-level recordAudit fields. Also remove noisy fileNames arrays
from bulk document upload audits (kept fileCount).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit): split audit types from server-only log module

Extract AuditAction, AuditResourceType, and their types into
lib/audit/types.ts (client-safe, no @sim/db dependency). The
server-only recordAudit stays in log.ts and re-exports the types
for backwards compatibility. constants.ts now imports from types.ts
directly, breaking the postgres -> tls client bundle chain.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit): escape LIKE wildcards in audit log search query

Escape %, _, and \ characters in the search parameter before embedding
in the LIKE pattern to prevent unintended broad matches.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit): use actual deletedCount in bulk API key revoke description

The description was using keys.length (requested count) instead of
deletedCount (actual count), which could differ if some keys didn't
exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(audit-logs): fix OAuth label displaying as "Oauth" in filter dropdown

ACRONYMS set stored 'OAuth' but lookup used toUpperCase() producing
'OAUTH' which never matched. Now store all acronyms uppercase and use
a display override map for special casing like OAuth.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-11 16:15:48 -07:00
Waleed
1189400167 feat(enterprise): cloud whitelabeling for enterprise orgs (#4047)
* feat(enterprise): cloud whitelabeling for enterprise orgs

* fix(enterprise): scope enterprise plan check to target org in whitelabel PUT

* fix(enterprise): use isOrganizationOnEnterprisePlan for org-scoped enterprise check

* fix(enterprise): allow clearing whitelabel fields and guard against empty update result

* fix(enterprise): remove webp from logo accept attribute to match upload hook validation

* improvement(billing): use isBillingEnabled instead of isProd for plan gate bypasses

* fix(enterprise): show whitelabeling nav item when billing is enabled on non-hosted environments

* fix(enterprise): accept relative paths for logoUrl since upload API returns /api/files/serve/ paths

* fix(whitelabeling): prevent logo flash on refresh by hiding logo while branding loads

* fix(whitelabeling): wire hover color through CSS token on tertiary buttons

* fix(whitelabeling): show sim logo by default, only replace when org logo loads

* fix(whitelabeling): cache org logo url in localstorage to eliminate flash on repeat visits

* feat(whitelabeling): add wordmark support with drag/drop upload

* updated turbo

* fix(whitelabeling): defer localstorage read to effect to prevent hydration mismatch

* fix(whitelabeling): use layout effect for cache read to eliminate logo flash before paint

* fix(whitelabeling): cache theme css to eliminate color flash before org settings resolve

* fix(whitelabeling): deduplicate HEX_COLOR_REGEX into lib/branding and remove mutation from useCallback deps

* fix(whitelabeling): use cookie-based SSR cache to eliminate brand flash on all page loads

* fix(whitelabeling): use !orgSettings condition to fix SSR brand cache injection

React Query returns isLoading: false with data: undefined during SSR, so the
previous brandingLoading condition was always false on the server — initialCache
was never injected into brandConfig. Changing to !orgSettings correctly applies
the cookie cache both during SSR and while the client-side query loads, eliminating
the logo flash on hard refresh.
2026-04-08 12:33:26 -07:00
Vikhyath Mondreti
efb582e96a feat(voice): voice input migration to eleven labs (#4041)
* feat(speech): unified voice interface

* add metering for voice input usage

* ip key

* use shared getclientip helper, fix deployed chat

* cleanup code

* prep merge

* merge staging in

* add billing check

* add voice input section

* remove skip billing

* address comments
2026-04-08 01:01:51 -07:00
mini
f46886e6cf fix(sso): default tokenEndpointAuthentication to client_secret_post (#3627)
* fix(sso): default tokenEndpointAuthentication to client_secret_post

better-auth's SSO plugin does not URL-encode credentials before Base64
encoding in client_secret_basic mode (RFC 6749 §2.3.1). When the client
secret contains special characters (+, =, /), OIDC providers decode them
incorrectly, causing invalid_client errors.

Default to client_secret_post when tokenEndpointAuthentication is not
explicitly set to avoid this upstream encoding issue.

Fixes #3626

* fix(sso): use nullish coalescing and add env var for tokenEndpointAuthentication

- Use ?? instead of || for semantic correctness
- Add SSO_OIDC_TOKEN_ENDPOINT_AUTH env var so users can explicitly
  set client_secret_basic when their provider requires it

* docs(sso): add SSO_OIDC_TOKEN_ENDPOINT_AUTH to script usage comment

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

* fix(sso): validate SSO_OIDC_TOKEN_ENDPOINT_AUTH env var value

Replace unsafe `as` type cast with runtime validation to ensure only
'client_secret_post' or 'client_secret_basic' are accepted. Invalid
values (typos, empty strings) now fall back to undefined, letting the
downstream ?? fallback apply correctly.

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>

---------

Signed-off-by: Mini Jeong <mini.jeong@navercorp.com>
2026-04-07 08:46:18 -07:00
Waleed
89ae738745 feat(folders): soft-delete folders and show in Recently Deleted (#4001)
* feat(folders): soft-delete folders and show in Recently Deleted

Folders are now soft-deleted (archived) instead of permanently removed,
matching the existing pattern for workflows, tables, and knowledge bases.
Users can restore folders from Settings > Recently Deleted.

- Add `archivedAt` column to `workflowFolder` schema with index
- Change folder deletion to set `archivedAt` instead of hard-delete
- Add folder restore endpoint (POST /api/folders/[id]/restore)
- Batch-restore all workflows inside restored folders in one transaction
- Add scope filter to GET /api/folders (active/archived)
- Add Folders tab to Recently Deleted settings page
- Update delete modal messaging for restorable items
- Change "This action cannot be undone" styling to muted text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(testing): add FOLDER_RESTORED to audit mock

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): atomic restore transaction and scope to folder-deleted workflows

Address two review findings:
- Wrap entire folder restore in a single DB transaction to prevent
  partial state if any step fails
- Only restore workflows archived within 5s of the folder's archivedAt,
  so individually-deleted workflows are not silently un-deleted
- Add folder_restored to PostHog event map

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(folders): simplify restore to remove hacky 5s time window

The 5-second time window for scoping which workflows to restore was
a fragile heuristic (magic number, race-prone, non-deterministic).
Restoring a folder now restores all archived workflows in it, matching
standard trash/recycle-bin behavior. Users can re-delete any workflow
they don't want after restore.

The single-transaction wrapping from the prior commit is kept — that
was a legitimate atomicity fix.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(db): regenerate folder soft-delete migration with drizzle-kit

Replace manually created migration with proper drizzle-kit generated
one that includes the snapshot file, fixing CI schema sync check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(db): fix migration metadata formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): scope restore to folder-deleted workflows via shared timestamp

Use a single timestamp across the entire folder deletion — folders,
workflows, schedules, webhooks, etc. all get the exact same archivedAt.
On restore, match workflows by exact archivedAt equality with the
folder's timestamp, so individually-deleted workflows are not
silently un-deleted.

- Add optional archivedAt to ArchiveWorkflowOptions (backwards-compatible)
- Pass shared timestamp through deleteFolderRecursively → archiveWorkflowsByIdsInWorkspace
- Filter restore with eq(workflow.archivedAt, folderArchivedAt) instead of isNotNull

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(workflows): clear folderId on restore when folder is archived or missing

When individually restoring a workflow from Recently Deleted, check if
its folder still exists and is active. If the folder is archived or
missing, clear folderId so the workflow appears at root instead of
being orphaned (invisible in sidebar).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): format restoreFolderRecursively call to satisfy biome

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): close remaining restore edge cases

Three issues caught by audit:

1. Child folder restore used isNotNull instead of timestamp matching,
   so individually-deleted child folders would be incorrectly restored.
   Now uses eq(archivedAt, folderArchivedAt) for both workflows AND
   child folders — consistent and deterministic.

2. No workspace archived check — could restore a folder into an
   archived workspace. Now checks getWorkspaceWithOwner, matching
   the existing restoreWorkflow pattern.

3. Re-restoring an already-restored folder returned an error. Now
   returns success with zero counts (idempotent).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): add archivedAt to optimistic folder creation objects

Ensures optimistic folder objects include archivedAt: null for
consistency with the database schema shape.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(folders): handle missing parent folder during restore reparenting

If the parent folder row no longer exists (not just archived), the
restored folder now correctly gets reparented to root instead of
retaining a dangling parentId reference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-06 20:06:04 -07:00
Waleed
a680cec78f fix(core): consolidate ID generation to prevent HTTP self-hosted crashes (#3977)
* fix(core): consolidate ID generation to prevent HTTP self-hosted crashes

crypto.randomUUID() requires a secure context (HTTPS) in browsers,
causing white-screen crashes on self-hosted HTTP deployments. This
replaces all direct usage of crypto.randomUUID(), nanoid, and the uuid
package with a central utility that falls back to crypto.getRandomValues()
which works in all contexts.

- Add generateId(), generateShortId(), isValidUuid() in @/lib/core/utils/uuid
- Replace crypto.randomUUID() imports across ~220 server + client files
- Replace nanoid imports with generateShortId()
- Replace uuid package validate with isValidUuid()
- Remove nanoid dependency from apps/sim and packages/testing
- Remove browser polyfill script from layout.tsx
- Update test mocks to target @/lib/core/utils/uuid
- Update CLAUDE.md, AGENTS.md, cursor rules, claude rules

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* update bunlock

* fix(core): remove UUID_REGEX shim, use isValidUuid directly

* fix(core): remove deprecated uuid mock helpers that use vi.doMock

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 11:28:54 -07:00
Waleed
ce53275e9d feat(knowledge): add Live sync option to KB connectors + fix embedding billing (#3959)
* feat(knowledge): add Live sync option to KB connector modal for Max/Enterprise users

Adds a "Live" (every 5 min) sync frequency option gated to Max and Enterprise plan users.
Includes client-side badge + disabled state, shared sync intervals constant, and server-side
plan validation on both POST and PATCH connector routes.

* fix(knowledge): record embedding usage cost for KB document processing

Adds billing tracking to the KB embedding pipeline, which was previously
generating OpenAI API calls with no cost recorded. Token counts are now
captured from the actual API response and recorded via recordUsage after
successful embedding insertion. BYOK workspaces are excluded from billing.
Applies to all execution paths: direct, BullMQ, and Trigger.dev.

* fix(knowledge): simplify embedding billing — use calculateCost, return modelName

- Use calculateCost() from @/providers/utils instead of inline formula, consistent
  with how LLM billing works throughout the platform
- Return modelName from GenerateEmbeddingsResult so billing uses the actual model
  (handles custom Azure deployments) instead of a hardcoded fallback string
- Fix docs-chunker.ts empty-path fallback to satisfy full GenerateEmbeddingsResult type

* fix(knowledge): remove dev bypass from hasLiveSyncAccess

* chore(knowledge): rename sync-intervals to consts, fix stale TSDoc comment

* improvement(knowledge): extract MaxBadge component, capture billing config once per document

* fix(knowledge): add knowledge-base to usage_log_source enum, fix docs-chunker type

* fix(knowledge): generate migration for knowledge-base usage_log_source enum value

* fix(knowledge): add knowledge-base to usage_log_source enum via drizzle-kit

* fix(knowledge): fix search embedding test mocks, parallelize billing lookups

* fix(knowledge): warn when embedding model has no pricing entry

* fix(knowledge): call checkAndBillOverageThreshold after embedding usage
2026-04-04 16:49:42 -07:00
abhinavDhulipala
7971a64e63 fix(setup): db migrate hard fail and correct ini env (#3946) 2026-04-04 16:22:19 -07:00
Waleed
a78f3f9c2e fix(credential): fix service_account migration to avoid unsafe enum usage in same transaction (#3897) 2026-04-02 10:16:08 -07:00
Theodore Li
fc6fe193fa fix(credential) fix credential migration (#3896)
* fix(credential) fix credential migration

* Fix lint

---------

Co-authored-by: Theodore Li <theo@sim.ai>
2026-04-02 04:29:01 -04:00
Theodore Li
bbc704fe05 feat(credentials) Add google service account support (#3828)
* feat(auth): allow google service account

* Add gmail support for google services

* Refresh creds on typing in impersonated email

* Switch to adding subblock impersonateUserEmail conditionally

* Directly pass subblock for impersonateUserEmail

* Fix lint

* Update documentation for google service accounts

* Fix lint

* Address comments

* Remove hardcoded scopes, remove orphaned migration script

* Simplify subblocks for google service account

* Fix lint

* Fix build error

* Fix documentation scopes listed for google service accounts

* Fix issue with credential selector, remove bigquery and ad support

* create credentialCondition

* Shift conditional render out of subblock

* Simplify sublock values

* Fix security message

* Handle tool service accounts

* Address bugbot

* Fix lint

* Fix manual credential input not showing impersonate

* Fix tests

* Allow watching param id and subblock ids

* Fix bad test

---------

Co-authored-by: Theodore Li <theo@sim.ai>
2026-04-02 03:08:13 -04:00
Waleed
42fb434354 fix(encryption): specify authTagLength on all AES-GCM cipher/decipher calls (#3883)
* fix: specify authTagLength in AES-GCM decipheriv calls

Fixes missing authTagLength parameter in createDecipheriv calls using
AES-256-GCM mode. Without explicit tag length specification, the
application may be tricked into accepting shorter authentication tags,
potentially allowing ciphertext spoofing.

CWE-310: Cryptographic Issues (gcm-no-tag-length)

* fix: specify authTagLength on createCipheriv calls for AES-GCM consistency

Complements #3881 by adding explicit authTagLength: 16 to the encrypt
side as well, ensuring both cipher and decipher specify the tag length.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: clean up crypto modules

- Fix error: any → error: unknown with proper type guard in encryption.ts
- Eliminate duplicate iv.toString('hex') calls in both encrypt functions
- Remove redundant string split in decryptApiKey (was splitting twice)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* new turborepo version

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lakee Sivaraya <71339072+lakeesiv@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
Co-authored-by: NLmejiro <kuroda.k1021@gmail.com>
2026-04-01 10:46:58 -07:00
Vikhyath Mondreti
0abeac77e1 improvement(platform): standardize perms, audit logging, lifecycle across admin, copilot, ui actions (#3858)
* improvement(platform): standardize perms, audit logging, lifecycle mgmt across admin, copilot, ui actions

* address comments

* improve error codes

* address bugbot comments

* fix test
2026-03-30 20:25:38 -07:00
Theodore Li
d3d58a9615 Feat/improved logging (#3833)
* feat(logs): add additional metadata for workflow execution logs

* Revert "Feat(logs) upgrade mothership chat messages to error (#3772)"

This reverts commit 9d1b9763c5.

* Fix lint, address greptile comments

* improvement(sidebar): expand sidebar by hovering and clicking the edge (#3830)

* improvement(sidebar): expand sidebar by hovering and clicking the edge

* improvement(sidebar): add keyboard shortcuts for new workflow/task, center search modal, fix edge ARIA

* improvement(sidebar): use Tooltip.Shortcut for inline shortcut display

* fix(sidebar): change new workflow shortcut from Mod+Shift+W to Mod+Shift+P to avoid browser close-window conflict

* fix(hotkeys): fall back to event.code for international keyboard layout compatibility

* fix(sidebar): guard add-workflow shortcut with canEdit and isCreatingWorkflow checks

* feat(ui): handle image paste (#3826)

* feat(ui): handle image paste

* Fix lint

* Fix type error

---------

Co-authored-by: Theodore Li <theo@sim.ai>

* feat(files): interactive markdown checkbox toggling in preview (#3829)

* feat(files): interactive markdown checkbox toggling in preview

* fix(files): handle ordered-list checkboxes and fix index drift

* lint

* fix(files): remove counter offset that prevented checkbox toggling

* fix(files): apply task-list styling to ordered lists too

* fix(files): render single pass when interactive to avoid index drift

* fix(files): move useMemo above conditional return to fix Rules of Hooks

* fix(files): pass content directly to preview when not streaming to avoid stale frame

* improvement(home): position @ mention popup at caret and fix icon consistency (#3831)

* improvement(home): position @ mention popup at caret and fix icon consistency

* fix(home): pin mirror div to document origin and guard button anchor

* chore(auth): restore hybrid.ts to staging

* improvement(ui): sidebar (#3832)

* Fix logger tests

* Add metadata to mothership logs

---------

Co-authored-by: Theodore Li <theo@sim.ai>
Co-authored-by: Waleed <walif6@gmail.com>
Co-authored-by: Theodore Li <theo@sim.ai>
2026-03-30 19:02:17 -04:00
Waleed
e2be99263c feat(academy): Sim Academy — interactive partner certification platform (#3824)
* fix(import): dedup workflow name (#3813)

* feat(concurrency): bullmq based concurrency control system (#3605)

* feat(concurrency): bullmq based queueing system

* fix bun lock

* remove manual execs off queues

* address comments

* fix legacy team limits

* cleanup enterprise typing code

* inline child triggers

* fix status check

* address more comments

* optimize reconciler scan

* remove dead code

* add to landing page

* Add load testing framework

* update bullmq

* fix

* fix headless path

---------

Co-authored-by: Theodore Li <teddy@zenobiapay.com>

* fix(linear): add default null for after cursor (#3814)

* fix(knowledge): reject non-alphanumeric file extensions from document names (#3816)

* fix(knowledge): reject non-alphanumeric file extensions from document names

* fix(knowledge): improve error message when extension is non-alphanumeric

* fix(security): SSRF, access control, and info disclosure (#3815)

* fix(security): scope copilot feedback GET endpoint to authenticated user

Add WHERE clause to filter feedback records by the authenticated user's
ID, preventing any authenticated user from reading all users' copilot
interactions, queries, and workflow YAML (IDOR / CWE-639).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(smtp): add SSRF validation and genericize network error messages

Prevent SSRF via user-controlled smtpHost by validating with
validateDatabaseHost before creating the nodemailer transporter.
Collapse distinct network error messages (ECONNREFUSED, ECONNRESET,
ETIMEDOUT) into a single generic message to prevent port-state leakage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): add SSRF validation to SFTP/SSH and access control to workspace invitations

Add `validateDatabaseHost` checks to SFTP and SSH connection utilities to
block connections to private/reserved IPs and localhost, matching the
existing pattern used by all database tools. Add authorization check to
the workspace invitation GET endpoint so only the invitee or a workspace
admin can view invitation details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(smtp): restore SMTP response code handling for post-connection errors

SMTP 4xx/5xx response codes are application-level errors (invalid
recipient, mailbox full, server error) unrelated to the SSRF hardening
goal. Restore response code differentiation and logging to preserve
actionable user-facing error messages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): use session email directly instead of extra DB query

Addresses PR review feedback — align with the workspace invitation
route pattern by using session.user.email instead of re-fetching
from the database.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* fix(auth): revert lint autofix that broke hasExternalApiCredentials return type

Biome auto-fixed `return auth !== null && auth.startsWith(...)` to
`return auth?.startsWith(...)` which returns `boolean | undefined`,
not `boolean`, causing a TypeScript build failure.

* fix(smtp): pin resolved IP to prevent DNS rebinding (TOCTOU)

Use the pre-resolved IP from validateDatabaseHost instead of the
original hostname when creating the nodemailer transporter. Set
servername to the original hostname to preserve TLS SNI validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(security): extract createPinnedLookup helper for DNS rebinding prevention

Extract reusable createPinnedLookup from secureFetchWithPinnedIP so
non-HTTP transports (SSH, SFTP, IMAP) can pin resolved IPs at the
socket level. SMTP route uses host+servername pinning instead since
nodemailer doesn't reliably pass lookup to both secure/plaintext paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): pin IMAP connections to validated resolved IP

Pass the resolved IP from validateDatabaseHost to ImapFlow as host,
with the original hostname as servername for TLS SNI verification.
Closes the DNS TOCTOU rebinding window.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* fix(auth): revert lint autofix on hasExternalApiCredentials return type

Also pin SFTP/SSH connections to validated resolved IP to prevent DNS rebinding.

* fix(security): short-circuit admin check when caller is invitee

Skip the hasWorkspaceAdminAccess DB query when the caller is already
the invitee, avoiding an unnecessary round-trip. Aligns with the org
invitation route pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* fix(worker): dockerfile + helm updates (#3818)

* fix(worker): dockerfile + helm updates

* address comments

* update dockerfile (#3819)

* fix dockerfile

* fix(security): pentest remediation — condition escaping, SSRF hardening, ReDoS protection (#3820)

* fix(executor): escape newline characters in condition expression strings

Unescaped newline/carriage-return characters in resolved string values
cause unterminated string literals in generated JS, crashing condition
evaluation with a SyntaxError.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): prevent ReDoS in guardrails regex validation

Add safe-regex2 to reject catastrophic backtracking patterns before
execution and cap input length at 10k characters.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): SSRF localhost hardening and regex DoS protection

Block localhost/loopback URLs in hosted environments using isHosted flag
instead of allowHttp. Add safe-regex2 validation and input length limits
to regex guardrails to prevent catastrophic backtracking.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): validate regex syntax before safety check

Move new RegExp() before safe() so invalid patterns get a proper syntax
error instead of a misleading "catastrophic backtracking" message.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): address PR review feedback

- Hoist isLocalhost && isHosted guard to single early-return before
  protocol checks, removing redundant duplicate block
- Move regex syntax validation (new RegExp) before safe-regex2 check
  so invalid patterns get proper syntax error instead of misleading
  "catastrophic backtracking" message

* fix(security): remove input length cap from regex validation

The 10k character cap would block legitimate guardrail checks on long
LLM outputs. Input length doesn't affect ReDoS risk — the safe-regex2
pattern check already prevents catastrophic backtracking.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(tests): mock isHosted in input-validation and function-execute tests

Tests that assert self-hosted localhost behavior need isHosted=false,
which is not guaranteed in CI where NEXT_PUBLIC_APP_URL is set to the
hosted domain.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* improvement(worker): configuration defaults (#3821)

* improvement(worker): configuration defaults

* update readmes

* realtime curl import

* improvement(tour): remove auto-start, only trigger on explicit user action (#3823)

* fix(mcp): use correct modal for creating workflow MCP servers in deploy (#3822)

* fix(mcp): use correct modal for creating workflow MCP servers in deploy

* fix(mcp): show workflows field during loading and when empty

* mock course

* fix(db): use bigint for token counter columns in user_stats (#3755)

* mock course

* updates

* updated X handle for emir

* cleanup: audit and clean academy implementation

* fix(academy): add label to ValidationRule, fix quiz gating, simplify getRuleMessage

* cleanup: remove unnecessary comments across academy files

* refactor(academy): simplify abstractions and fix perf issues

* perf(academy): convert course detail page to server component with client island

* fix(academy): null-safe canAdvance, render exercise instructions, remove stale comments

* fix(academy): remove orphaned migration, fix getCourseById, clean up comments

- Delete 0181_academy_certificate.sql (orphaned duplicate not in journal)
- Add getCourseById() to content/index.ts; use it in certificates API
  (was using getCourse which searches by slug, not stable id)
- Remove JSX comments from catalog page
- Remove redundant `passed` recomputation in LessonQuiz

* chore(db): regenerate academy_certificate migration with drizzle-kit

* chore: include blog mdx and components changes

* fix(blog): correct cn import path

* fix(academy): constrain progress bar to max-w-3xl with proper padding

* feat(academy): show back-to-course button on first lesson

* fix(academy): force dark theme on all /academy routes

* content(academy): rewrite sim-foundations course with full 6-module curriculum

* fix(academy): correct edge handles, quiz explanation, and starter mock outputs

- Fix Exercise 2 initial edge handles: 'starter-1-source'/'agent-1-target' → 'source'/'target' (React Flow actual IDs)
- Fix M1-L4 Q4 quiz explanation: remove non-existent Ctrl/Cmd+D and Alt+drag shortcuts
- Add starter mock output to all exercises so run animation shows feedback on the first block

* refine(academy): fix inaccurate content and improve exercise clarity

- Fix Exercise 3: replace hardcoded <agent-1.content> (invalid UUID-based ref) with reference picker instructions
- Fix M4 Quiz Q5: Loop block (subflow container) is correct answer, not the Workflow block
- Fix M4 Quiz Q4: clarify fan-out vs Parallel block distinction in explanation
- Fix M4-L2 video description: accurately describe Loop and Parallel subflow blocks
- Fix M2 Quiz Q3: make response format question conceptual rather than syntax-specific
- Improve Exercise 4 branching instructions: clarify top=true / bottom=false output handles
- Improve Final Project instructions: step-by-step numbered flow

* fix(academy): remove double border on quiz question cards

* fix(academy): single scroll container on lesson pages — remove nested flex scroll

* fix(academy): remove min-h-screen from root layout — fixes double scrollbar on lesson pages

* fix(academy): use fixed inset-0 on lesson page to eliminate document-level scrollbar

* fix(academy): replace sr-only radio/checkbox inputs with buttons to prevent scroll-on-focus; restore layout min-h-screen

* improvement(academy): polish, security hardening, and certificate claim UI

- Replace raw localStorage with BrowserStorage utility in local-progress
- Pre-compute slug/id Maps in content/index for O(1) course lookups
- Move blockMap construction into edge_exists branch only in validation
- Extract navBtnClass constant and MetaRow/formatDate helpers in UI
- Add rate limiting, server-side completion verification, audit logging, and nanoid cert numbers to certificate issuance endpoint
- Add useIssueCertificate mutation hook with completedLessonIds
- Wire certificate claim UI into CourseProgress: sign-in prompt, claim button with loading state, and post-issuance view with link to certificate page
- Fix lesson page scroll container and quiz scroll-on-focus bug

* fix(academy): validate condition branch handles in edge_exists rules

- Add sourceHandle field to edge_exists ValidationRule type
- Check sourceHandle in validation.ts when specified
- Require both condition-if and condition-else branches to be connected in the branching and final project exercises

* fix(academy): address PR review — isHosted regression, stuck isExecuting, revoked cert 500, certificate SSR

- Restore env-var-based isHosted check (was hardcoded true, breaking self-hosted deployments)
- Fix isExecuting stuck at true when mock run fails validation — set isMockRunningRef immediately and reset both flags on early exit
- Fix revoked/expired certificate causing 500 — any existing record (not just active) now returns 409 instead of falling through to INSERT
- Convert certificate verification page from client component to server component — direct DB fetch, notFound() on missing cert, generateMetadata for SEO/social previews

* fix(auth): restore hybrid.ts from staging to fix CI type error

* fix(academy): mark video lessons complete on visit and fix sign-in path

* fix(academy): replace useEffect+setState with lazy useState initializer in CourseProgress

* fix(academy): reset exerciseComplete on lesson navigation, remove unused useAcademyCertificate hook

* fix(academy): useState for slug-change reset, cache() for cert page, handleMockRunRef for stale closure

* fix(academy): replace shadcn theme vars with explicit hex in LessonVideo fallback

* fix(academy): reset completedRef on exercise change, conditional verified badge, multi-select empty guard

* fix(academy): type safety fixes — null metadata fallbacks, returning() guard, exhaustive union, empty catch

* fix(academy): reset ExerciseView completed banner on nav; fix CourseProgress hydration mismatch

* fix(lightbox): guard effect body with isOpen to prevent spurious overflow reset

* fix(academy): reset LessonQuiz state on lesson change to prevent stale answers persisting

* fix(academy): course not-found metadata title; try-finally guard in mock run loop

* fix(academy): type safety, cert persistence, regex guard, mixed-lesson video, shorts support

- Derive AcademyCertificate from db $inferSelect to prevent schema drift
- Add useCourseCertificate query hook; GET /api/academy/certificates now accepts courseId for authenticated lookup
- Use useCourseCertificate in CourseProgress so certificate state survives page refresh
- Guard new RegExp(valuePattern) in validation.ts with try/catch; log warn on invalid pattern
- Add logger.warn for custom validation rules so content authors are alerted
- Add YouTube Shorts URL support to LessonVideo (youtube.com/shorts/VIDEO_ID)
- Fix mixed-lesson video gap: render videoUrl above quiz when mixed has quiz but no exercise
- Add academy-scoped not-found.tsx with link back to /academy

* fix(academy): reset hintIndex when exercise changes

* chore: remove ban-spam-accounts script (wrong branch)

* fix(academy): enforce availableBlocks in toolbar; fix mixed exercise+quiz rendering

- Add useSandboxBlockConstraints context; SandboxCanvasProvider provides exerciseConfig.availableBlocks so the toolbar only shows permitted block types. Empty array hides all blocks (configure-only exercises); non-null array restricts to listed types; triggers always hidden in sandbox.
- Fix mixed lesson with both exerciseConfig and quizConfig: exercise renders first, quiz reveals after exercise completes (sequential pedagogy). canAdvance now requires both exerciseComplete && quizComplete when both are present.

* chore(academy): remove extraneous inline comments

* fix(academy): blank mixed lesson, quiz canAdvance flag, empty-array valueNotEmpty

* prep for merge

* chore(db): regenerate academy certificate migration after staging merge

* fix(academy): disable auto-connect in sandbox mode

* fix(academy): render video in mixed lesson with no exercise or quiz

* fix(academy): mark mixed video-only lessons complete; handle cert insert race

* fix(canvas): add sandbox and embedded to nodes useMemo deps

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lakee Sivaraya <71339072+lakeesiv@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
Co-authored-by: Theodore Li <teddy@zenobiapay.com>
2026-03-28 12:46:04 -07:00
Waleed
e4d35735b1 fix(knowledge): give users choice to keep or delete documents when removing connector (#3825)
* fix(knowledge): give users choice to keep or delete documents when removing connector

* refactor(knowledge): clean up connector delete and extract shared extension validator

- Extract `isAlphanumericExtension` helper to deduplicate regex across parser-extension.ts and validation.ts
- Extract `closeDeleteModal` callback to eliminate 4x scattered state resets
- Add archivedAt/deletedAt filters to UPDATE query in keep-docs delete path
- Parallelize storage file cleanup and tag definition cleanup with Promise.all
- Deduplicate URL construction in delete connector hook

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(knowledge): remove duplicate extension list from parser-extension

Use SUPPORTED_DOCUMENT_EXTENSIONS and isSupportedExtension from
validation.ts instead of maintaining a separate identical list.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(db): change document.connectorId FK from cascade to set null

The cascade behavior meant deleting a connector would always delete
its documents, contradicting the "keep documents" option. With set null,
the database automatically nullifies connectorId when a connector is
removed, and we only need explicit deletion when the user opts in.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(db): add migration metadata for connectorId FK change

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(knowledge): fix connector delete test and use URL-safe searchParams

Use `new URL(request.url).searchParams` instead of `request.nextUrl.searchParams`
for compatibility with test mocks. Add missing `connectorType` to test fixture.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* spacing

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-28 12:04:26 -07:00
Waleed
8a481b612d chore(config): clean up bun, turbo, and next.js config (#3788)
* chore(config): clean up bun, turbo, and next.js config

* chore(ci): bump bun to 1.3.11 in dockerfiles and workflows
2026-03-26 13:04:23 -07:00
Waleed
666dc67aa2 fix(db): use bigint for token counter columns in user_stats (#3755) 2026-03-24 21:08:07 -07:00
Siddharth Ganesan
775daed2ea fix(mothership): tool call loop (#3729)
* v0

* Fix ppt load

* Fixes

* Fixes

* Fix lint

* Fix wid

* Download image

* Update tools

* Fix lint

* Fix error msg

* Tool fixes

* Reenable subagent stream

* Subagent stream

* Fix edit workflow hydration

* Throw func execute error on error

* Rewrite

* Remove promptForToolApproval flag, fix workflow terminal logs

* Fixes

* Fix buffer

* Fix

* Fix claimed by

* Cleanup v1

* Tool call loop

* Fixes

* Fixes

* Fix subaget aborts

* Fix diff

* Add delegating state to subagents

* Fix build

* Fix sandbox

* Fix lint

---------

Co-authored-by: Waleed <walif6@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lakee Sivaraya <71339072+lakeesiv@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Theodore Li <teddy@zenobiapay.com>
2026-03-23 18:11:06 -07:00
Vikhyath Mondreti
161424601f fix migration 2026-03-22 01:19:19 -07:00
Siddharth Ganesan
d6bf12da24 improvement(mothership): copilot, files, compaction, tools, persistence, duplication constraints (#3682)
* Improve

* Hide is hosted

* Remove hardcoded

* fix

* Fixes

* v0

* Fix bugs

* Restore settings

* Handle compaction event type

* Add keepalive

* File streaming

* Error tags

* Abort defense

* Edit hashes

* DB backed tools

* Fixes

* progress on autolayout improvements

* Abort fixes

* vertical insertion improvement

* Consolidate file attachments

* Fix lint

* Manage agent result card fix

* Remove hardcoded ff

* Fix file streaming

* Fix persisted writing file tab

* Fix lint

* Fix streaming file flash

* Always set url to /file on file view

* Edit perms for tables

* Fix file edit perms

* remove inline tool call json dump

* Enforce name uniqueness (#3679)

* Enforce name uniqueness

* Use established pattern for error handling

* Fix lint

* Fix lint

* Add kb name uniqueness to db

* Fix lint

* Handle name getting taken before restore

* Enforce duplicate file name

* Fix lint

---------

Co-authored-by: Theodore Li <theo@sim.ai>

* fix temp file creation

* fix types

* Streaming fixes

* type xml tag structures + return invalid id linter errors back to LLM

* Add image gen and viz tools

* Tags

* Workflow tags

* Fix lint

* Fix subagent abort

* Fix subagent persistence

* Fix subagent aborts

* Nuke db migs

* Re add db migrations

* Fix lint

---------

Co-authored-by: Theodore Li <teddy@zenobiapay.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Theodore Li <theodoreqili@gmail.com>
Co-authored-by: Theodore Li <theo@sim.ai>
2026-03-22 00:46:13 -07:00
Waleed
4a34ac3015 feat(auth): add Turnstile captcha + harmony disposable email blocking (#3699)
* feat(turnstile): conditionally added CF turnstile to signup

* feat(auth): add execute-on-submit Turnstile, conditional harmony, and feature flag

- Switch Turnstile to execution: 'execute' mode so challenge runs on
  form submit (fresh token every time, no expiry issues)
- Make emailHarmony conditional via SIGNUP_EMAIL_VALIDATION_ENABLED
  feature flag so self-hosted users can opt out
- Add isSignupEmailValidationEnabled to feature-flags.ts following
  existing pattern
- Add better-auth-harmony to Next.js transpilePackages (required for
  validator.js ESM compatibility)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(validation): remove dead validateEmail and checkMXRecord

Server-side disposable email blocking is now handled by
better-auth-harmony. The async validateEmail (with MX check) had no
remaining callers. Only quickValidateEmail remains for client-side
form feedback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(auth): add 15s timeout to Turnstile captcha promise

Prevents form from hanging indefinitely if Turnstile never fires
onSuccess/onError (e.g. script fails to load, network drop).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(helm): add Turnstile and harmony env vars to values.yaml

Adds TURNSTILE_SECRET_KEY, NEXT_PUBLIC_TURNSTILE_SITE_KEY, and
SIGNUP_EMAIL_VALIDATION_ENABLED to the helm chart so self-hosted
deployments can configure captcha and disposable email blocking.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(auth): reject captcha promise on token expiry

onExpire now rejects the pending promise so the form doesn't hang
if the Turnstile token expires mid-challenge.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(login): replace useEffect keydown listener with form onSubmit

The forgot-password modal used a global window keydown listener in a
useEffect to handle Enter key — a "you might not need an effect"
anti-pattern with a stale closure risk. Replaced with a native
<form onSubmit> wrapper which handles Enter natively, eliminating
the useEffect, the global listener, and the stale closure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(auth): clear dangling timeout after captcha promise settles

Use .finally(() => clearTimeout(timeoutId)) to clean up the 15s
timeout timer when the captcha resolves before the deadline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(auth): use getResponsePromise() for Turnstile token retrieval

Replace the manual Promise + refs + timeout pattern with the
documented getResponsePromise(timeout) API from @marsidev/react-turnstile.
This eliminates captchaToken state, captchaResolveRef, captchaRejectRef,
and all callback wiring on the Turnstile component.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(auth): show captcha errors as form-level message, not password error

Captcha failures were misleadingly displayed under the password field.
Added a dedicated formError state that renders above the submit button,
making it clear the issue is with verification, not the password.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-21 11:23:45 -07:00
Waleed
a64afac075 feat(kb): harden sync engine and add connector audit logging (#3697)
* feat(kb): harden sync engine and add connector audit logging

- Fix stuck syncing status: added finally block in executeSync + stale lock recovery in cron scheduler (2hr TTL)
- Fix token expiry mid-sync: refresh OAuth token between pagination pages and before deferred content hydration
- GitHub deferred content loading: use Git blob SHA for change detection, only fetch content for new/changed docs
- Add network error keywords to isRetryableError (fetch failed, econnreset, etc.)
- Extract sanitizeStorageTitle helper to fix S3 key length limit issues
- Add audit logging for connector CRUD, sync triggers, document exclude/restore, and resource restoration paths

* lint

* fix(tests): update audit mock and route tests for new audit actions

* fix(kb): address PR review - finally block race, contentHash propagation, resourceName

- Replace DB-read finally block with local syncExitedCleanly flag to avoid race condition
- Propagate fullDoc.contentHash during deferred content hydration
- Add resourceName to file restore audit record

* fix(audit): include fileId in file restore audit description
2026-03-21 09:36:43 -07:00
Waleed
b84f30e9e7 fix(db): reduce connection pool sizes to prevent exhaustion (#3649) 2026-03-18 02:20:55 -07:00
Waleed
25a03f1f3c feat(auth): migrate to better-auth admin plugin with unified Admin tab (#3612)
* feat(auth): migrate to better-auth admin plugin

* feat(settings): add unified Admin tab with user management

Consolidate superuser features into a single Admin settings tab:
- Super admin mode toggle (moved from General)
- Workflow import (moved from Debug)
- User management via better-auth admin (list, set role, ban/unban)

Replace Debug tab with Admin tab gated by requiresAdminRole.
Add React Query hooks for admin user operations.

* fix(db): backfill existing super users to admin role in migration

Add UPDATE statement to promote is_super_user=true rows to role='admin'
before dropping the is_super_user column, preventing silent demotion.

* fix(admin): resolve type errors in admin tab

- Fix cn import path to @/lib/core/utils/cn
- Use valid Badge variants (blue/gray/red/green instead of secondary/destructive)
- Type setRole param as 'user' | 'admin' union

* improvement(auth): remove /api/user/super-user route, use session role

Include user.role in customSession so it's available client-side.
Replace all useSuperUserStatus() calls with session.user.role === 'admin'.
Delete the now-redundant /api/user/super-user endpoint.

* chore(auth): remove redundant role override in customSession

The admin plugin already includes role on the user object.
No need to manually spread it in customSession.

* improvement(queries): clean up admin-users hooks per React Query best practices

- Remove unsafe unknown/Record casting, use better-auth typed response
- Add placeholderData: keepPreviousData for paginated variable-key query
- Remove nullable types where defaults are always applied

* fix(admin): address review feedback on admin tab

- Fix superUserModeEnabled default to false (matches sidebar behavior)
- Reset banReason when switching ban target to prevent state bleed
- Guard admin section render with session role check for direct URL access

* fix(settings): align superUserModeEnabled default to false everywhere

Three places defaulted to true while admin tab and sidebar used false.
Align all to false so new admins see consistent behavior.

* fix(admin): fix stale pendingUserId, add isPending guard and error feedback

- Only read mutation.variables when mutation isPending (prevents stale ID)
- Add isPending guard to super user mode toggle (prevents concurrent mutations)
- Show inline error message when setRole/ban/unban mutations fail

* fix(admin): concurrent pending users Set, session loading guard, domain blocking

- Replace pendingUserId scalar with pendingUserIds Set (useMemo) so concurrent
  mutations across different users each disable their own row correctly
- Add sessionLoading guard to admin section redirect to prevent flash on direct
  /settings/admin navigation before session resolves
- Add BLOCKED_SIGNUP_DOMAINS env var and before-hook for email domain denylist,
  parsed once at module init as a Set for O(1) per-request lookups
- Add trailing newline to migration file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(admin): close OAuth domain bypass, fix stale errors, deduplicate icon

- Add databaseHooks.user.create.before to enforce BLOCKED_SIGNUP_DOMAINS at
  the model level, covering all signup vectors (email, OAuth, social) not just
  /sign-up paths
- Call .reset() on each mutation before firing to clear stale error state from
  previous operations
- Change Admin nav icon from ShieldCheck to Lock to avoid duplicate with
  Access Control tab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 15:04:54 -07:00