Compare commits

..

43 Commits

Author SHA1 Message Date
Waleed
028bc652c2 v0.5.76: posthog improvements, readme updates 2026-01-29 00:13:19 -08:00
Waleed
c6bf5cd58c v0.5.75: search modal overhaul, helm chart updates, run from block, terminal and visual debugging improvements 2026-01-28 22:54:13 -08:00
Vikhyath Mondreti
11dc18a80d v0.5.74: autolayout improvements, clerk integration, auth enforcements 2026-01-27 20:37:39 -08:00
Waleed
ab4e9dc72f v0.5.73: ci, helm updates, kb, ui fixes, note block enhancements 2026-01-26 22:04:35 -08:00
Vikhyath Mondreti
1c58c35bd8 v0.5.72: azure connection string, supabase improvement, multitrigger resolution, docs quick reference 2026-01-25 23:42:27 -08:00
Waleed
d63a5cb504 v0.5.71: ux, ci improvements, docs updates 2026-01-25 03:08:08 -08:00
Waleed
8bd5d41723 v0.5.70: router fix, anthropic agent response format adherence 2026-01-24 20:57:02 -08:00
Waleed
c12931bc50 v0.5.69: kb upgrades, blog, copilot improvements, auth consolidation (#2973)
* fix(subflows): tag dropdown + resolution logic (#2949)

* fix(subflows): tag dropdown + resolution logic

* fixes;

* revert parallel change

* chore(deps): bump posthog-js to 1.334.1 (#2948)

* fix(idempotency): add conflict target to atomicallyClaimDb query + remove redundant db namespace tracking (#2950)

* fix(idempotency): add conflict target to atomicallyClaimDb query

* delete needs to account for namespace

* simplify namespace filtering logic

* fix cleanup

* consistent target

* improvement(kb): add document filtering, select all, and React Query migration (#2951)

* improvement(kb): add document filtering, select all, and React Query migration

* test(kb): update tests for enabledFilter and removed userId params

* fix(kb): remove non-null assertion, add explicit guard

* improvement(logs): trace span, details (#2952)

* improvement(action-bar): ordering

* improvement(logs): details, trace span

* feat(blog): v0.5 release post (#2953)

* feat(blog): v0.5 post

* improvement(blog): simplify title and remove code block header

- Simplified blog title from Introducing Sim Studio v0.5 to Introducing Sim v0.5
- Removed language label header and copy button from code blocks for cleaner appearance

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ack PR comments

* small styling improvements

* created system to create post-specific components

* updated componnet

* cache invalidation

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* feat(admin): add credits endpoint to issue credits to users (#2954)

* feat(admin): add credits endpoint to issue credits to users

* fix(admin): use existing credit functions and handle enterprise seats

* fix(admin): reject NaN and Infinity in amount validation

* styling

* fix(admin): validate userId and email are strings

* improvement(copilot): fast mode, subagent tool responses and allow preferences (#2955)

* Improvements

* Fix actions mapping

* Remove console logs

* fix(billing): handle missing userStats and prevent crashes (#2956)

* fix(billing): handle missing userStats and prevent crashes

* fix(billing): correct import path for getFilledPillColor

* fix(billing): add Number.isFinite check to lastPeriodCost

* fix(logs): refresh logic to refresh logs details (#2958)

* fix(security): add authentication and input validation to API routes (#2959)

* fix(security): add authentication and input validation to API routes

* moved utils

* remove extraneous commetns

* removed unused dep

* improvement(helm): add internal ingress support and same-host path consolidation (#2960)

* improvement(helm): add internal ingress support and same-host path consolidation

* improvement(helm): clean up ingress template comments

Simplify verbose inline Helm comments and section dividers to match the
minimal style used in services.yaml.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(helm): add missing copilot path consolidation for realtime host

When copilot.host equals realtime.host but differs from app.host,
copilot paths were not being routed. Added logic to consolidate
copilot paths into the realtime rule for this scenario.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* improvement(helm): follow ingress best practices

- Remove orphan comments that appeared when services were disabled
- Add documentation about path ordering requirements
- Paths rendered in order: realtime, copilot, app (specific before catch-all)
- Clean template output matching industry Helm chart standards

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* feat(blog): enterprise post (#2961)

* feat(blog): enterprise post

* added more images, styling

* more content

* updated v0-5 post

* remove unused transition

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>

* fix(envvars): resolution standardized (#2957)

* fix(envvars): resolution standardized

* remove comments

* address bugbot

* fix highlighting for env vars

* remove comments

* address greptile

* address bugbot

* fix(copilot): mask credentials fix (#2963)

* Fix copilot masking

* Clean up

* Lint

* improvement(webhooks): remove dead code (#2965)

* fix(webhooks): subscription recreation path

* improvement(webhooks): remove dead code

* fix tests

* address bugbot comments

* fix restoration edge case

* fix more edge cases

* address bugbot comments

* fix gmail polling

* add warnings for UI indication for credential sets

* fix(preview): subblock values (#2969)

* fix(child-workflow): nested spans handoff (#2966)

* fix(child-workflow): nested spans handoff

* remove overly defensive programming

* update type check

* type more code

* remove more dead code

* address bugbot comments

* fix(security): restrict API key access on internal-only routes (#2964)

* fix(security): restrict API key access on internal-only routes

* test(security): update function execute tests for checkInternalAuth

* updated agent handler

* move session check higher in checkSessionOrInternalAuth

* extracted duplicate code into helper for resolving user from jwt

* fix(copilot): update copilot chat title (#2968)

* fix(hitl): fix condition blocks after hitl (#2967)

* fix(notes): ghost edges (#2970)

* fix(notes): ghost edges

* fix deployed state fallback

* fallback

* remove UI level checks

* annotation missing from autoconnect source check

* improvement(docs): loop and parallel var reference syntax (#2975)

* fix(blog): slash actions description (#2976)

* improvement(docs): loop and parallel var reference syntax

* fix(blog): slash actions description

* fix(auth): copilot routes (#2977)

* Fix copilot auth

* Fix

* Fix

* Fix

* fix(copilot): fix edit summary for loops/parallels (#2978)

* fix(integrations): hide from tool bar (#2544)

* fix(landing): ui (#2979)

* fix(edge-validation): race condition on collaborative add (#2980)

* fix(variables): boolean type support and input improvements (#2981)

* fix(variables): boolean type support and input improvements

* fix formatting

---------

Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
2026-01-24 14:29:53 -08:00
Waleed
e9c4251c1c v0.5.68: router block reasoning, executor improvements, variable resolution consolidation, helm updates (#2946)
* improvement(workflow-item): stabilize avatar layout and fix name truncation (#2939)

* improvement(workflow-item): stabilize avatar layout and fix name truncation

* fix(avatars): revert overflow bg to hardcoded color for contrast

* fix(executor): stop parallel execution when block errors (#2940)

* improvement(helm): add per-deployment extraVolumes support (#2942)

* fix(gmail): expose messageId field in read email block (#2943)

* fix(resolver): consolidate reference resolution  (#2941)

* fix(resolver): consolidate code to resolve references

* fix edge cases

* use already formatted error

* fix multi index

* fix backwards compat reachability

* handle backwards compatibility accurately

* use shared constant correctly

* feat(router): expose reasoning output in router v2 block (#2945)

* fix(copilot): always allow, credential masking (#2947)

* Fix always allow, credential validation

* Credential masking

* Autoload

* fix(executor): handle condition dead-end branches in loops (#2944)

---------

Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Siddharth Ganesan <33737564+Sg312@users.noreply.github.com>
2026-01-22 13:48:15 -08:00
Waleed
cc2be33d6b v0.5.67: loading, password reset, ui improvements, helm updates (#2928)
* fix(zustand): updated to useShallow from deprecated createWithEqualityFn (#2919)

* fix(logger): use direct env access for webpack inlining (#2920)

* fix(notifications): text overflow with line-clamp (#2921)

* chore(helm): add env vars for Vertex AI, orgs, and telemetry (#2922)

* fix(auth): improve reset password flow and consolidate brand detection (#2924)

* fix(auth): improve reset password flow and consolidate brand detection

* fix(auth): set errorHandled for EMAIL_NOT_VERIFIED to prevent duplicate error

* fix(auth): clear success message on login errors

* chore(auth): fix import order per lint

* fix(action-bar): duplicate subflows with children (#2923)

* fix(action-bar): duplicate subflows with children

* fix(action-bar): add validateTriggerPaste for subflow duplicate

* fix(resolver): agent response format, input formats, root level (#2925)

* fix(resolvers): agent response format, input formats, root level

* fix response block initial seeding

* fix tests

* fix(messages-input): fix cursor alignment and auto-resize with overlay (#2926)

* fix(messages-input): fix cursor alignment and auto-resize with overlay

* fixed remaining zustand warnings

* fix(stores): remove dead code causing log spam on startup (#2927)

* fix(stores): remove dead code causing log spam on startup

* fix(stores): replace custom tools zustand store with react query cache

* improvement(ui): use BrandedButton and BrandedLink components (#2930)

- Refactor auth forms to use BrandedButton component
- Add BrandedLink component for changelog page
- Reduce code duplication in login, signup, reset-password forms
- Update star count default value

* fix(custom-tools): remove unsafe title fallback in getCustomTool (#2929)

* fix(custom-tools): remove unsafe title fallback in getCustomTool

* fix(custom-tools): restore title fallback in getCustomTool lookup

Custom tools are referenced by title (custom_${title}), not database ID.
The title fallback is required for client-side tool resolution to work.

* fix(null-bodies): empty bodies handling (#2931)

* fix(null-statuses): empty bodies handling

* address bugbot comment

* fix(token-refresh): microsoft, notion, x, linear (#2933)

* fix(microsoft): proactive refresh needed

* fix(x): missing token refresh flag

* notion and linear missing flag too

* address bugbot comment

* fix(auth): handle EMAIL_NOT_VERIFIED in onError callback (#2932)

* fix(auth): handle EMAIL_NOT_VERIFIED in onError callback

* refactor(auth): extract redirectToVerify helper to reduce duplication

* fix(workflow-selector): use dedicated selector for workflow dropdown (#2934)

* feat(workflow-block): preview (#2935)

* improvement(copilot): tool configs to show nested props (#2936)

* fix(auth): add genericOAuth providers to trustedProviders (#2937)

---------

Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com>
2026-01-21 22:53:25 -08:00
Vikhyath Mondreti
45371e521e v0.5.66: external http requests fix, ring highlighting 2026-01-21 02:55:39 -08:00
Waleed
0ce0f98aa5 v0.5.65: gemini updates, textract integration, ui updates (#2909)
* fix(google): wrap primitive tool responses for Gemini API compatibility (#2900)

* fix(canonical): copilot path + update parent (#2901)

* fix(rss): add top-level title, link, pubDate fields to RSS trigger output (#2902)

* fix(rss): add top-level title, link, pubDate fields to RSS trigger output

* fix(imap): add top-level fields to IMAP trigger output

* improvement(browseruse): add profile id param (#2903)

* improvement(browseruse): add profile id param

* make request a stub since we have directExec

* improvement(executor): upgraded abort controller to handle aborts for loops and parallels (#2880)

* improvement(executor): upgraded abort controller to handle aborts for loops and parallels

* comments

* improvement(files): update execution for passing base64 strings (#2906)

* progress

* improvement(execution): update execution for passing base64 strings

* fix types

* cleanup comments

* path security vuln

* reject promise correctly

* fix redirect case

* remove proxy routes

* fix tests

* use ipaddr

* feat(tools): added textract, added v2 for mistral, updated tag dropdown (#2904)

* feat(tools): added textract

* cleanup

* ack pr comments

* reorder

* removed upload for textract async version

* fix additional fields dropdown in editor, update parser to leave validation to be done on the server

* added mistral v2, files v2, and finalized textract

* updated the rest of the old file patterns, updated mistral outputs for v2

* updated tag dropdown to parse non-operation fields as well

* updated extension finder

* cleanup

* added description for inputs to workflow

* use helper for internal route check

* fix tag dropdown merge conflict change

* remove duplicate code

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>

* fix(ui): change add inputs button to match output selector (#2907)

* fix(canvas): removed invite to workspace from canvas popover (#2908)

* fix(canvas): removed invite to workspace

* removed unused props

* fix(copilot): legacy tool display names (#2911)

* fix(a2a): canonical merge  (#2912)

* fix canonical merge

* fix empty array case

* fix(change-detection): copilot diffs have extra field (#2913)

* improvement(logs): improved logs ui bugs, added subflow disable UI (#2910)

* improvement(logs): improved logs ui bugs, added subflow disable UI

* added duplicate to action bar for subflows

* feat(broadcast): email v0.5 (#2905)

---------

Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com>
2026-01-20 23:54:55 -08:00
Waleed
dff1c9d083 v0.5.64: unsubscribe, search improvements, metrics, additional SSO configuration 2026-01-20 00:34:11 -08:00
Vikhyath Mondreti
b09f683072 v0.5.63: ui and performance improvements, more google tools 2026-01-18 15:22:42 -08:00
Vikhyath Mondreti
a8bb0db660 v0.5.62: webhook bug fixes, seeding default subblock values, block selection fixes 2026-01-16 20:27:06 -08:00
Waleed
af82820a28 v0.5.61: webhook improvements, workflow controls, react query for deployment status, chat fixes, reducto and pulse OCR, linear fixes 2026-01-16 18:06:23 -08:00
Waleed
4372841797 v0.5.60: invitation flow improvements, chat fixes, a2a improvements, additional copilot actions 2026-01-15 00:02:18 -08:00
Waleed
5e8c843241 v0.5.59: a2a support, documentation 2026-01-13 13:21:21 -08:00
Waleed
7bf3d73ee6 v0.5.58: export folders, new tools, permissions groups enhancements 2026-01-13 00:56:59 -08:00
Vikhyath Mondreti
7ffc11a738 v0.5.57: subagents, context menu improvements, bug fixes 2026-01-11 11:38:40 -08:00
Waleed
be578e2ed7 v0.5.56: batch operations, access control and permission groups, billing fixes 2026-01-10 00:31:34 -08:00
Waleed
f415e5edc4 v0.5.55: polling groups, bedrock provider, devcontainer fixes, workflow preview enhancements 2026-01-08 23:36:56 -08:00
Waleed
13a6e6c3fa v0.5.54: seo, model blacklist, helm chart updates, fireflies integration, autoconnect improvements, billing fixes 2026-01-07 16:09:45 -08:00
Waleed
f5ab7f21ae v0.5.53: hotkey improvements, added redis fallback, fixes for workflow tool 2026-01-06 23:34:52 -08:00
Waleed
bfb6fffe38 v0.5.52: new port-based router block, combobox expression and variable support 2026-01-06 16:14:10 -08:00
Waleed
4fbec0a43f v0.5.51: triggers, kb, condition block improvements, supabase and grain integration updates 2026-01-06 14:26:46 -08:00
Waleed
585f5e365b v0.5.50: import improvements, ui upgrades, kb styling and performance improvements 2026-01-05 00:35:55 -08:00
Waleed
3792bdd252 v0.5.49: hitl improvements, new email styles, imap trigger, logs context menu (#2672)
* feat(logs-context-menu): consolidated logs utils and types, added logs record context menu (#2659)

* feat(email): welcome email; improvement(emails): ui/ux (#2658)

* feat(email): welcome email; improvement(emails): ui/ux

* improvement(emails): links, accounts, preview

* refactor(emails): file structure and wrapper components

* added envvar for personal emails sent, added isHosted gate

* fixed failing tests, added env mock

* fix: removed comment

---------

Co-authored-by: waleed <walif6@gmail.com>

* fix(logging): hitl + trigger dev crash protection (#2664)

* hitl gaps

* deal with trigger worker crashes

* cleanup import strcuture

* feat(imap): added support for imap trigger (#2663)

* feat(tools): added support for imap trigger

* feat(imap): added parity, tested

* ack PR comments

* final cleanup

* feat(i18n): update translations (#2665)

Co-authored-by: waleedlatif1 <waleedlatif1@users.noreply.github.com>

* fix(grain): updated grain trigger to auto-establish trigger (#2666)

Co-authored-by: aadamgough <adam@sim.ai>

* feat(admin): routes to manage deployments (#2667)

* feat(admin): routes to manage deployments

* fix naming fo deployed by

* feat(time-picker): added timepicker emcn component, added to playground, added searchable prop for dropdown, added more timezones for schedule, updated license and notice date (#2668)

* feat(time-picker): added timepicker emcn component, added to playground, added searchable prop for dropdown, added more timezones for schedule, updated license and notice date

* removed unused params, cleaned up redundant utils

* improvement(invite): aligned styling (#2669)

* improvement(invite): aligned with rest of app

* fix(invite): error handling

* fix: addressed comments

---------

Co-authored-by: Emir Karabeg <78010029+emir-karabeg@users.noreply.github.com>
Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com>
Co-authored-by: waleedlatif1 <waleedlatif1@users.noreply.github.com>
Co-authored-by: Adam Gough <77861281+aadamgough@users.noreply.github.com>
Co-authored-by: aadamgough <adam@sim.ai>
2026-01-03 13:19:18 -08:00
Waleed
eb5d1f3e5b v0.5.48: copy-paste workflow blocks, docs updates, mcp tool fixes 2025-12-31 18:00:04 -08:00
Waleed
54ab82c8dd v0.5.47: deploy workflow as mcp, kb chunks tokenizer, UI improvements, jira service management tools 2025-12-30 23:18:58 -08:00
Waleed
f895bf469b v0.5.46: build improvements, greptile, light mode improvements 2025-12-29 02:17:52 -08:00
Waleed
dd3209af06 v0.5.45: light mode fixes, realtime usage indicator, docker build improvements 2025-12-27 19:57:42 -08:00
Waleed
b6ba3b50a7 v0.5.44: keyboard shortcuts, autolayout, light mode, byok, testing improvements 2025-12-26 21:25:19 -08:00
Waleed
b304233062 v0.5.43: export logs, circleback, grain, vertex, code hygiene, schedule improvements 2025-12-23 19:19:18 -08:00
Vikhyath Mondreti
57e4b49bd6 v0.5.42: fix memory migration 2025-12-23 01:24:54 -08:00
Vikhyath Mondreti
e12dd204ed v0.5.41: memory fixes, copilot improvements, knowledgebase improvements, LLM providers standardization 2025-12-23 00:15:18 -08:00
Vikhyath Mondreti
3d9d9cbc54 v0.5.40: supabase ops to allow non-public schemas, jira uuid 2025-12-21 22:28:05 -08:00
Waleed
0f4ec962ad v0.5.39: notion, workflow variables fixes 2025-12-20 20:44:00 -08:00
Waleed
4827866f9a v0.5.38: snap to grid, copilot ux improvements, billing line items 2025-12-20 17:24:38 -08:00
Waleed
3e697d9ed9 v0.5.37: redaction utils consolidation, logs updates, autoconnect improvements, additional kb tag types 2025-12-19 22:31:55 -08:00
Martin Yankov
4431a1a484 fix(helm): add custom egress rules to realtime network policy (#2481)
The realtime service network policy was missing the custom egress rules section
that allows configuration of additional egress rules via values.yaml. This caused
the realtime pods to be unable to connect to external databases (e.g., PostgreSQL
on port 5432) when using external database configurations.

The app network policy already had this section, but the realtime network policy
was missing it, creating an inconsistency and preventing the realtime service
from accessing external databases configured via networkPolicy.egress values.

This fix adds the same custom egress rules template section to the realtime
network policy, matching the app network policy behavior and allowing users to
configure database connectivity via values.yaml.
2025-12-19 18:59:08 -08:00
Waleed
4d1a9a3f22 v0.5.36: hitl improvements, opengraph, slack fixes, one-click unsubscribe, auth checks, new db indexes 2025-12-19 01:27:49 -08:00
Vikhyath Mondreti
eb07a080fb v0.5.35: helm updates, copilot improvements, 404 for docs, salesforce fixes, subflow resize clamping 2025-12-18 16:23:19 -08:00
17 changed files with 237 additions and 1894 deletions

View File

@@ -7,24 +7,13 @@ import {
useRef, useRef,
useState, useState,
} from 'react' } from 'react'
import { createLogger } from '@sim/logger'
import { isEqual } from 'lodash' import { isEqual } from 'lodash'
import { ArrowLeftRight, ChevronDown, ChevronsUpDown, ChevronUp, Plus } from 'lucide-react' import { ChevronDown, ChevronsUpDown, ChevronUp, Plus } from 'lucide-react'
import { useParams } from 'next/navigation' import { Button, Popover, PopoverContent, PopoverItem, PopoverTrigger } from '@/components/emcn'
import {
Button,
Popover,
PopoverContent,
PopoverItem,
PopoverTrigger,
Tooltip,
} from '@/components/emcn'
import { Trash } from '@/components/emcn/icons/trash' import { Trash } from '@/components/emcn/icons/trash'
import { cn } from '@/lib/core/utils/cn' import { cn } from '@/lib/core/utils/cn'
import { EnvVarDropdown } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/env-var-dropdown' import { EnvVarDropdown } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/env-var-dropdown'
import { FileUpload } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/file-upload/file-upload'
import { formatDisplayText } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/formatted-text' import { formatDisplayText } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/formatted-text'
import { ShortInput } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/short-input/short-input'
import { TagDropdown } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/tag-dropdown/tag-dropdown' import { TagDropdown } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/tag-dropdown/tag-dropdown'
import { useSubBlockInput } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/hooks/use-sub-block-input' import { useSubBlockInput } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/hooks/use-sub-block-input'
import { useSubBlockValue } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/hooks/use-sub-block-value' import { useSubBlockValue } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/hooks/use-sub-block-value'
@@ -32,32 +21,19 @@ import type { WandControlHandlers } from '@/app/workspace/[workspaceId]/w/[workf
import { useAccessibleReferencePrefixes } from '@/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-accessible-reference-prefixes' import { useAccessibleReferencePrefixes } from '@/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-accessible-reference-prefixes'
import { useWand } from '@/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-wand' import { useWand } from '@/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-wand'
import type { SubBlockConfig } from '@/blocks/types' import type { SubBlockConfig } from '@/blocks/types'
import { supportsVision } from '@/providers/utils'
import { useWorkflowRegistry } from '@/stores/workflows/registry/store'
import { useSubBlockStore } from '@/stores/workflows/subblock/store'
const logger = createLogger('MessagesInput')
const MIN_TEXTAREA_HEIGHT_PX = 80 const MIN_TEXTAREA_HEIGHT_PX = 80
/** Workspace file record from API */
interface WorkspaceFile {
id: string
name: string
path: string
type: string
}
const MAX_TEXTAREA_HEIGHT_PX = 320 const MAX_TEXTAREA_HEIGHT_PX = 320
/** Pattern to match complete message objects in JSON */ /** Pattern to match complete message objects in JSON */
const COMPLETE_MESSAGE_PATTERN = const COMPLETE_MESSAGE_PATTERN =
/"role"\s*:\s*"(system|user|assistant|attachment)"[^}]*"content"\s*:\s*"((?:[^"\\]|\\.)*)"/g /"role"\s*:\s*"(system|user|assistant)"[^}]*"content"\s*:\s*"((?:[^"\\]|\\.)*)"/g
/** Pattern to match incomplete content at end of buffer */ /** Pattern to match incomplete content at end of buffer */
const INCOMPLETE_CONTENT_PATTERN = /"content"\s*:\s*"((?:[^"\\]|\\.)*)$/ const INCOMPLETE_CONTENT_PATTERN = /"content"\s*:\s*"((?:[^"\\]|\\.)*)$/
/** Pattern to match role before content */ /** Pattern to match role before content */
const ROLE_BEFORE_CONTENT_PATTERN = /"role"\s*:\s*"(system|user|assistant|attachment)"[^{]*$/ const ROLE_BEFORE_CONTENT_PATTERN = /"role"\s*:\s*"(system|user|assistant)"[^{]*$/
/** /**
* Unescapes JSON string content * Unescapes JSON string content
@@ -65,46 +41,41 @@ const ROLE_BEFORE_CONTENT_PATTERN = /"role"\s*:\s*"(system|user|assistant|attach
const unescapeContent = (str: string): string => const unescapeContent = (str: string): string =>
str.replace(/\\n/g, '\n').replace(/\\"/g, '"').replace(/\\\\/g, '\\') str.replace(/\\n/g, '\n').replace(/\\"/g, '"').replace(/\\\\/g, '\\')
/**
* Attachment content (files, images, documents)
*/
interface AttachmentContent {
/** Source type: how the data was provided */
sourceType: 'url' | 'base64' | 'file'
/** The URL or base64 data */
data: string
/** MIME type (e.g., 'image/png', 'application/pdf', 'audio/mp3') */
mimeType?: string
/** Optional filename for file uploads */
fileName?: string
/** Optional workspace file ID (used by wand to select existing files) */
fileId?: string
}
/** /**
* Interface for individual message in the messages array * Interface for individual message in the messages array
*/ */
interface Message { interface Message {
role: 'system' | 'user' | 'assistant' | 'attachment' role: 'system' | 'user' | 'assistant'
content: string content: string
attachment?: AttachmentContent
} }
/** /**
* Props for the MessagesInput component * Props for the MessagesInput component
*/ */
interface MessagesInputProps { interface MessagesInputProps {
/** Unique identifier for the block */
blockId: string blockId: string
/** Unique identifier for the sub-block */
subBlockId: string subBlockId: string
/** Configuration object for the sub-block */
config: SubBlockConfig config: SubBlockConfig
/** Whether component is in preview mode */
isPreview?: boolean isPreview?: boolean
/** Value to display in preview mode */
previewValue?: Message[] | null previewValue?: Message[] | null
/** Whether the input is disabled */
disabled?: boolean disabled?: boolean
/** Ref to expose wand control handlers to parent */
wandControlRef?: React.MutableRefObject<WandControlHandlers | null> wandControlRef?: React.MutableRefObject<WandControlHandlers | null>
} }
/** /**
* MessagesInput component for managing LLM message history * MessagesInput component for managing LLM message history
*
* @remarks
* - Manages an array of messages with role and content
* - Each message can be edited, removed, or reordered
* - Stores data in LLM-compatible format: [{ role, content }]
*/ */
export function MessagesInput({ export function MessagesInput({
blockId, blockId,
@@ -115,163 +86,10 @@ export function MessagesInput({
disabled = false, disabled = false,
wandControlRef, wandControlRef,
}: MessagesInputProps) { }: MessagesInputProps) {
const params = useParams()
const workspaceId = params?.workspaceId as string
const [messages, setMessages] = useSubBlockValue<Message[]>(blockId, subBlockId, false) const [messages, setMessages] = useSubBlockValue<Message[]>(blockId, subBlockId, false)
const [localMessages, setLocalMessages] = useState<Message[]>([{ role: 'user', content: '' }]) const [localMessages, setLocalMessages] = useState<Message[]>([{ role: 'user', content: '' }])
const accessiblePrefixes = useAccessibleReferencePrefixes(blockId) const accessiblePrefixes = useAccessibleReferencePrefixes(blockId)
const [openPopoverIndex, setOpenPopoverIndex] = useState<number | null>(null) const [openPopoverIndex, setOpenPopoverIndex] = useState<number | null>(null)
const { activeWorkflowId } = useWorkflowRegistry()
// Local attachment mode state - basic = FileUpload, advanced = URL/base64 textarea
const [attachmentMode, setAttachmentMode] = useState<'basic' | 'advanced'>('basic')
// Workspace files for wand context
const [workspaceFiles, setWorkspaceFiles] = useState<WorkspaceFile[]>([])
// Fetch workspace files for wand context
const loadWorkspaceFiles = useCallback(async () => {
if (!workspaceId || isPreview) return
try {
const response = await fetch(`/api/workspaces/${workspaceId}/files`)
const data = await response.json()
if (data.success) {
setWorkspaceFiles(data.files || [])
}
} catch (error) {
logger.error('Error loading workspace files:', error)
}
}, [workspaceId, isPreview])
// Load workspace files on mount
useEffect(() => {
void loadWorkspaceFiles()
}, [loadWorkspaceFiles])
// Build sources string for wand - available workspace files
const sourcesInfo = useMemo(() => {
if (workspaceFiles.length === 0) {
return 'No workspace files available. The user can upload files manually after generation.'
}
const filesList = workspaceFiles
.filter(
(f) =>
f.type.startsWith('image/') ||
f.type.startsWith('audio/') ||
f.type.startsWith('video/') ||
f.type === 'application/pdf'
)
.map((f) => ` - id: "${f.id}", name: "${f.name}", type: "${f.type}"`)
.join('\n')
if (!filesList) {
return 'No files in workspace. The user can upload files manually after generation.'
}
return `AVAILABLE WORKSPACE FILES (optional - you don't have to select one):\n${filesList}\n\nTo use a file, include "fileId": "<id>" in the attachment object. If not selecting a file, omit the fileId field.`
}, [workspaceFiles])
// Get indices of attachment messages for subscription
const attachmentIndices = useMemo(
() =>
localMessages
.map((msg, index) => (msg.role === 'attachment' ? index : -1))
.filter((i) => i !== -1),
[localMessages]
)
// Subscribe to model value to check vision capability
const modelSupportsVision = useSubBlockStore(
useCallback(
(state) => {
if (!activeWorkflowId) return true // Default to allowing attachments
const blockValues = state.workflowValues[activeWorkflowId]?.[blockId] ?? {}
const modelValue = blockValues.model as string | undefined
if (!modelValue) return true // No model selected, allow attachments
return supportsVision(modelValue)
},
[activeWorkflowId, blockId]
)
)
// Determine available roles based on model capabilities
const availableRoles = useMemo(() => {
const baseRoles: Array<'system' | 'user' | 'assistant' | 'attachment'> = [
'system',
'user',
'assistant',
]
if (modelSupportsVision) {
baseRoles.push('attachment')
}
return baseRoles
}, [modelSupportsVision])
// Subscribe to file upload values for all attachment messages
const fileUploadValues = useSubBlockStore(
useCallback(
(state) => {
if (!activeWorkflowId) return {}
const blockValues = state.workflowValues[activeWorkflowId]?.[blockId] ?? {}
const result: Record<number, { name: string; path: string; type: string; size: number }> =
{}
for (const index of attachmentIndices) {
const fileUploadKey = `${subBlockId}-attachment-${index}`
const fileValue = blockValues[fileUploadKey]
if (fileValue && typeof fileValue === 'object' && 'path' in fileValue) {
result[index] = fileValue as { name: string; path: string; type: string; size: number }
}
}
return result
},
[activeWorkflowId, blockId, subBlockId, attachmentIndices]
)
)
// Effect to sync FileUpload values to message attachment objects
useEffect(() => {
if (!activeWorkflowId || isPreview) return
let hasChanges = false
const updatedMessages = localMessages.map((msg, index) => {
if (msg.role !== 'attachment') return msg
const uploadedFile = fileUploadValues[index]
if (uploadedFile) {
const newAttachment: AttachmentContent = {
sourceType: 'file',
data: uploadedFile.path,
mimeType: uploadedFile.type,
fileName: uploadedFile.name,
}
// Only update if different
if (
msg.attachment?.data !== newAttachment.data ||
msg.attachment?.sourceType !== newAttachment.sourceType ||
msg.attachment?.mimeType !== newAttachment.mimeType ||
msg.attachment?.fileName !== newAttachment.fileName
) {
hasChanges = true
return {
...msg,
content: uploadedFile.name || msg.content,
attachment: newAttachment,
}
}
}
return msg
})
if (hasChanges) {
setLocalMessages(updatedMessages)
setMessages(updatedMessages)
}
}, [activeWorkflowId, localMessages, isPreview, setMessages, fileUploadValues])
const subBlockInput = useSubBlockInput({ const subBlockInput = useSubBlockInput({
blockId, blockId,
subBlockId, subBlockId,
@@ -280,40 +98,43 @@ export function MessagesInput({
disabled, disabled,
}) })
/**
* Gets the current messages as JSON string for wand context
*/
const getMessagesJson = useCallback((): string => { const getMessagesJson = useCallback((): string => {
if (localMessages.length === 0) return '' if (localMessages.length === 0) return ''
// Filter out empty messages for cleaner context
const nonEmptyMessages = localMessages.filter((m) => m.content.trim() !== '') const nonEmptyMessages = localMessages.filter((m) => m.content.trim() !== '')
if (nonEmptyMessages.length === 0) return '' if (nonEmptyMessages.length === 0) return ''
return JSON.stringify(nonEmptyMessages, null, 2) return JSON.stringify(nonEmptyMessages, null, 2)
}, [localMessages]) }, [localMessages])
/**
* Streaming buffer for accumulating JSON content
*/
const streamBufferRef = useRef<string>('') const streamBufferRef = useRef<string>('')
/**
* Parses and validates messages from JSON content
*/
const parseMessages = useCallback((content: string): Message[] | null => { const parseMessages = useCallback((content: string): Message[] | null => {
try { try {
const parsed = JSON.parse(content) const parsed = JSON.parse(content)
if (Array.isArray(parsed)) { if (Array.isArray(parsed)) {
const validMessages: Message[] = parsed const validMessages: Message[] = parsed
.filter( .filter(
(m): m is { role: string; content: string; attachment?: AttachmentContent } => (m): m is { role: string; content: string } =>
typeof m === 'object' && typeof m === 'object' &&
m !== null && m !== null &&
typeof m.role === 'string' && typeof m.role === 'string' &&
typeof m.content === 'string' typeof m.content === 'string'
) )
.map((m) => { .map((m) => ({
const role = ['system', 'user', 'assistant', 'attachment'].includes(m.role) role: (['system', 'user', 'assistant'].includes(m.role)
? m.role ? m.role
: 'user' : 'user') as Message['role'],
const message: Message = { content: m.content,
role: role as Message['role'], }))
content: m.content,
}
if (m.attachment) {
message.attachment = m.attachment
}
return message
})
return validMessages.length > 0 ? validMessages : null return validMessages.length > 0 ? validMessages : null
} }
} catch { } catch {
@@ -322,19 +143,26 @@ export function MessagesInput({
return null return null
}, []) }, [])
/**
* Extracts messages from streaming JSON buffer
* Uses simple pattern matching for efficiency
*/
const extractStreamingMessages = useCallback( const extractStreamingMessages = useCallback(
(buffer: string): Message[] => { (buffer: string): Message[] => {
// Try complete JSON parse first
const complete = parseMessages(buffer) const complete = parseMessages(buffer)
if (complete) return complete if (complete) return complete
const result: Message[] = [] const result: Message[] = []
// Reset regex lastIndex for global pattern
COMPLETE_MESSAGE_PATTERN.lastIndex = 0 COMPLETE_MESSAGE_PATTERN.lastIndex = 0
let match let match
while ((match = COMPLETE_MESSAGE_PATTERN.exec(buffer)) !== null) { while ((match = COMPLETE_MESSAGE_PATTERN.exec(buffer)) !== null) {
result.push({ role: match[1] as Message['role'], content: unescapeContent(match[2]) }) result.push({ role: match[1] as Message['role'], content: unescapeContent(match[2]) })
} }
// Check for incomplete message at end (content still streaming)
const lastContentIdx = buffer.lastIndexOf('"content"') const lastContentIdx = buffer.lastIndexOf('"content"')
if (lastContentIdx !== -1) { if (lastContentIdx !== -1) {
const tail = buffer.slice(lastContentIdx) const tail = buffer.slice(lastContentIdx)
@@ -344,6 +172,7 @@ export function MessagesInput({
const roleMatch = head.match(ROLE_BEFORE_CONTENT_PATTERN) const roleMatch = head.match(ROLE_BEFORE_CONTENT_PATTERN)
if (roleMatch) { if (roleMatch) {
const content = unescapeContent(incomplete[1]) const content = unescapeContent(incomplete[1])
// Only add if not duplicate of last complete message
if (result.length === 0 || result[result.length - 1].content !== content) { if (result.length === 0 || result[result.length - 1].content !== content) {
result.push({ role: roleMatch[1] as Message['role'], content }) result.push({ role: roleMatch[1] as Message['role'], content })
} }
@@ -356,10 +185,12 @@ export function MessagesInput({
[parseMessages] [parseMessages]
) )
/**
* Wand hook for AI-assisted content generation
*/
const wandHook = useWand({ const wandHook = useWand({
wandConfig: config.wandConfig, wandConfig: config.wandConfig,
currentValue: getMessagesJson(), currentValue: getMessagesJson(),
sources: sourcesInfo,
onStreamStart: () => { onStreamStart: () => {
streamBufferRef.current = '' streamBufferRef.current = ''
setLocalMessages([{ role: 'system', content: '' }]) setLocalMessages([{ role: 'system', content: '' }])
@@ -374,50 +205,10 @@ export function MessagesInput({
onGeneratedContent: (content) => { onGeneratedContent: (content) => {
const validMessages = parseMessages(content) const validMessages = parseMessages(content)
if (validMessages) { if (validMessages) {
// Process attachment messages - only allow fileId to set files, sanitize other attempts
validMessages.forEach((msg, index) => {
if (msg.role === 'attachment') {
// Check if this is an existing file with valid data (preserve it)
const hasExistingFile =
msg.attachment?.sourceType === 'file' &&
msg.attachment?.data?.startsWith('/api/') &&
msg.attachment?.fileName
if (hasExistingFile) {
// Preserve existing file data as-is
return
}
// Check if wand provided a fileId to select a workspace file
if (msg.attachment?.fileId) {
const file = workspaceFiles.find((f) => f.id === msg.attachment?.fileId)
if (file) {
// Set the file value in SubBlockStore so FileUpload picks it up
const fileUploadKey = `${subBlockId}-attachment-${index}`
const uploadedFile = {
name: file.name,
path: file.path,
type: file.type,
size: 0, // Size not available from workspace files list
}
useSubBlockStore.getState().setValue(blockId, fileUploadKey, uploadedFile)
// Clear the attachment object - the FileUpload will sync the file data via useEffect
// DON'T set attachment.data here as it would appear in the ShortInput (advanced mode)
msg.attachment = undefined
return
}
}
// Sanitize: clear any attachment object that isn't a valid existing file or fileId match
// This prevents the LLM from setting arbitrary data/variable references
msg.attachment = undefined
}
})
setLocalMessages(validMessages) setLocalMessages(validMessages)
setMessages(validMessages) setMessages(validMessages)
} else { } else {
// Fallback: treat as raw system prompt
const trimmed = content.trim() const trimmed = content.trim()
if (trimmed) { if (trimmed) {
const fallback: Message[] = [{ role: 'system', content: trimmed }] const fallback: Message[] = [{ role: 'system', content: trimmed }]
@@ -428,6 +219,9 @@ export function MessagesInput({
}, },
}) })
/**
* Expose wand control handlers to parent via ref
*/
useImperativeHandle( useImperativeHandle(
wandControlRef, wandControlRef,
() => ({ () => ({
@@ -455,6 +249,9 @@ export function MessagesInput({
} }
}, [isPreview, previewValue, messages]) }, [isPreview, previewValue, messages])
/**
* Gets the current messages array
*/
const currentMessages = useMemo<Message[]>(() => { const currentMessages = useMemo<Message[]>(() => {
if (isPreview && previewValue && Array.isArray(previewValue)) { if (isPreview && previewValue && Array.isArray(previewValue)) {
return previewValue return previewValue
@@ -472,6 +269,9 @@ export function MessagesInput({
startHeight: number startHeight: number
} | null>(null) } | null>(null)
/**
* Updates a specific message's content
*/
const updateMessageContent = useCallback( const updateMessageContent = useCallback(
(index: number, content: string) => { (index: number, content: string) => {
if (isPreview || disabled) return if (isPreview || disabled) return
@@ -487,27 +287,17 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Updates a specific message's role
*/
const updateMessageRole = useCallback( const updateMessageRole = useCallback(
(index: number, role: 'system' | 'user' | 'assistant' | 'attachment') => { (index: number, role: 'system' | 'user' | 'assistant') => {
if (isPreview || disabled) return if (isPreview || disabled) return
const updatedMessages = [...localMessages] const updatedMessages = [...localMessages]
if (role === 'attachment') { updatedMessages[index] = {
updatedMessages[index] = { ...updatedMessages[index],
...updatedMessages[index], role,
role,
content: updatedMessages[index].content || '',
attachment: updatedMessages[index].attachment || {
sourceType: 'file',
data: '',
},
}
} else {
const { attachment: _, ...rest } = updatedMessages[index]
updatedMessages[index] = {
...rest,
role,
}
} }
setLocalMessages(updatedMessages) setLocalMessages(updatedMessages)
setMessages(updatedMessages) setMessages(updatedMessages)
@@ -515,6 +305,9 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Adds a message after the specified index
*/
const addMessageAfter = useCallback( const addMessageAfter = useCallback(
(index: number) => { (index: number) => {
if (isPreview || disabled) return if (isPreview || disabled) return
@@ -527,6 +320,9 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Deletes a message at the specified index
*/
const deleteMessage = useCallback( const deleteMessage = useCallback(
(index: number) => { (index: number) => {
if (isPreview || disabled) return if (isPreview || disabled) return
@@ -539,6 +335,9 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Moves a message up in the list
*/
const moveMessageUp = useCallback( const moveMessageUp = useCallback(
(index: number) => { (index: number) => {
if (isPreview || disabled || index === 0) return if (isPreview || disabled || index === 0) return
@@ -553,6 +352,9 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Moves a message down in the list
*/
const moveMessageDown = useCallback( const moveMessageDown = useCallback(
(index: number) => { (index: number) => {
if (isPreview || disabled || index === localMessages.length - 1) return if (isPreview || disabled || index === localMessages.length - 1) return
@@ -567,11 +369,18 @@ export function MessagesInput({
[localMessages, setMessages, isPreview, disabled] [localMessages, setMessages, isPreview, disabled]
) )
/**
* Capitalizes the first letter of the role
*/
const formatRole = (role: string): string => { const formatRole = (role: string): string => {
return role.charAt(0).toUpperCase() + role.slice(1) return role.charAt(0).toUpperCase() + role.slice(1)
} }
/**
* Handles header click to focus the textarea
*/
const handleHeaderClick = useCallback((index: number, e: React.MouseEvent) => { const handleHeaderClick = useCallback((index: number, e: React.MouseEvent) => {
// Don't focus if clicking on interactive elements
const target = e.target as HTMLElement const target = e.target as HTMLElement
if (target.closest('button') || target.closest('[data-radix-popper-content-wrapper]')) { if (target.closest('button') || target.closest('[data-radix-popper-content-wrapper]')) {
return return
@@ -761,52 +570,50 @@ export function MessagesInput({
className='flex cursor-pointer items-center justify-between px-[8px] pt-[6px]' className='flex cursor-pointer items-center justify-between px-[8px] pt-[6px]'
onClick={(e) => handleHeaderClick(index, e)} onClick={(e) => handleHeaderClick(index, e)}
> >
<div className='flex items-center'> <Popover
<Popover open={openPopoverIndex === index}
open={openPopoverIndex === index} onOpenChange={(open) => setOpenPopoverIndex(open ? index : null)}
onOpenChange={(open) => setOpenPopoverIndex(open ? index : null)} >
> <PopoverTrigger asChild>
<PopoverTrigger asChild> <button
<button type='button'
type='button' disabled={isPreview || disabled}
disabled={isPreview || disabled} className={cn(
className={cn( 'group -ml-1.5 -my-1 flex items-center gap-1 rounded px-1.5 py-1 font-medium text-[13px] text-[var(--text-primary)] leading-none transition-colors hover:bg-[var(--surface-5)] hover:text-[var(--text-secondary)]',
'group -ml-1.5 -my-1 flex items-center gap-1 rounded px-1.5 py-1 font-medium text-[13px] text-[var(--text-primary)] leading-none transition-colors hover:bg-[var(--surface-5)] hover:text-[var(--text-secondary)]', (isPreview || disabled) &&
(isPreview || disabled) && 'cursor-default hover:bg-transparent hover:text-[var(--text-primary)]'
'cursor-default hover:bg-transparent hover:text-[var(--text-primary)]' )}
)} onClick={(e) => e.stopPropagation()}
onClick={(e) => e.stopPropagation()} aria-label='Select message role'
aria-label='Select message role' >
> {formatRole(message.role)}
{formatRole(message.role)} {!isPreview && !disabled && (
{!isPreview && !disabled && ( <ChevronDown
<ChevronDown className={cn(
className={cn( 'h-3 w-3 flex-shrink-0 transition-transform duration-100',
'h-3 w-3 flex-shrink-0 transition-transform duration-100', openPopoverIndex === index && 'rotate-180'
openPopoverIndex === index && 'rotate-180' )}
)} />
/> )}
)} </button>
</button> </PopoverTrigger>
</PopoverTrigger> <PopoverContent minWidth={140} align='start'>
<PopoverContent minWidth={140} align='start'> <div className='flex flex-col gap-[2px]'>
<div className='flex flex-col gap-[2px]'> {(['system', 'user', 'assistant'] as const).map((role) => (
{availableRoles.map((role) => ( <PopoverItem
<PopoverItem key={role}
key={role} active={message.role === role}
active={message.role === role} onClick={() => {
onClick={() => { updateMessageRole(index, role)
updateMessageRole(index, role) setOpenPopoverIndex(null)
setOpenPopoverIndex(null) }}
}} >
> <span>{formatRole(role)}</span>
<span>{formatRole(role)}</span> </PopoverItem>
</PopoverItem> ))}
))} </div>
</div> </PopoverContent>
</PopoverContent> </Popover>
</Popover>
</div>
{!isPreview && !disabled && ( {!isPreview && !disabled && (
<div className='flex items-center'> <div className='flex items-center'>
@@ -850,43 +657,6 @@ export function MessagesInput({
</Button> </Button>
</> </>
)} )}
{/* Mode toggle for attachment messages */}
{message.role === 'attachment' && (
<Tooltip.Root>
<Tooltip.Trigger asChild>
<Button
variant='ghost'
onClick={(e: React.MouseEvent) => {
e.stopPropagation()
setAttachmentMode((m) => (m === 'basic' ? 'advanced' : 'basic'))
}}
disabled={disabled}
className='-my-1 -mr-1 h-6 w-6 p-0'
aria-label={
attachmentMode === 'advanced'
? 'Switch to file upload'
: 'Switch to URL/text input'
}
>
<ArrowLeftRight
className={cn(
'h-3 w-3',
attachmentMode === 'advanced'
? 'text-[var(--text-primary)]'
: 'text-[var(--text-secondary)]'
)}
/>
</Button>
</Tooltip.Trigger>
<Tooltip.Content side='top'>
<p>
{attachmentMode === 'advanced'
? 'Switch to file upload'
: 'Switch to URL/text input'}
</p>
</Tooltip.Content>
</Tooltip.Root>
)}
<Button <Button
variant='ghost' variant='ghost'
onClick={(e: React.MouseEvent) => { onClick={(e: React.MouseEvent) => {
@@ -903,152 +673,98 @@ export function MessagesInput({
)} )}
</div> </div>
{/* Content Input - different for attachment vs text messages */} {/* Content Input with overlay for variable highlighting */}
{message.role === 'attachment' ? ( <div className='relative w-full overflow-hidden'>
<div className='relative w-full px-[8px] py-[8px]'> <textarea
{attachmentMode === 'basic' ? ( ref={(el) => {
<FileUpload textareaRefs.current[fieldId] = el
blockId={blockId} }}
subBlockId={`${subBlockId}-attachment-${index}`} className='relative z-[2] m-0 box-border h-auto min-h-[80px] w-full resize-none overflow-y-auto overflow-x-hidden whitespace-pre-wrap break-words border-none bg-transparent px-[8px] py-[8px] font-medium font-sans text-sm text-transparent leading-[1.5] caret-[var(--text-primary)] outline-none [-ms-overflow-style:none] [scrollbar-width:none] placeholder:text-[var(--text-muted)] focus:outline-none focus-visible:outline-none disabled:cursor-not-allowed [&::-webkit-scrollbar]:hidden'
acceptedTypes='image/*,audio/*,video/*,application/pdf,.doc,.docx,.txt' placeholder='Enter message content...'
multiple={false} value={message.content}
isPreview={isPreview} onChange={fieldHandlers.onChange}
disabled={disabled} onKeyDown={(e) => {
/> if (e.key === 'Tab' && !isPreview && !disabled) {
) : ( e.preventDefault()
<ShortInput const direction = e.shiftKey ? -1 : 1
blockId={blockId} const nextIndex = index + direction
subBlockId={`${subBlockId}-attachment-ref-${index}`}
placeholder='Reference file from previous block...' if (nextIndex >= 0 && nextIndex < currentMessages.length) {
config={{ const nextFieldId = `message-${nextIndex}`
id: `${subBlockId}-attachment-ref-${index}`, const nextTextarea = textareaRefs.current[nextFieldId]
type: 'short-input', if (nextTextarea) {
}} nextTextarea.focus()
value={ nextTextarea.selectionStart = nextTextarea.value.length
// Only show value for variable references, not file uploads nextTextarea.selectionEnd = nextTextarea.value.length
message.attachment?.sourceType === 'file'
? ''
: message.attachment?.data || ''
}
onChange={(newValue: string) => {
const updatedMessages = [...localMessages]
if (updatedMessages[index].role === 'attachment') {
// Determine sourceType based on content
let sourceType: 'url' | 'base64' = 'url'
if (newValue.startsWith('data:') || newValue.includes(';base64,')) {
sourceType = 'base64'
}
updatedMessages[index] = {
...updatedMessages[index],
content: newValue.substring(0, 50),
attachment: {
...updatedMessages[index].attachment,
sourceType,
data: newValue,
},
}
setLocalMessages(updatedMessages)
setMessages(updatedMessages)
} }
}} }
isPreview={isPreview} return
disabled={disabled} }
/>
)} fieldHandlers.onKeyDown(e)
}}
onDrop={fieldHandlers.onDrop}
onDragOver={fieldHandlers.onDragOver}
onFocus={fieldHandlers.onFocus}
onScroll={(e) => {
const overlay = overlayRefs.current[fieldId]
if (overlay) {
overlay.scrollTop = e.currentTarget.scrollTop
overlay.scrollLeft = e.currentTarget.scrollLeft
}
}}
disabled={isPreview || disabled}
/>
<div
ref={(el) => {
overlayRefs.current[fieldId] = el
}}
className='pointer-events-none absolute top-0 left-0 z-[1] m-0 box-border w-full overflow-y-auto overflow-x-hidden whitespace-pre-wrap break-words border-none bg-transparent px-[8px] py-[8px] font-medium font-sans text-[var(--text-primary)] text-sm leading-[1.5] [-ms-overflow-style:none] [scrollbar-width:none] [&::-webkit-scrollbar]:hidden'
>
{formatDisplayText(message.content, {
accessiblePrefixes,
highlightAll: !accessiblePrefixes,
})}
{message.content.endsWith('\n') && '\u200B'}
</div> </div>
) : (
<div className='relative w-full overflow-hidden'>
<textarea
ref={(el) => {
textareaRefs.current[fieldId] = el
}}
className='relative z-[2] m-0 box-border h-auto min-h-[80px] w-full resize-none overflow-y-auto overflow-x-hidden whitespace-pre-wrap break-words border-none bg-transparent px-[8px] py-[8px] font-medium font-sans text-sm text-transparent leading-[1.5] caret-[var(--text-primary)] outline-none [-ms-overflow-style:none] [scrollbar-width:none] placeholder:text-[var(--text-muted)] focus:outline-none focus-visible:outline-none disabled:cursor-not-allowed [&::-webkit-scrollbar]:hidden'
placeholder='Enter message content...'
value={message.content}
onChange={fieldHandlers.onChange}
onKeyDown={(e) => {
if (e.key === 'Tab' && !isPreview && !disabled) {
e.preventDefault()
const direction = e.shiftKey ? -1 : 1
const nextIndex = index + direction
if (nextIndex >= 0 && nextIndex < currentMessages.length) { {/* Env var dropdown for this message */}
const nextFieldId = `message-${nextIndex}` <EnvVarDropdown
const nextTextarea = textareaRefs.current[nextFieldId] visible={fieldState.showEnvVars && !isPreview && !disabled}
if (nextTextarea) { onSelect={handleEnvSelect}
nextTextarea.focus() searchTerm={fieldState.searchTerm}
nextTextarea.selectionStart = nextTextarea.value.length inputValue={message.content}
nextTextarea.selectionEnd = nextTextarea.value.length cursorPosition={fieldState.cursorPosition}
} onClose={() => subBlockInput.fieldHelpers.hideFieldDropdowns(fieldId)}
} workspaceId={subBlockInput.workspaceId}
return maxHeight='192px'
} inputRef={textareaRefObject}
/>
fieldHandlers.onKeyDown(e) {/* Tag dropdown for this message */}
}} <TagDropdown
onDrop={fieldHandlers.onDrop} visible={fieldState.showTags && !isPreview && !disabled}
onDragOver={fieldHandlers.onDragOver} onSelect={handleTagSelect}
onFocus={fieldHandlers.onFocus} blockId={blockId}
onScroll={(e) => { activeSourceBlockId={fieldState.activeSourceBlockId}
const overlay = overlayRefs.current[fieldId] inputValue={message.content}
if (overlay) { cursorPosition={fieldState.cursorPosition}
overlay.scrollTop = e.currentTarget.scrollTop onClose={() => subBlockInput.fieldHelpers.hideFieldDropdowns(fieldId)}
overlay.scrollLeft = e.currentTarget.scrollLeft inputRef={textareaRefObject}
} />
}}
disabled={isPreview || disabled} {!isPreview && !disabled && (
/>
<div <div
ref={(el) => { className='absolute right-1 bottom-1 z-[3] flex h-4 w-4 cursor-ns-resize items-center justify-center rounded-[4px] border border-[var(--border-1)] bg-[var(--surface-5)] dark:bg-[var(--surface-5)]'
overlayRefs.current[fieldId] = el onMouseDown={(e) => handleResizeStart(fieldId, e)}
onDragStart={(e) => {
e.preventDefault()
}} }}
className='pointer-events-none absolute top-0 left-0 z-[1] m-0 box-border w-full overflow-y-auto overflow-x-hidden whitespace-pre-wrap break-words border-none bg-transparent px-[8px] py-[8px] font-medium font-sans text-[var(--text-primary)] text-sm leading-[1.5] [-ms-overflow-style:none] [scrollbar-width:none] [&::-webkit-scrollbar]:hidden'
> >
{formatDisplayText(message.content, { <ChevronsUpDown className='h-3 w-3 text-[var(--text-muted)]' />
accessiblePrefixes,
highlightAll: !accessiblePrefixes,
})}
{message.content.endsWith('\n') && '\u200B'}
</div> </div>
)}
{/* Env var dropdown for this message */} </div>
<EnvVarDropdown
visible={fieldState.showEnvVars && !isPreview && !disabled}
onSelect={handleEnvSelect}
searchTerm={fieldState.searchTerm}
inputValue={message.content}
cursorPosition={fieldState.cursorPosition}
onClose={() => subBlockInput.fieldHelpers.hideFieldDropdowns(fieldId)}
workspaceId={subBlockInput.workspaceId}
maxHeight='192px'
inputRef={textareaRefObject}
/>
{/* Tag dropdown for this message */}
<TagDropdown
visible={fieldState.showTags && !isPreview && !disabled}
onSelect={handleTagSelect}
blockId={blockId}
activeSourceBlockId={fieldState.activeSourceBlockId}
inputValue={message.content}
cursorPosition={fieldState.cursorPosition}
onClose={() => subBlockInput.fieldHelpers.hideFieldDropdowns(fieldId)}
inputRef={textareaRefObject}
/>
{!isPreview && !disabled && (
<div
className='absolute right-1 bottom-1 z-[3] flex h-4 w-4 cursor-ns-resize items-center justify-center rounded-[4px] border border-[var(--border-1)] bg-[var(--surface-5)] dark:bg-[var(--surface-5)]'
onMouseDown={(e) => handleResizeStart(fieldId, e)}
onDragStart={(e) => {
e.preventDefault()
}}
>
<ChevronsUpDown className='h-3 w-3 text-[var(--text-muted)]' />
</div>
)}
</div>
)}
</> </>
) )
})()} })()}

View File

@@ -63,8 +63,6 @@ export interface WandConfig {
interface UseWandProps { interface UseWandProps {
wandConfig?: WandConfig wandConfig?: WandConfig
currentValue?: string currentValue?: string
/** Additional context about available sources/references for the prompt */
sources?: string
onGeneratedContent: (content: string) => void onGeneratedContent: (content: string) => void
onStreamChunk?: (chunk: string) => void onStreamChunk?: (chunk: string) => void
onStreamStart?: () => void onStreamStart?: () => void
@@ -74,7 +72,6 @@ interface UseWandProps {
export function useWand({ export function useWand({
wandConfig, wandConfig,
currentValue, currentValue,
sources,
onGeneratedContent, onGeneratedContent,
onStreamChunk, onStreamChunk,
onStreamStart, onStreamStart,
@@ -157,12 +154,6 @@ export function useWand({
if (systemPrompt.includes('{context}')) { if (systemPrompt.includes('{context}')) {
systemPrompt = systemPrompt.replace('{context}', contextInfo) systemPrompt = systemPrompt.replace('{context}', contextInfo)
} }
if (systemPrompt.includes('{sources}')) {
systemPrompt = systemPrompt.replace(
'{sources}',
sources || 'No upstream sources available'
)
}
const userMessage = prompt const userMessage = prompt

View File

@@ -85,9 +85,7 @@ export const AgentBlock: BlockConfig<AgentResponse> = {
id: 'messages', id: 'messages',
title: 'Messages', title: 'Messages',
type: 'messages-input', type: 'messages-input',
canonicalParamId: 'messages',
placeholder: 'Enter messages...', placeholder: 'Enter messages...',
mode: 'basic',
wandConfig: { wandConfig: {
enabled: true, enabled: true,
maintainHistory: true, maintainHistory: true,
@@ -95,12 +93,10 @@ export const AgentBlock: BlockConfig<AgentResponse> = {
Current messages: {context} Current messages: {context}
{sources}
RULES: RULES:
1. Generate ONLY a valid JSON array - no markdown, no explanations 1. Generate ONLY a valid JSON array - no markdown, no explanations
2. Each message object must have "role" and "content" properties 2. Each message object must have "role" (system/user/assistant) and "content" (string)
3. Valid roles are: "system", "user", "assistant", "attachment" 3. You can generate any number of messages as needed
4. Content can be as long as necessary - don't truncate 4. Content can be as long as necessary - don't truncate
5. If editing existing messages, preserve structure unless asked to change it 5. If editing existing messages, preserve structure unless asked to change it
6. For new agents, create DETAILED, PROFESSIONAL system prompts that include: 6. For new agents, create DETAILED, PROFESSIONAL system prompts that include:
@@ -110,16 +106,6 @@ RULES:
- Critical thinking or quality guidelines - Critical thinking or quality guidelines
- How to handle edge cases and uncertainty - How to handle edge cases and uncertainty
ATTACHMENTS:
- Use role "attachment" to include images, audio, video, or documents in a multimodal conversation
- IMPORTANT: If an attachment message in the current context has an "attachment" object with file data, ALWAYS preserve that entire "attachment" object exactly as-is
- When creating NEW attachment messages, you can either:
1. Just set role to "attachment" with descriptive content - user will upload the file manually
2. Select a file from the available workspace files by including "fileId" in the attachment object (optional)
- You do NOT have to select a file - it's completely optional
- Example without file: {"role": "attachment", "content": "Analyze this image for text and objects"}
- Example with file selection: {"role": "attachment", "content": "Analyze this image", "attachment": {"fileId": "abc123"}}
EXAMPLES: EXAMPLES:
Research agent: Research agent:
@@ -128,23 +114,14 @@ Research agent:
Code reviewer: Code reviewer:
[{"role": "system", "content": "You are a Senior Code Reviewer with expertise in software architecture, security, and best practices. Your role is to provide thorough, constructive code reviews that improve code quality and help developers grow.\\n\\n## Review Methodology\\n\\n1. **Security First**: Check for vulnerabilities including injection attacks, authentication flaws, data exposure, and insecure dependencies.\\n\\n2. **Code Quality**: Evaluate readability, maintainability, adherence to DRY/SOLID principles, and appropriate abstraction levels.\\n\\n3. **Performance**: Identify potential bottlenecks, unnecessary computations, memory leaks, and optimization opportunities.\\n\\n4. **Testing**: Assess test coverage, edge case handling, and testability of the code structure.\\n\\n## Output Format\\n\\n### Summary\\nBrief overview of the code's purpose and overall assessment.\\n\\n### Critical Issues\\nSecurity vulnerabilities or bugs that must be fixed before merging.\\n\\n### Improvements\\nSuggested enhancements with clear explanations of why and how.\\n\\n### Positive Aspects\\nHighlight well-written code to reinforce good practices.\\n\\nBe specific with line references. Provide code examples for suggested changes. Balance critique with encouragement."}, {"role": "user", "content": "<start.input>"}] [{"role": "system", "content": "You are a Senior Code Reviewer with expertise in software architecture, security, and best practices. Your role is to provide thorough, constructive code reviews that improve code quality and help developers grow.\\n\\n## Review Methodology\\n\\n1. **Security First**: Check for vulnerabilities including injection attacks, authentication flaws, data exposure, and insecure dependencies.\\n\\n2. **Code Quality**: Evaluate readability, maintainability, adherence to DRY/SOLID principles, and appropriate abstraction levels.\\n\\n3. **Performance**: Identify potential bottlenecks, unnecessary computations, memory leaks, and optimization opportunities.\\n\\n4. **Testing**: Assess test coverage, edge case handling, and testability of the code structure.\\n\\n## Output Format\\n\\n### Summary\\nBrief overview of the code's purpose and overall assessment.\\n\\n### Critical Issues\\nSecurity vulnerabilities or bugs that must be fixed before merging.\\n\\n### Improvements\\nSuggested enhancements with clear explanations of why and how.\\n\\n### Positive Aspects\\nHighlight well-written code to reinforce good practices.\\n\\nBe specific with line references. Provide code examples for suggested changes. Balance critique with encouragement."}, {"role": "user", "content": "<start.input>"}]
Image analysis agent: Writing assistant:
[{"role": "system", "content": "You are an expert image analyst. Describe images in detail, identify objects, text, and patterns. Provide structured analysis."}, {"role": "attachment", "content": "Analyze this image"}] [{"role": "system", "content": "You are a skilled Writing Editor and Coach. Your role is to help users improve their writing through constructive feedback, editing suggestions, and guidance on style, clarity, and structure.\\n\\n## Editing Approach\\n\\n1. **Clarity**: Ensure ideas are expressed clearly and concisely. Eliminate jargon unless appropriate for the audience.\\n\\n2. **Structure**: Evaluate logical flow, paragraph organization, and transitions between ideas.\\n\\n3. **Voice & Tone**: Maintain consistency and appropriateness for the intended audience and purpose.\\n\\n4. **Grammar & Style**: Correct errors while respecting the author's voice.\\n\\n## Output Format\\n\\n### Overall Impression\\nBrief assessment of the piece's strengths and areas for improvement.\\n\\n### Structural Feedback\\nComments on organization, flow, and logical progression.\\n\\n### Line-Level Edits\\nSpecific suggestions with explanations, not just corrections.\\n\\n### Revised Version\\nWhen appropriate, provide an edited version demonstrating improvements.\\n\\nBe encouraging while honest. Explain the reasoning behind suggestions to help the writer improve."}, {"role": "user", "content": "<start.input>"}]
Return ONLY the JSON array.`, Return ONLY the JSON array.`,
placeholder: 'Describe what you want to create or change...', placeholder: 'Describe what you want to create or change...',
generationType: 'json-object', generationType: 'json-object',
}, },
}, },
{
id: 'messagesRaw',
title: 'Messages',
type: 'code',
canonicalParamId: 'messages',
placeholder: '[{"role": "system", "content": "..."}, {"role": "user", "content": "..."}]',
language: 'json',
mode: 'advanced',
},
{ {
id: 'model', id: 'model',
title: 'Model', title: 'Model',

View File

@@ -2417,177 +2417,4 @@ describe('EdgeManager', () => {
expect(successReady).toContain(targetId) expect(successReady).toContain(targetId)
}) })
}) })
describe('Condition with loop downstream - deactivation propagation', () => {
it('should deactivate nodes after loop when condition branch containing loop is deactivated', () => {
// Scenario: condition → (if) → sentinel_start → loopBody → sentinel_end → (loop_exit) → after_loop
// → (else) → other_branch
// When condition takes "else" path, the entire if-branch including nodes after the loop should be deactivated
const conditionId = 'condition'
const sentinelStartId = 'sentinel-start'
const loopBodyId = 'loop-body'
const sentinelEndId = 'sentinel-end'
const afterLoopId = 'after-loop'
const otherBranchId = 'other-branch'
const conditionNode = createMockNode(conditionId, [
{ target: sentinelStartId, sourceHandle: 'condition-if' },
{ target: otherBranchId, sourceHandle: 'condition-else' },
])
const sentinelStartNode = createMockNode(
sentinelStartId,
[{ target: loopBodyId }],
[conditionId]
)
const loopBodyNode = createMockNode(
loopBodyId,
[{ target: sentinelEndId }],
[sentinelStartId]
)
const sentinelEndNode = createMockNode(
sentinelEndId,
[
{ target: sentinelStartId, sourceHandle: 'loop_continue' },
{ target: afterLoopId, sourceHandle: 'loop_exit' },
],
[loopBodyId]
)
const afterLoopNode = createMockNode(afterLoopId, [], [sentinelEndId])
const otherBranchNode = createMockNode(otherBranchId, [], [conditionId])
const nodes = new Map<string, DAGNode>([
[conditionId, conditionNode],
[sentinelStartId, sentinelStartNode],
[loopBodyId, loopBodyNode],
[sentinelEndId, sentinelEndNode],
[afterLoopId, afterLoopNode],
[otherBranchId, otherBranchNode],
])
const dag = createMockDAG(nodes)
const edgeManager = new EdgeManager(dag)
// Condition selects "else" branch, deactivating the "if" branch (which contains the loop)
const readyNodes = edgeManager.processOutgoingEdges(conditionNode, { selectedOption: 'else' })
// Only otherBranch should be ready
expect(readyNodes).toContain(otherBranchId)
expect(readyNodes).not.toContain(sentinelStartId)
// afterLoop should NOT be ready - its incoming edge from sentinel_end should be deactivated
expect(readyNodes).not.toContain(afterLoopId)
// Verify that countActiveIncomingEdges returns 0 for afterLoop
// (meaning the loop_exit edge was properly deactivated)
// Note: isNodeReady returns true when all edges are deactivated (no pending deps),
// but the node won't be in readyNodes since it wasn't reached via an active path
expect(edgeManager.isNodeReady(afterLoopNode)).toBe(true) // All edges deactivated = no blocking deps
})
it('should deactivate nodes after parallel when condition branch containing parallel is deactivated', () => {
// Similar scenario with parallel instead of loop
const conditionId = 'condition'
const parallelStartId = 'parallel-start'
const parallelBodyId = 'parallel-body'
const parallelEndId = 'parallel-end'
const afterParallelId = 'after-parallel'
const otherBranchId = 'other-branch'
const conditionNode = createMockNode(conditionId, [
{ target: parallelStartId, sourceHandle: 'condition-if' },
{ target: otherBranchId, sourceHandle: 'condition-else' },
])
const parallelStartNode = createMockNode(
parallelStartId,
[{ target: parallelBodyId }],
[conditionId]
)
const parallelBodyNode = createMockNode(
parallelBodyId,
[{ target: parallelEndId }],
[parallelStartId]
)
const parallelEndNode = createMockNode(
parallelEndId,
[{ target: afterParallelId, sourceHandle: 'parallel_exit' }],
[parallelBodyId]
)
const afterParallelNode = createMockNode(afterParallelId, [], [parallelEndId])
const otherBranchNode = createMockNode(otherBranchId, [], [conditionId])
const nodes = new Map<string, DAGNode>([
[conditionId, conditionNode],
[parallelStartId, parallelStartNode],
[parallelBodyId, parallelBodyNode],
[parallelEndId, parallelEndNode],
[afterParallelId, afterParallelNode],
[otherBranchId, otherBranchNode],
])
const dag = createMockDAG(nodes)
const edgeManager = new EdgeManager(dag)
// Condition selects "else" branch
const readyNodes = edgeManager.processOutgoingEdges(conditionNode, { selectedOption: 'else' })
expect(readyNodes).toContain(otherBranchId)
expect(readyNodes).not.toContain(parallelStartId)
expect(readyNodes).not.toContain(afterParallelId)
// isNodeReady returns true when all edges are deactivated (no pending deps)
expect(edgeManager.isNodeReady(afterParallelNode)).toBe(true)
})
it('should still correctly handle normal loop exit (not deactivate when loop runs)', () => {
// When a loop actually executes and exits normally, after_loop should become ready
const sentinelStartId = 'sentinel-start'
const loopBodyId = 'loop-body'
const sentinelEndId = 'sentinel-end'
const afterLoopId = 'after-loop'
const sentinelStartNode = createMockNode(sentinelStartId, [{ target: loopBodyId }])
const loopBodyNode = createMockNode(
loopBodyId,
[{ target: sentinelEndId }],
[sentinelStartId]
)
const sentinelEndNode = createMockNode(
sentinelEndId,
[
{ target: sentinelStartId, sourceHandle: 'loop_continue' },
{ target: afterLoopId, sourceHandle: 'loop_exit' },
],
[loopBodyId]
)
const afterLoopNode = createMockNode(afterLoopId, [], [sentinelEndId])
const nodes = new Map<string, DAGNode>([
[sentinelStartId, sentinelStartNode],
[loopBodyId, loopBodyNode],
[sentinelEndId, sentinelEndNode],
[afterLoopId, afterLoopNode],
])
const dag = createMockDAG(nodes)
const edgeManager = new EdgeManager(dag)
// Simulate sentinel_end completing with loop_exit (loop is done)
const readyNodes = edgeManager.processOutgoingEdges(sentinelEndNode, {
selectedRoute: 'loop_exit',
})
// afterLoop should be ready
expect(readyNodes).toContain(afterLoopId)
})
})
}) })

View File

@@ -243,7 +243,7 @@ export class EdgeManager {
} }
for (const [, outgoingEdge] of targetNode.outgoingEdges) { for (const [, outgoingEdge] of targetNode.outgoingEdges) {
if (!this.isBackwardsEdge(outgoingEdge.sourceHandle)) { if (!this.isControlEdge(outgoingEdge.sourceHandle)) {
this.deactivateEdgeAndDescendants( this.deactivateEdgeAndDescendants(
targetId, targetId,
outgoingEdge.target, outgoingEdge.target,

View File

@@ -25,8 +25,6 @@ import {
validateModelProvider, validateModelProvider,
} from '@/executor/utils/permission-check' } from '@/executor/utils/permission-check'
import { executeProviderRequest } from '@/providers' import { executeProviderRequest } from '@/providers'
import { transformAttachmentMessages } from '@/providers/attachment'
import type { ProviderId } from '@/providers/types'
import { getProviderFromModel, transformBlockTool } from '@/providers/utils' import { getProviderFromModel, transformBlockTool } from '@/providers/utils'
import type { SerializedBlock } from '@/serializer/types' import type { SerializedBlock } from '@/serializer/types'
import { executeTool } from '@/tools' import { executeTool } from '@/tools'
@@ -60,15 +58,7 @@ export class AgentBlockHandler implements BlockHandler {
const providerId = getProviderFromModel(model) const providerId = getProviderFromModel(model)
const formattedTools = await this.formatTools(ctx, filteredInputs.tools || []) const formattedTools = await this.formatTools(ctx, filteredInputs.tools || [])
const streamingConfig = this.getStreamingConfig(ctx, block) const streamingConfig = this.getStreamingConfig(ctx, block)
const rawMessages = await this.buildMessages(ctx, filteredInputs) const messages = await this.buildMessages(ctx, filteredInputs)
// Transform attachment messages to provider-specific format (async for file fetching)
const messages = rawMessages
? await transformAttachmentMessages(rawMessages, {
providerId: providerId as ProviderId,
model,
})
: undefined
const providerRequest = this.buildProviderRequest({ const providerRequest = this.buildProviderRequest({
ctx, ctx,
@@ -816,44 +806,17 @@ export class AgentBlockHandler implements BlockHandler {
return messages.length > 0 ? messages : undefined return messages.length > 0 ? messages : undefined
} }
private extractValidMessages(messages?: Message[] | string): Message[] { private extractValidMessages(messages?: Message[]): Message[] {
if (!messages) return [] if (!messages || !Array.isArray(messages)) return []
// Handle raw JSON string input (from advanced mode) return messages.filter(
let messageArray: unknown[] (msg): msg is Message =>
if (typeof messages === 'string') { msg &&
const trimmed = messages.trim() typeof msg === 'object' &&
if (!trimmed) return [] 'role' in msg &&
try { 'content' in msg &&
const parsed = JSON.parse(trimmed) ['system', 'user', 'assistant'].includes(msg.role)
if (!Array.isArray(parsed)) { )
logger.warn('Parsed messages JSON is not an array', { parsed })
return []
}
messageArray = parsed
} catch (error) {
logger.warn('Failed to parse messages JSON string', {
error,
messages: trimmed.substring(0, 100),
})
return []
}
} else if (Array.isArray(messages)) {
messageArray = messages
} else {
return []
}
return messageArray.filter((msg): msg is Message => {
if (!msg || typeof msg !== 'object') return false
const m = msg as Record<string, unknown>
return (
'role' in m &&
'content' in m &&
typeof m.role === 'string' &&
['system', 'user', 'assistant', 'attachment'].includes(m.role)
)
})
} }
private processMemories(memories: any): Message[] { private processMemories(memories: any): Message[] {

View File

@@ -6,8 +6,8 @@ export interface AgentInputs {
systemPrompt?: string systemPrompt?: string
userPrompt?: string | object userPrompt?: string | object
memories?: any // Legacy memory block output memories?: any // Legacy memory block output
// New message array input (from messages-input subblock or raw JSON from advanced mode) // New message array input (from messages-input subblock)
messages?: Message[] | string messages?: Message[]
// Memory configuration // Memory configuration
memoryType?: 'none' | 'conversation' | 'sliding_window' | 'sliding_window_tokens' memoryType?: 'none' | 'conversation' | 'sliding_window' | 'sliding_window_tokens'
conversationId?: string // Required for all non-none memory types conversationId?: string // Required for all non-none memory types
@@ -42,25 +42,9 @@ export interface ToolInput {
customToolId?: string customToolId?: string
} }
/**
* Attachment content (files, images, documents)
*/
export interface AttachmentContent {
/** Source type: how the data was provided */
sourceType: 'url' | 'base64' | 'file'
/** The URL or base64 data */
data: string
/** MIME type (e.g., 'image/png', 'application/pdf', 'audio/mp3') */
mimeType?: string
/** Optional filename for file uploads */
fileName?: string
}
export interface Message { export interface Message {
role: 'system' | 'user' | 'assistant' | 'attachment' role: 'system' | 'user' | 'assistant'
content: string content: string
/** Attachment content for 'attachment' role messages */
attachment?: AttachmentContent
executionId?: string executionId?: string
function_call?: any function_call?: any
tool_calls?: any[] tool_calls?: any[]

View File

@@ -10,7 +10,6 @@ import {
type KnowledgeBaseArgs, type KnowledgeBaseArgs,
} from '@/lib/copilot/tools/shared/schemas' } from '@/lib/copilot/tools/shared/schemas'
import { useCopilotStore } from '@/stores/panel/copilot/store' import { useCopilotStore } from '@/stores/panel/copilot/store'
import { useWorkflowRegistry } from '@/stores/workflows/registry/store'
/** /**
* Client tool for knowledge base operations * Client tool for knowledge base operations
@@ -103,19 +102,7 @@ export class KnowledgeBaseClientTool extends BaseClientTool {
const logger = createLogger('KnowledgeBaseClientTool') const logger = createLogger('KnowledgeBaseClientTool')
try { try {
this.setState(ClientToolCallState.executing) this.setState(ClientToolCallState.executing)
const payload: KnowledgeBaseArgs = { ...(args || { operation: 'list' }) }
// Get the workspace ID from the workflow registry hydration state
const { hydration } = useWorkflowRegistry.getState()
const workspaceId = hydration.workspaceId
// Build payload with workspace ID included in args
const payload: KnowledgeBaseArgs = {
...(args || { operation: 'list' }),
args: {
...(args?.args || {}),
workspaceId: workspaceId || undefined,
},
}
const res = await fetch('/api/copilot/execute-copilot-server-tool', { const res = await fetch('/api/copilot/execute-copilot-server-tool', {
method: 'POST', method: 'POST',

View File

@@ -2508,10 +2508,6 @@ async function validateWorkflowSelectorIds(
for (const subBlockConfig of blockConfig.subBlocks) { for (const subBlockConfig of blockConfig.subBlocks) {
if (!SELECTOR_TYPES.has(subBlockConfig.type)) continue if (!SELECTOR_TYPES.has(subBlockConfig.type)) continue
// Skip oauth-input - credentials are pre-validated before edit application
// This allows existing collaborator credentials to remain untouched
if (subBlockConfig.type === 'oauth-input') continue
const subBlockValue = blockData.subBlocks?.[subBlockConfig.id]?.value const subBlockValue = blockData.subBlocks?.[subBlockConfig.id]?.value
if (!subBlockValue) continue if (!subBlockValue) continue
@@ -2577,295 +2573,6 @@ async function validateWorkflowSelectorIds(
return errors return errors
} }
/**
* Pre-validates credential and apiKey inputs in operations before they are applied.
* - Validates oauth-input (credential) IDs belong to the user
* - Filters out apiKey inputs for hosted models when isHosted is true
* - Also validates credentials and apiKeys in nestedNodes (blocks inside loop/parallel)
* Returns validation errors for any removed inputs.
*/
async function preValidateCredentialInputs(
operations: EditWorkflowOperation[],
context: { userId: string },
workflowState?: Record<string, unknown>
): Promise<{ filteredOperations: EditWorkflowOperation[]; errors: ValidationError[] }> {
const { isHosted } = await import('@/lib/core/config/feature-flags')
const { getHostedModels } = await import('@/providers/utils')
const logger = createLogger('PreValidateCredentials')
const errors: ValidationError[] = []
// Collect credential and apiKey inputs that need validation/filtering
const credentialInputs: Array<{
operationIndex: number
blockId: string
blockType: string
fieldName: string
value: string
nestedBlockId?: string
}> = []
const hostedApiKeyInputs: Array<{
operationIndex: number
blockId: string
blockType: string
model: string
nestedBlockId?: string
}> = []
const hostedModelsLower = isHosted ? new Set(getHostedModels().map((m) => m.toLowerCase())) : null
/**
* Collect credential inputs from a block's inputs based on its block config
*/
function collectCredentialInputs(
blockConfig: ReturnType<typeof getBlock>,
inputs: Record<string, unknown>,
opIndex: number,
blockId: string,
blockType: string,
nestedBlockId?: string
) {
if (!blockConfig) return
for (const subBlockConfig of blockConfig.subBlocks) {
if (subBlockConfig.type !== 'oauth-input') continue
const inputValue = inputs[subBlockConfig.id]
if (!inputValue || typeof inputValue !== 'string' || inputValue.trim() === '') continue
credentialInputs.push({
operationIndex: opIndex,
blockId,
blockType,
fieldName: subBlockConfig.id,
value: inputValue,
nestedBlockId,
})
}
}
/**
* Check if apiKey should be filtered for a block with the given model
*/
function collectHostedApiKeyInput(
inputs: Record<string, unknown>,
modelValue: string | undefined,
opIndex: number,
blockId: string,
blockType: string,
nestedBlockId?: string
) {
if (!hostedModelsLower || !inputs.apiKey) return
if (!modelValue || typeof modelValue !== 'string') return
if (hostedModelsLower.has(modelValue.toLowerCase())) {
hostedApiKeyInputs.push({
operationIndex: opIndex,
blockId,
blockType,
model: modelValue,
nestedBlockId,
})
}
}
operations.forEach((op, opIndex) => {
// Process main block inputs
if (op.params?.inputs && op.params?.type) {
const blockConfig = getBlock(op.params.type)
if (blockConfig) {
// Collect credentials from main block
collectCredentialInputs(
blockConfig,
op.params.inputs as Record<string, unknown>,
opIndex,
op.block_id,
op.params.type
)
// Check for apiKey inputs on hosted models
let modelValue = (op.params.inputs as Record<string, unknown>).model as string | undefined
// For edit operations, if model is not being changed, check existing block's model
if (
!modelValue &&
op.operation_type === 'edit' &&
(op.params.inputs as Record<string, unknown>).apiKey &&
workflowState
) {
const existingBlock = (workflowState.blocks as Record<string, unknown>)?.[op.block_id] as
| Record<string, unknown>
| undefined
const existingSubBlocks = existingBlock?.subBlocks as Record<string, unknown> | undefined
const existingModelSubBlock = existingSubBlocks?.model as
| Record<string, unknown>
| undefined
modelValue = existingModelSubBlock?.value as string | undefined
}
collectHostedApiKeyInput(
op.params.inputs as Record<string, unknown>,
modelValue,
opIndex,
op.block_id,
op.params.type
)
}
}
// Process nested nodes (blocks inside loop/parallel containers)
const nestedNodes = op.params?.nestedNodes as
| Record<string, Record<string, unknown>>
| undefined
if (nestedNodes) {
Object.entries(nestedNodes).forEach(([childId, childBlock]) => {
const childType = childBlock.type as string | undefined
const childInputs = childBlock.inputs as Record<string, unknown> | undefined
if (!childType || !childInputs) return
const childBlockConfig = getBlock(childType)
if (!childBlockConfig) return
// Collect credentials from nested block
collectCredentialInputs(
childBlockConfig,
childInputs,
opIndex,
op.block_id,
childType,
childId
)
// Check for apiKey inputs on hosted models in nested block
const modelValue = childInputs.model as string | undefined
collectHostedApiKeyInput(childInputs, modelValue, opIndex, op.block_id, childType, childId)
})
}
})
const hasCredentialsToValidate = credentialInputs.length > 0
const hasHostedApiKeysToFilter = hostedApiKeyInputs.length > 0
if (!hasCredentialsToValidate && !hasHostedApiKeysToFilter) {
return { filteredOperations: operations, errors }
}
// Deep clone operations so we can modify them
const filteredOperations = structuredClone(operations)
// Filter out apiKey inputs for hosted models and add validation errors
if (hasHostedApiKeysToFilter) {
logger.info('Filtering apiKey inputs for hosted models', { count: hostedApiKeyInputs.length })
for (const apiKeyInput of hostedApiKeyInputs) {
const op = filteredOperations[apiKeyInput.operationIndex]
// Handle nested block apiKey filtering
if (apiKeyInput.nestedBlockId) {
const nestedNodes = op.params?.nestedNodes as
| Record<string, Record<string, unknown>>
| undefined
const nestedBlock = nestedNodes?.[apiKeyInput.nestedBlockId]
const nestedInputs = nestedBlock?.inputs as Record<string, unknown> | undefined
if (nestedInputs?.apiKey) {
nestedInputs.apiKey = undefined
logger.debug('Filtered apiKey for hosted model in nested block', {
parentBlockId: apiKeyInput.blockId,
nestedBlockId: apiKeyInput.nestedBlockId,
model: apiKeyInput.model,
})
errors.push({
blockId: apiKeyInput.nestedBlockId,
blockType: apiKeyInput.blockType,
field: 'apiKey',
value: '[redacted]',
error: `Cannot set API key for hosted model "${apiKeyInput.model}" - API keys are managed by the platform when using hosted models`,
})
}
} else if (op.params?.inputs?.apiKey) {
// Handle main block apiKey filtering
op.params.inputs.apiKey = undefined
logger.debug('Filtered apiKey for hosted model', {
blockId: apiKeyInput.blockId,
model: apiKeyInput.model,
})
errors.push({
blockId: apiKeyInput.blockId,
blockType: apiKeyInput.blockType,
field: 'apiKey',
value: '[redacted]',
error: `Cannot set API key for hosted model "${apiKeyInput.model}" - API keys are managed by the platform when using hosted models`,
})
}
}
}
// Validate credential inputs
if (hasCredentialsToValidate) {
logger.info('Pre-validating credential inputs', {
credentialCount: credentialInputs.length,
userId: context.userId,
})
const allCredentialIds = credentialInputs.map((c) => c.value)
const validationResult = await validateSelectorIds('oauth-input', allCredentialIds, context)
const invalidSet = new Set(validationResult.invalid)
if (invalidSet.size > 0) {
for (const credInput of credentialInputs) {
if (!invalidSet.has(credInput.value)) continue
const op = filteredOperations[credInput.operationIndex]
// Handle nested block credential removal
if (credInput.nestedBlockId) {
const nestedNodes = op.params?.nestedNodes as
| Record<string, Record<string, unknown>>
| undefined
const nestedBlock = nestedNodes?.[credInput.nestedBlockId]
const nestedInputs = nestedBlock?.inputs as Record<string, unknown> | undefined
if (nestedInputs?.[credInput.fieldName]) {
delete nestedInputs[credInput.fieldName]
logger.info('Removed invalid credential from nested block', {
parentBlockId: credInput.blockId,
nestedBlockId: credInput.nestedBlockId,
field: credInput.fieldName,
invalidValue: credInput.value,
})
}
} else if (op.params?.inputs?.[credInput.fieldName]) {
// Handle main block credential removal
delete op.params.inputs[credInput.fieldName]
logger.info('Removed invalid credential from operation', {
blockId: credInput.blockId,
field: credInput.fieldName,
invalidValue: credInput.value,
})
}
const warningInfo = validationResult.warning ? `. ${validationResult.warning}` : ''
const errorBlockId = credInput.nestedBlockId ?? credInput.blockId
errors.push({
blockId: errorBlockId,
blockType: credInput.blockType,
field: credInput.fieldName,
value: credInput.value,
error: `Invalid credential ID "${credInput.value}" - credential does not exist or user doesn't have access${warningInfo}`,
})
}
logger.warn('Filtered out invalid credentials', {
invalidCount: invalidSet.size,
})
}
}
return { filteredOperations, errors }
}
async function getCurrentWorkflowStateFromDb( async function getCurrentWorkflowStateFromDb(
workflowId: string workflowId: string
): Promise<{ workflowState: any; subBlockValues: Record<string, Record<string, any>> }> { ): Promise<{ workflowState: any; subBlockValues: Record<string, Record<string, any>> }> {
@@ -2950,29 +2657,12 @@ export const editWorkflowServerTool: BaseServerTool<EditWorkflowParams, any> = {
// Get permission config for the user // Get permission config for the user
const permissionConfig = context?.userId ? await getUserPermissionConfig(context.userId) : null const permissionConfig = context?.userId ? await getUserPermissionConfig(context.userId) : null
// Pre-validate credential and apiKey inputs before applying operations
// This filters out invalid credentials and apiKeys for hosted models
let operationsToApply = operations
const credentialErrors: ValidationError[] = []
if (context?.userId) {
const { filteredOperations, errors: credErrors } = await preValidateCredentialInputs(
operations,
{ userId: context.userId },
workflowState
)
operationsToApply = filteredOperations
credentialErrors.push(...credErrors)
}
// Apply operations directly to the workflow state // Apply operations directly to the workflow state
const { const {
state: modifiedWorkflowState, state: modifiedWorkflowState,
validationErrors, validationErrors,
skippedItems, skippedItems,
} = applyOperationsToWorkflowState(workflowState, operationsToApply, permissionConfig) } = applyOperationsToWorkflowState(workflowState, operations, permissionConfig)
// Add credential validation errors
validationErrors.push(...credentialErrors)
// Get workspaceId for selector validation // Get workspaceId for selector validation
let workspaceId: string | undefined let workspaceId: string | undefined

View File

@@ -109,15 +109,9 @@ export const anthropicProvider: ProviderConfig = {
], ],
}) })
} else { } else {
// Handle content that's already in array format (from transformAttachmentMessages)
const content = Array.isArray(msg.content)
? msg.content
: msg.content
? [{ type: 'text', text: msg.content }]
: []
messages.push({ messages.push({
role: msg.role === 'assistant' ? 'assistant' : 'user', role: msg.role === 'assistant' ? 'assistant' : 'user',
content, content: msg.content ? [{ type: 'text', text: msg.content }] : [],
}) })
} }
}) })

View File

@@ -1,397 +0,0 @@
/**
* Centralized attachment content transformation for all providers.
*
* Strategy: Always normalize to base64 first, then create provider-specific formats.
* This eliminates URL accessibility issues and simplifies provider handling.
*/
import { createLogger } from '@sim/logger'
import { bufferToBase64 } from '@/lib/uploads/utils/file-utils'
import { downloadFileFromUrl } from '@/lib/uploads/utils/file-utils.server'
import { supportsVision } from '@/providers/models'
import type { ProviderId } from '@/providers/types'
const logger = createLogger('AttachmentTransformer')
/**
* Generic message type for attachment transformation.
*/
interface TransformableMessage {
role: string
content: string | any[] | null
attachment?: AttachmentContent
[key: string]: any
}
/**
* Attachment content (files, images, documents)
*/
export interface AttachmentContent {
sourceType: 'url' | 'base64' | 'file'
data: string
mimeType?: string
fileName?: string
}
/**
* Normalized attachment data (always base64)
*/
interface NormalizedAttachment {
base64: string
mimeType: string
}
/**
* Configuration for attachment transformation
*/
interface AttachmentTransformConfig {
providerId: ProviderId
model: string
}
/**
* Checks if a model supports attachments (vision/multimodal content).
*/
export function modelSupportsAttachments(model: string): boolean {
return supportsVision(model)
}
/**
* Transforms messages with 'attachment' role into provider-compatible format.
*/
export async function transformAttachmentMessages<T extends TransformableMessage>(
messages: T[],
config: AttachmentTransformConfig
): Promise<T[]> {
const { providerId, model } = config
const supportsAttachments = modelSupportsAttachments(model)
if (!supportsAttachments) {
return transformAttachmentsToText(messages) as T[]
}
const result: T[] = []
for (const msg of messages) {
if (msg.role !== 'attachment') {
result.push(msg)
continue
}
const attachmentContent = await createProviderAttachmentContent(msg, providerId)
if (!attachmentContent) {
logger.warn('Could not create attachment content for message', { msg })
continue
}
// Merge with previous user message or create new one
const lastMessage = result[result.length - 1]
if (lastMessage && lastMessage.role === 'user') {
const existingContent = ensureContentArray(lastMessage, providerId)
existingContent.push(attachmentContent)
lastMessage.content = existingContent as any
} else {
result.push({
role: 'user',
content: [attachmentContent] as any,
} as T)
}
}
// Ensure all user messages have consistent content format
return result.map((msg) => {
if (msg.role === 'user' && typeof msg.content === 'string') {
return {
...msg,
content: [createTextContent(msg.content, providerId)] as any,
}
}
return msg
})
}
/**
* Transforms attachment messages to text placeholders for non-vision models
*/
function transformAttachmentsToText<T extends TransformableMessage>(messages: T[]): T[] {
const result: T[] = []
for (const msg of messages) {
if (msg.role !== 'attachment') {
result.push(msg)
continue
}
const attachment = msg.attachment
const mimeType = attachment?.mimeType || 'unknown type'
const fileName = attachment?.fileName || 'file'
const lastMessage = result[result.length - 1]
if (lastMessage && lastMessage.role === 'user') {
const currentContent = typeof lastMessage.content === 'string' ? lastMessage.content : ''
lastMessage.content = `${currentContent}\n[Attached file: ${fileName} (${mimeType}) - Note: This model does not support file/image inputs]`
} else {
result.push({
role: 'user',
content: `[Attached file: ${fileName} (${mimeType}) - Note: This model does not support file/image inputs]`,
} as T)
}
}
return result
}
/**
* Ensures a user message has content as an array for multimodal support
*/
function ensureContentArray(msg: TransformableMessage, providerId: ProviderId): any[] {
if (Array.isArray(msg.content)) {
return msg.content
}
if (typeof msg.content === 'string' && msg.content) {
return [createTextContent(msg.content, providerId)]
}
return []
}
/**
* Creates provider-specific text content block
*/
export function createTextContent(text: string, providerId: ProviderId): any {
switch (providerId) {
case 'google':
case 'vertex':
return { text }
default:
return { type: 'text', text }
}
}
/**
* Normalizes attachment data to base64.
* Fetches URLs and converts to base64, extracts base64 from data URLs.
*/
async function normalizeToBase64(
attachment: AttachmentContent
): Promise<NormalizedAttachment | null> {
const { sourceType, data, mimeType } = attachment
if (!data || !data.trim()) {
logger.warn('Empty attachment data')
return null
}
const trimmedData = data.trim()
// Already base64
if (sourceType === 'base64') {
// Handle data URL format: data:mime;base64,xxx
if (trimmedData.startsWith('data:')) {
const match = trimmedData.match(/^data:([^;]+);base64,(.+)$/)
if (match) {
return { base64: match[2], mimeType: match[1] }
}
}
// Raw base64
return { base64: trimmedData, mimeType: mimeType || 'application/octet-stream' }
}
// URL or file path - need to fetch
if (sourceType === 'url' || sourceType === 'file') {
try {
logger.info('Fetching attachment for base64 conversion', {
url: trimmedData.substring(0, 50),
})
const buffer = await downloadFileFromUrl(trimmedData)
const base64 = bufferToBase64(buffer)
return { base64, mimeType: mimeType || 'application/octet-stream' }
} catch (error) {
logger.error('Failed to fetch attachment', { error, url: trimmedData.substring(0, 50) })
return null
}
}
return null
}
/**
* Creates provider-specific attachment content from an attachment message.
* First normalizes to base64, then creates the provider format.
*/
async function createProviderAttachmentContent(
msg: TransformableMessage,
providerId: ProviderId
): Promise<any> {
const attachment = msg.attachment
if (!attachment) return null
// Normalize to base64 first
const normalized = await normalizeToBase64(attachment)
if (!normalized) {
return createTextContent('[Failed to load attachment]', providerId)
}
const { base64, mimeType } = normalized
switch (providerId) {
case 'anthropic':
return createAnthropicContent(base64, mimeType)
case 'google':
case 'vertex':
return createGeminiContent(base64, mimeType)
case 'mistral':
return createMistralContent(base64, mimeType)
case 'bedrock':
return createBedrockContent(base64, mimeType)
default:
// OpenAI format (OpenAI, Azure, xAI, DeepSeek, Cerebras, Groq, OpenRouter, Ollama, vLLM)
return createOpenAIContent(base64, mimeType)
}
}
/**
* OpenAI-compatible content (images only via base64 data URL)
*/
function createOpenAIContent(base64: string, mimeType: string): any {
const isImage = mimeType.startsWith('image/')
const isAudio = mimeType.startsWith('audio/')
if (isImage) {
return {
type: 'image_url',
image_url: {
url: `data:${mimeType};base64,${base64}`,
detail: 'auto',
},
}
}
if (isAudio) {
return {
type: 'input_audio',
input_audio: {
data: base64,
format: mimeType === 'audio/wav' ? 'wav' : 'mp3',
},
}
}
// OpenAI Chat API doesn't support other file types directly
// For PDFs/docs, return a text placeholder
logger.warn(`OpenAI does not support ${mimeType} attachments in Chat API`)
return {
type: 'text',
text: `[Attached file: ${mimeType} - OpenAI Chat API only supports images and audio]`,
}
}
/**
* Anthropic-compatible content (images and PDFs)
*/
function createAnthropicContent(base64: string, mimeType: string): any {
const isImage = mimeType.startsWith('image/')
const isPdf = mimeType === 'application/pdf'
if (isImage) {
return {
type: 'image',
source: {
type: 'base64',
media_type: mimeType,
data: base64,
},
}
}
if (isPdf) {
return {
type: 'document',
source: {
type: 'base64',
media_type: 'application/pdf',
data: base64,
},
}
}
return {
type: 'text',
text: `[Attached file: ${mimeType} - Anthropic supports images and PDFs only]`,
}
}
/**
* Google Gemini-compatible content (inlineData format)
*/
function createGeminiContent(base64: string, mimeType: string): any {
// Gemini supports a wide range of file types via inlineData
return {
inlineData: {
mimeType,
data: base64,
},
}
}
/**
* Mistral-compatible content (images only, data URL format)
*/
function createMistralContent(base64: string, mimeType: string): any {
const isImage = mimeType.startsWith('image/')
if (isImage) {
// Mistral uses direct string for image_url, not nested object
return {
type: 'image_url',
image_url: `data:${mimeType};base64,${base64}`,
}
}
return {
type: 'text',
text: `[Attached file: ${mimeType} - Mistral supports images only]`,
}
}
/**
* AWS Bedrock-compatible content (images and PDFs)
*/
function createBedrockContent(base64: string, mimeType: string): any {
const isImage = mimeType.startsWith('image/')
const isPdf = mimeType === 'application/pdf'
// Determine image format from mimeType
const getImageFormat = (mime: string): string => {
if (mime.includes('jpeg') || mime.includes('jpg')) return 'jpeg'
if (mime.includes('png')) return 'png'
if (mime.includes('gif')) return 'gif'
if (mime.includes('webp')) return 'webp'
return 'png'
}
if (isImage) {
// Return a marker object that the Bedrock provider will convert to proper format
return {
type: 'bedrock_image',
format: getImageFormat(mimeType),
data: base64,
}
}
if (isPdf) {
return {
type: 'bedrock_document',
format: 'pdf',
data: base64,
}
}
return {
type: 'text',
text: `[Attached file: ${mimeType} - Bedrock supports images and PDFs only]`,
}
}

View File

@@ -16,7 +16,6 @@ import type { StreamingExecution } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers' import { MAX_TOOL_ITERATIONS } from '@/providers'
import { import {
checkForForcedToolUsage, checkForForcedToolUsage,
convertToBedrockContentBlocks,
createReadableStreamFromBedrockStream, createReadableStreamFromBedrockStream,
generateToolUseId, generateToolUseId,
getBedrockInferenceProfileId, getBedrockInferenceProfileId,
@@ -117,11 +116,9 @@ export const bedrockProvider: ProviderConfig = {
} }
} else { } else {
const role: ConversationRole = msg.role === 'assistant' ? 'assistant' : 'user' const role: ConversationRole = msg.role === 'assistant' ? 'assistant' : 'user'
// Handle multimodal content arrays
const contentBlocks = convertToBedrockContentBlocks(msg.content || '')
messages.push({ messages.push({
role, role,
content: contentBlocks, content: [{ text: msg.content || '' }],
}) })
} }
} }

View File

@@ -1,199 +1,9 @@
import type { import type { ConverseStreamOutput } from '@aws-sdk/client-bedrock-runtime'
ContentBlock,
ConverseStreamOutput,
ImageFormat,
} from '@aws-sdk/client-bedrock-runtime'
import { createLogger } from '@sim/logger' import { createLogger } from '@sim/logger'
import { trackForcedToolUsage } from '@/providers/utils' import { trackForcedToolUsage } from '@/providers/utils'
const logger = createLogger('BedrockUtils') const logger = createLogger('BedrockUtils')
/**
* Converts message content (string or array) to Bedrock ContentBlock array.
* Handles multimodal content including images and documents.
*/
export function convertToBedrockContentBlocks(content: string | any[]): ContentBlock[] {
// Simple string content
if (typeof content === 'string') {
return [{ text: content || '' }]
}
// Array content - could be multimodal
if (!Array.isArray(content)) {
return [{ text: String(content) || '' }]
}
const blocks: ContentBlock[] = []
for (const item of content) {
if (!item) continue
// Text content
if (item.type === 'text' && item.text) {
blocks.push({ text: item.text })
continue
}
// Gemini-style text (just { text: "..." })
if (typeof item.text === 'string' && !item.type) {
blocks.push({ text: item.text })
continue
}
// Bedrock image content (from agent handler)
if (item.type === 'bedrock_image') {
const imageBlock = createBedrockImageBlock(item)
if (imageBlock) {
blocks.push(imageBlock)
}
continue
}
// Bedrock document content (from agent handler)
if (item.type === 'bedrock_document') {
const docBlock = createBedrockDocumentBlock(item)
if (docBlock) {
blocks.push(docBlock)
}
continue
}
// OpenAI-style image_url (fallback for direct OpenAI format)
if (item.type === 'image_url' && item.image_url) {
const url = typeof item.image_url === 'string' ? item.image_url : item.image_url?.url
if (url) {
const imageBlock = createBedrockImageBlockFromUrl(url)
if (imageBlock) {
blocks.push(imageBlock)
}
}
continue
}
// Unknown type - log warning and skip
logger.warn('Unknown content block type in Bedrock conversion:', { type: item.type })
}
// Ensure at least one text block
if (blocks.length === 0) {
blocks.push({ text: '' })
}
return blocks
}
/**
* Creates a Bedrock image ContentBlock from a bedrock_image item
*/
function createBedrockImageBlock(item: {
format: string
sourceType: string
data?: string
url?: string
}): ContentBlock | null {
const format = (item.format || 'png') as ImageFormat
if (item.sourceType === 'base64' && item.data) {
// Convert base64 to Uint8Array
const bytes = base64ToUint8Array(item.data)
return {
image: {
format,
source: { bytes },
},
}
}
if (item.sourceType === 'url' && item.url) {
// For URLs, we need to fetch the image and convert to bytes
// This is a limitation - Bedrock doesn't support URL sources directly
// The provider layer should handle this, or we log a warning
logger.warn('Bedrock does not support image URLs directly. Image will be skipped.', {
url: item.url,
})
// Return a text placeholder
return { text: `[Image from URL: ${item.url}]` }
}
return null
}
/**
* Creates a Bedrock document ContentBlock from a bedrock_document item
*/
function createBedrockDocumentBlock(item: {
format: string
sourceType: string
data?: string
url?: string
}): ContentBlock | null {
if (item.sourceType === 'base64' && item.data) {
const bytes = base64ToUint8Array(item.data)
return {
document: {
format: 'pdf',
name: 'document',
source: { bytes },
},
}
}
if (item.sourceType === 'url' && item.url) {
logger.warn('Bedrock does not support document URLs directly. Document will be skipped.', {
url: item.url,
})
return { text: `[Document from URL: ${item.url}]` }
}
return null
}
/**
* Creates a Bedrock image ContentBlock from a data URL or regular URL
*/
function createBedrockImageBlockFromUrl(url: string): ContentBlock | null {
// Check if it's a data URL (base64)
if (url.startsWith('data:')) {
const match = url.match(/^data:image\/(\w+);base64,(.+)$/)
if (match) {
let format: ImageFormat = match[1] as ImageFormat
// Normalize jpg to jpeg
if (format === ('jpg' as ImageFormat)) {
format = 'jpeg'
}
const base64Data = match[2]
const bytes = base64ToUint8Array(base64Data)
return {
image: {
format,
source: { bytes },
},
}
}
}
// Regular URL - Bedrock doesn't support this directly
logger.warn('Bedrock does not support image URLs directly. Image will be skipped.', { url })
return { text: `[Image from URL: ${url}]` }
}
/**
* Converts a base64 string to Uint8Array
*/
function base64ToUint8Array(base64: string): Uint8Array {
// Handle browser and Node.js environments
if (typeof Buffer !== 'undefined') {
return Buffer.from(base64, 'base64')
}
// Browser fallback
const binaryString = atob(base64)
const bytes = new Uint8Array(binaryString.length)
for (let i = 0; i < binaryString.length; i++) {
bytes[i] = binaryString.charCodeAt(i)
}
return bytes
}
export interface BedrockStreamUsage { export interface BedrockStreamUsage {
inputTokens: number inputTokens: number
outputTokens: number outputTokens: number

View File

@@ -72,75 +72,6 @@ export function cleanSchemaForGemini(schema: SchemaUnion): SchemaUnion {
return cleanedSchema return cleanedSchema
} }
/**
* Converts an array of content items to Gemini-compatible Part array.
* Handles various formats from the attachment transformer.
*/
function convertContentArrayToGeminiParts(contentArray: any[]): Part[] {
const parts: Part[] = []
for (const item of contentArray) {
if (!item) continue
// Gemini-native text format: { text: "..." }
if (typeof item.text === 'string') {
parts.push({ text: item.text })
continue
}
// OpenAI-style text: { type: 'text', text: '...' }
if (item.type === 'text' && typeof item.text === 'string') {
parts.push({ text: item.text })
continue
}
// Gemini-native inlineData format (from attachment transformer)
if (item.inlineData) {
parts.push({ inlineData: item.inlineData })
continue
}
// Gemini-native fileData format (from attachment transformer)
if (item.fileData) {
parts.push({ fileData: item.fileData })
continue
}
// OpenAI-style image_url - convert to Gemini format
if (item.type === 'image_url' && item.image_url) {
const url = typeof item.image_url === 'string' ? item.image_url : item.image_url?.url
if (url) {
// Check if it's a data URL (base64)
if (url.startsWith('data:')) {
const match = url.match(/^data:([^;]+);base64,(.+)$/)
if (match) {
parts.push({
inlineData: {
mimeType: match[1],
data: match[2],
},
})
}
} else {
// External URL
parts.push({
fileData: {
mimeType: 'image/jpeg', // Default, Gemini will detect actual type
fileUri: url,
},
})
}
}
continue
}
// Unknown type - log warning
logger.warn('Unknown content item type in Gemini conversion:', { type: item.type })
}
return parts
}
/** /**
* Extracts text content from a Gemini response candidate. * Extracts text content from a Gemini response candidate.
* Filters out thought parts (model reasoning) from the output. * Filters out thought parts (model reasoning) from the output.
@@ -249,13 +180,7 @@ export function convertToGeminiFormat(request: ProviderRequest): {
} else if (message.role === 'user' || message.role === 'assistant') { } else if (message.role === 'user' || message.role === 'assistant') {
const geminiRole = message.role === 'user' ? 'user' : 'model' const geminiRole = message.role === 'user' ? 'user' : 'model'
// Handle multimodal content (arrays with text/image/file parts) if (message.content) {
if (Array.isArray(message.content)) {
const parts: Part[] = convertContentArrayToGeminiParts(message.content)
if (parts.length > 0) {
contents.push({ role: geminiRole, parts })
}
} else if (message.content) {
contents.push({ role: geminiRole, parts: [{ text: message.content }] }) contents.push({ role: geminiRole, parts: [{ text: message.content }] })
} }

View File

@@ -34,8 +34,6 @@ export interface ModelCapabilities {
toolUsageControl?: boolean toolUsageControl?: boolean
computerUse?: boolean computerUse?: boolean
nativeStructuredOutputs?: boolean nativeStructuredOutputs?: boolean
/** Whether the model supports vision/multimodal inputs (images, audio, video, PDFs) */
vision?: boolean
maxOutputTokens?: { maxOutputTokens?: {
/** Maximum tokens for streaming requests */ /** Maximum tokens for streaming requests */
max: number max: number
@@ -122,7 +120,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 2 }, temperature: { min: 0, max: 2 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -135,7 +132,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-12-11', updatedAt: '2025-12-11',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'minimal', 'low', 'medium', 'high', 'xhigh'], values: ['none', 'minimal', 'low', 'medium', 'high', 'xhigh'],
}, },
@@ -154,7 +150,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-11-14', updatedAt: '2025-11-14',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'low', 'medium', 'high'], values: ['none', 'low', 'medium', 'high'],
}, },
@@ -227,7 +222,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -246,7 +240,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -265,7 +258,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -295,7 +287,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-06-17', updatedAt: '2025-06-17',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['low', 'medium', 'high'], values: ['low', 'medium', 'high'],
}, },
@@ -311,7 +302,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-06-17', updatedAt: '2025-06-17',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['low', 'medium', 'high'], values: ['low', 'medium', 'high'],
}, },
@@ -327,7 +317,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-06-17', updatedAt: '2025-06-17',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['low', 'medium', 'high'], values: ['low', 'medium', 'high'],
}, },
@@ -344,7 +333,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 2 }, temperature: { min: 0, max: 2 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -358,7 +346,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 2 }, temperature: { min: 0, max: 2 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -372,7 +359,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 2 }, temperature: { min: 0, max: 2 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -399,7 +385,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 2 }, temperature: { min: 0, max: 2 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -412,7 +397,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-12-11', updatedAt: '2025-12-11',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'minimal', 'low', 'medium', 'high', 'xhigh'], values: ['none', 'minimal', 'low', 'medium', 'high', 'xhigh'],
}, },
@@ -431,7 +415,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-11-14', updatedAt: '2025-11-14',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'low', 'medium', 'high'], values: ['none', 'low', 'medium', 'high'],
}, },
@@ -450,7 +433,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-11-14', updatedAt: '2025-11-14',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'low', 'medium', 'high'], values: ['none', 'low', 'medium', 'high'],
}, },
@@ -469,7 +451,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-11-14', updatedAt: '2025-11-14',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'low', 'medium', 'high'], values: ['none', 'low', 'medium', 'high'],
}, },
@@ -488,7 +469,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-11-14', updatedAt: '2025-11-14',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['none', 'medium', 'high'], values: ['none', 'medium', 'high'],
}, },
@@ -507,7 +487,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -526,7 +505,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -545,7 +523,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-08-07', updatedAt: '2025-08-07',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['minimal', 'low', 'medium', 'high'], values: ['minimal', 'low', 'medium', 'high'],
}, },
@@ -575,7 +552,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-06-15', updatedAt: '2025-06-15',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['low', 'medium', 'high'], values: ['low', 'medium', 'high'],
}, },
@@ -591,7 +567,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
updatedAt: '2025-06-15', updatedAt: '2025-06-15',
}, },
capabilities: { capabilities: {
vision: true,
reasoningEffort: { reasoningEffort: {
values: ['low', 'medium', 'high'], values: ['low', 'medium', 'high'],
}, },
@@ -606,9 +581,7 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
output: 8.0, output: 8.0,
updatedAt: '2025-06-15', updatedAt: '2025-06-15',
}, },
capabilities: { capabilities: {},
vision: true,
},
contextWindow: 1000000, contextWindow: 1000000,
}, },
{ {
@@ -647,7 +620,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -663,7 +635,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -678,7 +649,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -694,7 +664,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -710,7 +679,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -725,7 +693,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -741,7 +708,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
computerUse: true, computerUse: true,
maxOutputTokens: { max: 8192, default: 8192 }, maxOutputTokens: { max: 8192, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -757,7 +723,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
computerUse: true, computerUse: true,
maxOutputTokens: { max: 8192, default: 8192 }, maxOutputTokens: { max: 8192, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -771,7 +736,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
modelPatterns: [/^gemini/], modelPatterns: [/^gemini/],
capabilities: { capabilities: {
toolUsageControl: true, toolUsageControl: true,
vision: true,
}, },
icon: GeminiIcon, icon: GeminiIcon,
models: [ models: [
@@ -883,7 +847,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
icon: VertexIcon, icon: VertexIcon,
capabilities: { capabilities: {
toolUsageControl: true, toolUsageControl: true,
vision: true,
}, },
models: [ models: [
{ {
@@ -1042,7 +1005,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
icon: xAIIcon, icon: xAIIcon,
capabilities: { capabilities: {
toolUsageControl: true, toolUsageControl: true,
vision: true,
}, },
models: [ models: [
{ {
@@ -1315,9 +1277,7 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
output: 0.34, output: 0.34,
updatedAt: '2026-01-27', updatedAt: '2026-01-27',
}, },
capabilities: { capabilities: {},
vision: true,
},
contextWindow: 131072, contextWindow: 131072,
}, },
{ {
@@ -1327,9 +1287,7 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
output: 0.6, output: 0.6,
updatedAt: '2026-01-27', updatedAt: '2026-01-27',
}, },
capabilities: { capabilities: {},
vision: true,
},
contextWindow: 131072, contextWindow: 131072,
}, },
{ {
@@ -1411,7 +1369,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1424,7 +1381,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1497,7 +1453,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1510,7 +1465,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1535,7 +1489,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1548,7 +1501,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1597,7 +1549,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1610,7 +1561,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1635,7 +1585,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1648,7 +1597,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1661,7 +1609,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1674,7 +1621,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1699,7 +1645,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1712,7 +1657,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 256000, contextWindow: 256000,
}, },
@@ -1766,7 +1710,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -1781,7 +1724,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -1796,7 +1738,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -1811,7 +1752,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
nativeStructuredOutputs: true, nativeStructuredOutputs: true,
maxOutputTokens: { max: 64000, default: 8192 }, maxOutputTokens: { max: 64000, default: 8192 },
vision: true,
}, },
contextWindow: 200000, contextWindow: 200000,
}, },
@@ -1824,7 +1764,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -1837,7 +1776,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -1850,7 +1788,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -1863,7 +1800,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 300000, contextWindow: 300000,
}, },
@@ -1876,7 +1812,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 300000, contextWindow: 300000,
}, },
@@ -1901,7 +1836,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 1000000, contextWindow: 1000000,
}, },
@@ -1914,7 +1848,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 3500000, contextWindow: 3500000,
}, },
@@ -1939,7 +1872,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -1952,7 +1884,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2025,7 +1956,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2062,7 +1992,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2087,7 +2016,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2100,7 +2028,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2113,7 +2040,6 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
}, },
capabilities: { capabilities: {
temperature: { min: 0, max: 1 }, temperature: { min: 0, max: 1 },
vision: true,
}, },
contextWindow: 128000, contextWindow: 128000,
}, },
@@ -2285,32 +2211,6 @@ export function getMaxTemperature(modelId: string): number | undefined {
return capabilities?.temperature?.max return capabilities?.temperature?.max
} }
/**
* Checks if a model supports vision/multimodal inputs (images, audio, video, PDFs)
*/
export function supportsVision(modelId: string): boolean {
const capabilities = getModelCapabilities(modelId)
return !!capabilities?.vision
}
/**
* Returns a list of all vision-capable models
*/
export function getVisionModels(): string[] {
const models: string[] = []
for (const provider of Object.values(PROVIDER_DEFINITIONS)) {
// Check if the provider has vision capability at the provider level
const providerHasVision = provider.capabilities?.vision
for (const model of provider.models) {
// Model has vision if either the model or provider has vision capability
if (model.capabilities.vision || providerHasVision) {
models.push(model.id)
}
}
}
return models
}
export function supportsToolUsageControl(providerId: string): boolean { export function supportsToolUsageControl(providerId: string): boolean {
return getProvidersWithToolUsageControl().includes(providerId) return getProvidersWithToolUsageControl().includes(providerId)
} }

View File

@@ -111,25 +111,9 @@ export interface ProviderToolConfig {
usageControl?: ToolUsageControl usageControl?: ToolUsageControl
} }
/**
* Attachment content (files, images, documents)
*/
export interface AttachmentContent {
/** Source type: how the data was provided */
sourceType: 'url' | 'base64' | 'file'
/** The URL or base64 data */
data: string
/** MIME type (e.g., 'image/png', 'application/pdf', 'audio/mp3') */
mimeType?: string
/** Optional filename for file uploads */
fileName?: string
}
export interface Message { export interface Message {
role: 'system' | 'user' | 'assistant' | 'function' | 'tool' | 'attachment' role: 'system' | 'user' | 'assistant' | 'function' | 'tool'
content: string | null content: string | null
/** Attachment content for 'attachment' role messages */
attachment?: AttachmentContent
name?: string name?: string
function_call?: { function_call?: {
name: string name: string

View File

@@ -23,11 +23,9 @@ import {
getReasoningEffortValuesForModel as getReasoningEffortValuesForModelFromDefinitions, getReasoningEffortValuesForModel as getReasoningEffortValuesForModelFromDefinitions,
getThinkingLevelsForModel as getThinkingLevelsForModelFromDefinitions, getThinkingLevelsForModel as getThinkingLevelsForModelFromDefinitions,
getVerbosityValuesForModel as getVerbosityValuesForModelFromDefinitions, getVerbosityValuesForModel as getVerbosityValuesForModelFromDefinitions,
getVisionModels,
PROVIDER_DEFINITIONS, PROVIDER_DEFINITIONS,
supportsTemperature as supportsTemperatureFromDefinitions, supportsTemperature as supportsTemperatureFromDefinitions,
supportsToolUsageControl as supportsToolUsageControlFromDefinitions, supportsToolUsageControl as supportsToolUsageControlFromDefinitions,
supportsVision,
updateOllamaModels as updateOllamaModelsInDefinitions, updateOllamaModels as updateOllamaModelsInDefinitions,
} from '@/providers/models' } from '@/providers/models'
import type { ProviderId, ProviderToolConfig } from '@/providers/types' import type { ProviderId, ProviderToolConfig } from '@/providers/types'
@@ -1154,6 +1152,3 @@ export function checkForForcedToolUsageOpenAI(
return { hasUsedForcedTool, usedForcedTools: updatedUsedForcedTools } return { hasUsedForcedTool, usedForcedTools: updatedUsedForcedTools }
} }
// Re-export vision capability functions
export { supportsVision, getVisionModels }