github/sim
1
1
Fork 0
mirror of https://github.com/simstudioai/sim.git synced 2026-04-06 03:00:16 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(connectors): add Fireflies connector and API key auth support (#3448)

Browse Source 
* feat(connectors): add Fireflies connector and API key auth support

Extend the connector system to support both OAuth and API key authentication
via a discriminated union (`ConnectorAuthConfig`). Add Fireflies as the first
API key connector, syncing meeting transcripts via the Fireflies GraphQL API.

Schema changes:
- Make `credentialId` nullable (null for API key connectors)
- Add `encryptedApiKey` column (AES-256-GCM encrypted, null for OAuth)

This eliminates the `'_apikey_'` sentinel and inline `sourceConfig._encryptedApiKey`
patterns, giving each auth mode its own clean column.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(fireflies): allow 0 for maxTranscripts (means unlimited)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Waleed
2026-03-06 16:48:39 -08:00
committed by GitHub
parent 1e53d5748a
commit 96c2ae2c39
37 changed files with 14017 additions and 1155 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
.claude/commands/add-connector.md
View File

@@ -11,8 +11,9 @@ You are an expert at adding knowledge base connectors to Sim. A connector syncs
When the user asks you to create a connector:
1. Use Context7 or WebFetch to read the service's API documentation
2. Create the connector directory and config
3. Register it in the connector registry
2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth)
3. Create the connector directory and config
4. Register it in the connector registry
## Directory Structure
@@ -23,8 +24,26 @@ connectors/{service}/
└── {service}.ts # ConnectorConfig definition
```
## Authentication
Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`):
```typescript
type ConnectorAuthConfig =
| { mode: 'oauth'; provider: OAuthService; requiredScopes?: string[] }
| { mode: 'apiKey'; label?: string; placeholder?: string }
```
### OAuth mode
For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The `provider` must match an `OAuthService`. The modal shows a credential picker and handles token refresh automatically.
### API key mode
For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`.
## ConnectorConfig Structure
### OAuth connector example
```typescript
import { createLogger } from '@sim/logger'
import { {Service}Icon } from '@/components/icons'
@@ -40,8 +59,8 @@ export const {service}Connector: ConnectorConfig = {
version: '1.0.0',
icon: {Service}Icon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: '{service}', // Must match OAuthService in lib/oauth/types.ts
requiredScopes: ['read:...'],
},
@@ -71,6 +90,29 @@ export const {service}Connector: ConnectorConfig = {
}
```
### API key connector example
```typescript
export const {service}Connector: ConnectorConfig = {
id: '{service}',
name: '{Service}',
description: 'Sync documents from {Service} into your knowledge base',
version: '1.0.0',
icon: {Service}Icon,
auth: {
mode: 'apiKey',
label: 'API Key', // Shown above the input field
placeholder: 'Enter your {Service} API key', // Input placeholder
},
configFields: [ /* ... */ ],
listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
}
```
## ConfigField Types
The add-connector modal renders these automatically — no custom UI needed.
@@ -210,13 +252,10 @@ The sync engine (`lib/knowledge/connectors/sync-engine.ts`) is connector-agnosti
2. Compares `contentHash` to detect new/changed/unchanged documents
3. Stores `sourceUrl` and calls `mapTags` on insert/update automatically
4. Handles soft-delete of removed documents
5. Resolves access tokens automatically — OAuth tokens are refreshed, API keys are decrypted from the `encryptedApiKey` column
You never need to modify the sync engine when adding a connector.
## OAuth Credential Reuse
Connectors reuse the existing OAuth infrastructure. The `oauth.provider` must match an `OAuthService` from `apps/sim/lib/oauth/types.ts`. Check existing providers before adding a new one.
## Icon
The `icon` field on `ConnectorConfig` is used throughout the UI — in the connector list, the add-connector modal, and as the document icon in the knowledge base table (replacing the generic file type icon for connector-sourced documents). The icon is read from `CONNECTOR_REGISTRY[connectorType].icon` at runtime — no separate icon map to maintain.
@@ -236,19 +275,18 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
}
```
## Reference Implementation
## Reference Implementations
See `apps/sim/connectors/confluence/confluence.ts` for a complete example with:
- Multiple config field types (text + dropdown)
- Label fetching and CQL search filtering
- Blogpost + page content types
- `mapTags` mapping labels, version, and dates to semantic keys
- **OAuth**: `apps/sim/connectors/confluence/confluence.ts` — multiple config field types, `mapTags`, label fetching
- **API key**: `apps/sim/connectors/fireflies/fireflies.ts` — GraphQL API with Bearer token auth
## Checklist
- [ ] Created `connectors/{service}/{service}.ts` with full ConnectorConfig
- [ ] Created `connectors/{service}/index.ts` barrel export
- [ ] `oauth.provider` matches an existing OAuthService in `lib/oauth/types.ts`
- [ ] **Auth configured correctly:**
- OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts`
- API key: `auth.label` and `auth.placeholder` set appropriately
- [ ] `listDocuments` handles pagination and computes content hashes
- [ ] `sourceUrl` set on each ExternalDocument (full URL, not relative)
- [ ] `metadata` includes source-specific data for tag mapping

34
apps/sim/app/api/knowledge/[id]/connectors/[connectorId]/route.ts
View File

@@ -9,6 +9,7 @@ import { createLogger } from '@sim/logger'
import { and, desc, eq, isNull } from 'drizzle-orm'
import { type NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { decryptApiKey } from '@/lib/api-key/crypto'
import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { cleanupUnusedTagDefinitions } from '@/lib/knowledge/tags/service'
@@ -68,10 +69,11 @@ export async function GET(request: NextRequest, { params }: RouteParams) {
.orderBy(desc(knowledgeConnectorSyncLog.startedAt))
.limit(10)
const { encryptedApiKey: _, ...connectorData } = connectorRows[0]
return NextResponse.json({
success: true,
data: {
...connectorRows[0],
...connectorData,
syncLogs,
},
})
@@ -146,11 +148,28 @@ export async function PATCH(request: NextRequest, { params }: RouteParams) {
return NextResponse.json({ error: 'Knowledge base not found' }, { status: 404 })
}
const accessToken = await refreshAccessTokenIfNeeded(
existing.credentialId,
kbRows[0].userId,
`patch-${connectorId}`
)
let accessToken: string | null = null
if (connectorConfig.auth.mode === 'apiKey') {
if (!existing.encryptedApiKey) {
return NextResponse.json(
{ error: 'API key not found. Please reconfigure the connector.' },
{ status: 400 }
)
}
accessToken = (await decryptApiKey(existing.encryptedApiKey)).decrypted
} else {
if (!existing.credentialId) {
return NextResponse.json(
{ error: 'OAuth credential not found. Please reconfigure the connector.' },
{ status: 400 }
)
}
accessToken = await refreshAccessTokenIfNeeded(
existing.credentialId,
kbRows[0].userId,
`patch-${connectorId}`
)
}
if (!accessToken) {
return NextResponse.json(
@@ -207,7 +226,8 @@ export async function PATCH(request: NextRequest, { params }: RouteParams) {
)
.limit(1)
return NextResponse.json({ success: true, data: updated[0] })
const { encryptedApiKey: __, ...updatedData } = updated[0]
return NextResponse.json({ success: true, data: updatedData })
} catch (error) {
logger.error(`[${requestId}] Error updating connector`, error)
return NextResponse.json({ error: 'Internal server error' }, { status: 500 })

67
apps/sim/app/api/knowledge/[id]/connectors/route.ts
View File

@@ -4,6 +4,7 @@ import { createLogger } from '@sim/logger'
import { and, desc, eq, isNull } from 'drizzle-orm'
import { type NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { encryptApiKey } from '@/lib/api-key/crypto'
import { checkSessionOrInternalAuth } from '@/lib/auth/hybrid'
import { generateRequestId } from '@/lib/core/utils/request'
import { dispatchSync } from '@/lib/knowledge/connectors/sync-engine'
@@ -17,7 +18,8 @@ const logger = createLogger('KnowledgeConnectorsAPI')
const CreateConnectorSchema = z.object({
connectorType: z.string().min(1),
credentialId: z.string().min(1),
credentialId: z.string().min(1).optional(),
apiKey: z.string().min(1).optional(),
sourceConfig: z.record(z.unknown()),
syncIntervalMinutes: z.number().int().min(0).default(1440),
})
@@ -52,7 +54,10 @@ export async function GET(request: NextRequest, { params }: { params: Promise<{
)
.orderBy(desc(knowledgeConnector.createdAt))
return NextResponse.json({ success: true, data: connectors })
return NextResponse.json({
success: true,
data: connectors.map(({ encryptedApiKey: _, ...rest }) => rest),
})
} catch (error) {
logger.error(`[${requestId}] Error listing connectors`, error)
return NextResponse.json({ error: 'Internal server error' }, { status: 500 })
@@ -87,7 +92,7 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
)
}
const { connectorType, credentialId, sourceConfig, syncIntervalMinutes } = parsed.data
const { connectorType, credentialId, apiKey, sourceConfig, syncIntervalMinutes } = parsed.data
const connectorConfig = CONNECTOR_REGISTRY[connectorType]
if (!connectorConfig) {
@@ -97,19 +102,37 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
)
}
const credential = await getCredential(requestId, credentialId, auth.userId)
if (!credential) {
return NextResponse.json({ error: 'Credential not found' }, { status: 400 })
let resolvedCredentialId: string | null = null
let resolvedEncryptedApiKey: string | null = null
let accessToken: string
if (connectorConfig.auth.mode === 'apiKey') {
if (!apiKey) {
return NextResponse.json({ error: 'API key is required' }, { status: 400 })
}
accessToken = apiKey
} else {
if (!credentialId) {
return NextResponse.json({ error: 'Credential is required' }, { status: 400 })
}
const credential = await getCredential(requestId, credentialId, auth.userId)
if (!credential) {
return NextResponse.json({ error: 'Credential not found' }, { status: 400 })
}
if (!credential.accessToken) {
return NextResponse.json(
{ error: 'Credential has no access token. Please reconnect your account.' },
{ status: 400 }
)
}
accessToken = credential.accessToken
resolvedCredentialId = credentialId
}
if (!credential.accessToken) {
return NextResponse.json(
{ error: 'Credential has no access token. Please reconnect your account.' },
{ status: 400 }
)
}
const validation = await connectorConfig.validateConfig(credential.accessToken, sourceConfig)
const validation = await connectorConfig.validateConfig(accessToken, sourceConfig)
if (!validation.valid) {
return NextResponse.json(
{ error: validation.error || 'Invalid source configuration' },
@@ -117,7 +140,13 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
)
}
let finalSourceConfig: Record<string, unknown> = sourceConfig
let finalSourceConfig: Record<string, unknown> = { ...sourceConfig }
if (connectorConfig.auth.mode === 'apiKey' && apiKey) {
const { encrypted } = await encryptApiKey(apiKey)
resolvedEncryptedApiKey = encrypted
}
const tagSlotMapping: Record<string, string> = {}
if (connectorConfig.tagDefinitions?.length) {
@@ -144,7 +173,7 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
)
}
finalSourceConfig = { ...sourceConfig, tagSlotMapping }
finalSourceConfig = { ...finalSourceConfig, tagSlotMapping }
}
const now = new Date()
@@ -171,7 +200,8 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
id: connectorId,
knowledgeBaseId,
connectorType,
credentialId,
credentialId: resolvedCredentialId,
encryptedApiKey: resolvedEncryptedApiKey,
sourceConfig: finalSourceConfig,
syncIntervalMinutes,
status: 'active',
@@ -196,7 +226,8 @@ export async function POST(request: NextRequest, { params }: { params: Promise<{
.where(eq(knowledgeConnector.id, connectorId))
.limit(1)
return NextResponse.json({ success: true, data: created[0] }, { status: 201 })
const { encryptedApiKey: _, ...createdData } = created[0]
return NextResponse.json({ success: true, data: createdData }, { status: 201 })
} catch (error) {
logger.error(`[${requestId}] Error creating connector`, error)
return NextResponse.json({ error: 'Internal server error' }, { status: 500 })

125
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/add-connector-modal/add-connector-modal.tsx
View File

@@ -1,6 +1,7 @@
'use client'
import { useMemo, useState } from 'react'
import { useParams } from 'next/navigation'
import { ArrowLeft, Loader2, Plus } from 'lucide-react'
import {
Button,
@@ -54,20 +55,24 @@ export function AddConnectorModal({ open, onOpenChange, knowledgeBaseId }: AddCo
const [error, setError] = useState<string | null>(null)
const [showOAuthModal, setShowOAuthModal] = useState(false)
const [apiKeyValue, setApiKeyValue] = useState('')
const { workspaceId } = useParams<{ workspaceId: string }>()
const { mutate: createConnector, isPending: isCreating } = useCreateConnector()
const connectorConfig = selectedType ? CONNECTOR_REGISTRY[selectedType] : null
const isApiKeyMode = connectorConfig?.auth.mode === 'apiKey'
const connectorProviderId = useMemo(
() =>
connectorConfig
? (getProviderIdFromServiceId(connectorConfig.oauth.provider) as OAuthProvider)
connectorConfig && connectorConfig.auth.mode === 'oauth'
? (getProviderIdFromServiceId(connectorConfig.auth.provider) as OAuthProvider)
: null,
[connectorConfig]
)
const { data: credentials = [], isLoading: credentialsLoading } = useOAuthCredentials(
connectorProviderId ?? undefined,
Boolean(connectorConfig)
{ enabled: Boolean(connectorConfig) && !isApiKeyMode, workspaceId }
)
const effectiveCredentialId =
@@ -75,18 +80,28 @@ export function AddConnectorModal({ open, onOpenChange, knowledgeBaseId }: AddCo
const handleSelectType = (type: string) => {
setSelectedType(type)
setSourceConfig({})
setSelectedCredentialId(null)
setApiKeyValue('')
setDisabledTagIds(new Set())
setError(null)
setStep('configure')
}
const canSubmit = useMemo(() => {
if (!connectorConfig || !effectiveCredentialId) return false
if (!connectorConfig) return false
if (isApiKeyMode) {
if (!apiKeyValue.trim()) return false
} else {
if (!effectiveCredentialId) return false
}
return connectorConfig.configFields
.filter((f) => f.required)
.every((f) => sourceConfig[f.id]?.trim())
}, [connectorConfig, effectiveCredentialId, sourceConfig])
}, [connectorConfig, isApiKeyMode, apiKeyValue, effectiveCredentialId, sourceConfig])
const handleSubmit = () => {
if (!selectedType || !effectiveCredentialId || !canSubmit) return
if (!selectedType || !canSubmit) return
setError(null)
@@ -99,7 +114,7 @@ export function AddConnectorModal({ open, onOpenChange, knowledgeBaseId }: AddCo
{
knowledgeBaseId,
connectorType: selectedType,
credentialId: effectiveCredentialId,
...(isApiKeyMode ? { apiKey: apiKeyValue } : { credentialId: effectiveCredentialId! }),
sourceConfig: finalSourceConfig,
syncIntervalMinutes: syncInterval,
},
@@ -149,44 +164,64 @@ export function AddConnectorModal({ open, onOpenChange, knowledgeBaseId }: AddCo
</div>
) : connectorConfig ? (
<div className='flex flex-col gap-[12px]'>
{/* Credential selection */}
<div className='flex flex-col gap-[4px]'>
<Label>Account</Label>
{credentialsLoading ? (
<div className='flex items-center gap-2 text-[13px] text-[var(--text-muted)]'>
<Loader2 className='h-4 w-4 animate-spin' />
Loading credentials...
</div>
) : (
<Combobox
size='sm'
options={[
...credentials.map(
(cred): ComboboxOption => ({
label: cred.name || cred.provider,
value: cred.id,
icon: connectorConfig.icon,
})
),
{
label: 'Connect new account',
value: '__connect_new__',
icon: Plus,
onSelect: () => {
setShowOAuthModal(true)
},
},
]}
value={effectiveCredentialId ?? undefined}
onChange={(value) => setSelectedCredentialId(value)}
{/* Auth: API key input or OAuth credential selection */}
{isApiKeyMode ? (
<div className='flex flex-col gap-[4px]'>
<Label>
{connectorConfig.auth.mode === 'apiKey' && connectorConfig.auth.label
? connectorConfig.auth.label
: 'API Key'}
</Label>
<Input
type='password'
value={apiKeyValue}
onChange={(e) => setApiKeyValue(e.target.value)}
placeholder={
credentials.length === 0
? `No ${connectorConfig.name} accounts`
: 'Select account'
connectorConfig.auth.mode === 'apiKey' && connectorConfig.auth.placeholder
? connectorConfig.auth.placeholder
: 'Enter API key'
}
/>
)}
</div>
</div>
) : (
<div className='flex flex-col gap-[4px]'>
<Label>Account</Label>
{credentialsLoading ? (
<div className='flex items-center gap-2 text-[13px] text-[var(--text-muted)]'>
<Loader2 className='h-4 w-4 animate-spin' />
Loading credentials...
</div>
) : (
<Combobox
size='sm'
options={[
...credentials.map(
(cred): ComboboxOption => ({
label: cred.name || cred.provider,
value: cred.id,
icon: connectorConfig.icon,
})
),
{
label: 'Connect new account',
value: '__connect_new__',
icon: Plus,
onSelect: () => {
setShowOAuthModal(true)
},
},
]}
value={effectiveCredentialId ?? undefined}
onChange={(value) => setSelectedCredentialId(value)}
placeholder={
credentials.length === 0
? `No ${connectorConfig.name} accounts`
: 'Select account'
}
/>
)}
</div>
)}
{/* Config fields */}
{connectorConfig.configFields.map((field) => (
@@ -309,15 +344,15 @@ export function AddConnectorModal({ open, onOpenChange, knowledgeBaseId }: AddCo
)}
</ModalContent>
</Modal>
{connectorConfig && connectorProviderId && (
{connectorConfig && connectorConfig.auth.mode === 'oauth' && connectorProviderId && (
<OAuthRequiredModal
isOpen={showOAuthModal}
onClose={() => setShowOAuthModal(false)}
provider={connectorProviderId}
toolName={connectorConfig.name}
requiredScopes={getCanonicalScopesForProvider(connectorProviderId)}
newScopes={connectorConfig.oauth.requiredScopes || []}
serviceId={connectorConfig.oauth.provider}
newScopes={connectorConfig.auth.requiredScopes || []}
serviceId={connectorConfig.auth.provider}
/>
)}
</>

5
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/connectors-section/connectors-section.tsx
View File

@@ -301,9 +301,10 @@ function ConnectorCard({
const statusConfig =
STATUS_CONFIG[connector.status as keyof typeof STATUS_CONFIG] || STATUS_CONFIG.active
const serviceId = connectorDef?.oauth.provider
const serviceId = connectorDef?.auth.mode === 'oauth' ? connectorDef.auth.provider : undefined
const providerId = serviceId ? getProviderIdFromServiceId(serviceId) : undefined
const requiredScopes = connectorDef?.oauth.requiredScopes ?? []
const requiredScopes =
connectorDef?.auth.mode === 'oauth' ? (connectorDef.auth.requiredScopes ?? []) : []
const { data: credentials } = useOAuthCredentials(providerId, { workspaceId })

4
apps/sim/connectors/airtable/airtable.ts
View File

@@ -78,8 +78,8 @@ export const airtableConnector: ConnectorConfig = {
version: '1.0.0',
icon: AirtableIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'airtable',
requiredScopes: ['data.records:read', 'schema.bases:read'],
},

6
apps/sim/connectors/asana/asana.ts
View File

@@ -136,11 +136,7 @@ export const asanaConnector: ConnectorConfig = {
version: '1.0.0',
icon: AsanaIcon,
oauth: {
required: true,
provider: 'asana',
requiredScopes: ['default'],
},
auth: { mode: 'oauth', provider: 'asana', requiredScopes: ['default'] },
configFields: [
{

4
apps/sim/connectors/confluence/confluence.ts
View File

@@ -110,8 +110,8 @@ export const confluenceConnector: ConnectorConfig = {
version: '1.1.0',
icon: ConfluenceIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'confluence',
requiredScopes: ['read:confluence-content.all', 'read:page:confluence', 'offline_access'],
},

4
apps/sim/connectors/dropbox/dropbox.ts
View File

@@ -117,8 +117,8 @@ export const dropboxConnector: ConnectorConfig = {
version: '1.0.0',
icon: DropboxIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'dropbox',
requiredScopes: ['files.metadata.read', 'files.content.read'],
},

390
apps/sim/connectors/fireflies/fireflies.ts Normal file
View File

@@ -0,0 +1,390 @@
import { createLogger } from '@sim/logger'
import { FirefliesIcon } from '@/components/icons'
import { fetchWithRetry, VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils'
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
import { computeContentHash, parseTagDate } from '@/connectors/utils'
const logger = createLogger('FirefliesConnector')
const FIREFLIES_GRAPHQL_URL = 'https://api.fireflies.ai/graphql'
const TRANSCRIPTS_PER_PAGE = 50
interface FirefliesTranscript {
id: string
title: string
date: number
duration: number
host_email?: string
organizer_email?: string
participants?: string[]
transcript_url?: string
speakers?: { id: string; name: string }[]
sentences?: { index: number; speaker_name: string; text: string }[]
summary?: {
keywords?: string[]
action_items?: string[]
overview?: string
short_summary?: string
}
}
/**
* Executes a GraphQL query against the Fireflies API.
*/
async function firefliesGraphQL(
accessToken: string,
query: string,
variables: Record<string, unknown> = {},
retryOptions?: Parameters<typeof fetchWithRetry>[2]
): Promise<Record<string, unknown>> {
const response = await fetchWithRetry(
FIREFLIES_GRAPHQL_URL,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${accessToken}`,
},
body: JSON.stringify({ query, variables }),
},
retryOptions
)
if (!response.ok) {
throw new Error(`Fireflies API HTTP error: ${response.status}`)
}
const data = await response.json()
if (data.errors) {
const message = (data.errors as { message: string }[])[0]?.message || 'Unknown GraphQL error'
throw new Error(`Fireflies API error: ${message}`)
}
return data.data as Record<string, unknown>
}
/**
* Formats transcript sentences into plain text content.
*/
function formatTranscriptContent(transcript: FirefliesTranscript): string {
const parts: string[] = []
if (transcript.title) {
parts.push(`Meeting: ${transcript.title}`)
}
if (transcript.date) {
parts.push(`Date: ${new Date(transcript.date).toISOString()}`)
}
if (transcript.duration) {
const minutes = Math.round(transcript.duration / 60)
parts.push(`Duration: ${minutes} minutes`)
}
if (transcript.host_email) {
parts.push(`Host: ${transcript.host_email}`)
}
if (transcript.participants && transcript.participants.length > 0) {
parts.push(`Participants: ${transcript.participants.join(', ')}`)
}
if (transcript.summary?.overview) {
parts.push('')
parts.push('--- Overview ---')
parts.push(transcript.summary.overview)
}
if (transcript.summary?.action_items && transcript.summary.action_items.length > 0) {
parts.push('')
parts.push('--- Action Items ---')
for (const item of transcript.summary.action_items) {
parts.push(`- ${item}`)
}
}
if (transcript.summary?.keywords && transcript.summary.keywords.length > 0) {
parts.push('')
parts.push(`Keywords: ${transcript.summary.keywords.join(', ')}`)
}
if (transcript.sentences && transcript.sentences.length > 0) {
parts.push('')
parts.push('--- Transcript ---')
for (const sentence of transcript.sentences) {
parts.push(`${sentence.speaker_name}: ${sentence.text}`)
}
}
return parts.join('\n')
}
export const firefliesConnector: ConnectorConfig = {
id: 'fireflies',
name: 'Fireflies',
description: 'Sync meeting transcripts from Fireflies.ai into your knowledge base',
version: '1.0.0',
icon: FirefliesIcon,
auth: {
mode: 'apiKey',
label: 'API Key',
placeholder: 'Enter your Fireflies API key',
},
configFields: [
{
id: 'hostEmail',
title: 'Filter by Host Email',
type: 'short-input',
placeholder: 'e.g. john@example.com',
required: false,
description: 'Only sync transcripts hosted by this email',
},
{
id: 'maxTranscripts',
title: 'Max Transcripts',
type: 'short-input',
required: false,
placeholder: 'e.g. 100 (default: unlimited)',
},
],
listDocuments: async (
accessToken: string,
sourceConfig: Record<string, unknown>,
cursor?: string,
syncContext?: Record<string, unknown>
): Promise<ExternalDocumentList> => {
const hostEmail = (sourceConfig.hostEmail as string) || ''
const maxTranscripts = sourceConfig.maxTranscripts ? Number(sourceConfig.maxTranscripts) : 0
const skip = cursor ? Number(cursor) : 0
const variables: Record<string, unknown> = {
limit: TRANSCRIPTS_PER_PAGE,
skip,
}
if (hostEmail.trim()) {
variables.host_email = hostEmail.trim()
}
logger.info('Listing Fireflies transcripts', { skip, limit: TRANSCRIPTS_PER_PAGE, hostEmail })
const data = await firefliesGraphQL(
accessToken,
`query Transcripts(
$limit: Int
$skip: Int
$host_email: String
) {
transcripts(
limit: $limit
skip: $skip
host_email: $host_email
) {
id
title
date
duration
host_email
organizer_email
participants
transcript_url
speakers {
id
name
}
sentences {
index
speaker_name
text
}
summary {
keywords
action_items
overview
short_summary
}
}
}`,
variables
)
const transcripts = (data.transcripts || []) as FirefliesTranscript[]
const documents: ExternalDocument[] = await Promise.all(
transcripts.map(async (transcript) => {
const content = formatTranscriptContent(transcript)
const contentHash = await computeContentHash(content)
const meetingDate = transcript.date ? new Date(transcript.date).toISOString() : undefined
const speakerNames = transcript.speakers?.map((s) => s.name).filter(Boolean) ?? []
return {
externalId: transcript.id,
title: transcript.title || 'Untitled Meeting',
content,
mimeType: 'text/plain' as const,
sourceUrl: transcript.transcript_url || undefined,
contentHash,
metadata: {
hostEmail: transcript.host_email,
duration: transcript.duration,
meetingDate,
participants: transcript.participants,
speakers: speakerNames,
keywords: transcript.summary?.keywords,
},
}
})
)
const totalFetched = ((syncContext?.totalDocsFetched as number) ?? 0) + documents.length
if (syncContext) syncContext.totalDocsFetched = totalFetched
const hitLimit = maxTranscripts > 0 && totalFetched >= maxTranscripts
const hasMore = !hitLimit && transcripts.length === TRANSCRIPTS_PER_PAGE
return {
documents,
nextCursor: hasMore ? String(skip + transcripts.length) : undefined,
hasMore,
}
},
getDocument: async (
accessToken: string,
_sourceConfig: Record<string, unknown>,
externalId: string
): Promise<ExternalDocument | null> => {
try {
const data = await firefliesGraphQL(
accessToken,
`query Transcript($id: String!) {
transcript(id: $id) {
id
title
date
duration
host_email
organizer_email
participants
transcript_url
speakers {
id
name
}
sentences {
index
speaker_name
text
}
summary {
keywords
action_items
overview
short_summary
}
}
}`,
{ id: externalId }
)
const transcript = data.transcript as FirefliesTranscript | null
if (!transcript) return null
const content = formatTranscriptContent(transcript)
const contentHash = await computeContentHash(content)
const meetingDate = transcript.date ? new Date(transcript.date).toISOString() : undefined
const speakerNames = transcript.speakers?.map((s) => s.name).filter(Boolean) ?? []
return {
externalId: transcript.id,
title: transcript.title || 'Untitled Meeting',
content,
mimeType: 'text/plain',
sourceUrl: transcript.transcript_url || undefined,
contentHash,
metadata: {
hostEmail: transcript.host_email,
duration: transcript.duration,
meetingDate,
participants: transcript.participants,
speakers: speakerNames,
keywords: transcript.summary?.keywords,
},
}
} catch (error) {
logger.warn('Failed to get Fireflies transcript', {
externalId,
error: error instanceof Error ? error.message : String(error),
})
return null
}
},
validateConfig: async (
accessToken: string,
sourceConfig: Record<string, unknown>
): Promise<{ valid: boolean; error?: string }> => {
const maxTranscripts = sourceConfig.maxTranscripts as string | undefined
if (maxTranscripts && (Number.isNaN(Number(maxTranscripts)) || Number(maxTranscripts) < 0)) {
return { valid: false, error: 'Max transcripts must be a non-negative number' }
}
try {
await firefliesGraphQL(
accessToken,
`query User {
user {
user_id
name
email
}
}`,
{},
VALIDATE_RETRY_OPTIONS
)
return { valid: true }
} catch (error) {
const message = error instanceof Error ? error.message : 'Failed to validate configuration'
return { valid: false, error: message }
}
},
tagDefinitions: [
{ id: 'hostEmail', displayName: 'Host Email', fieldType: 'text' },
{ id: 'speakers', displayName: 'Speakers', fieldType: 'text' },
{ id: 'duration', displayName: 'Duration (seconds)', fieldType: 'number' },
{ id: 'meetingDate', displayName: 'Meeting Date', fieldType: 'date' },
],
mapTags: (metadata: Record<string, unknown>): Record<string, unknown> => {
const result: Record<string, unknown> = {}
if (typeof metadata.hostEmail === 'string') {
result.hostEmail = metadata.hostEmail
}
const speakers = Array.isArray(metadata.speakers) ? (metadata.speakers as string[]) : []
if (speakers.length > 0) {
result.speakers = speakers.join(', ')
}
if (metadata.duration != null) {
const num = Number(metadata.duration)
if (!Number.isNaN(num)) result.duration = num
}
const meetingDate = parseTagDate(metadata.meetingDate)
if (meetingDate) result.meetingDate = meetingDate
return result
},
}

1
apps/sim/connectors/fireflies/index.ts Normal file
View File

@@ -0,0 +1 @@
export { firefliesConnector } from '@/connectors/fireflies/fireflies'

6
apps/sim/connectors/github/github.ts
View File

@@ -158,11 +158,7 @@ export const githubConnector: ConnectorConfig = {
version: '1.0.0',
icon: GithubIcon,
oauth: {
required: true,
provider: 'github',
requiredScopes: ['repo'],
},
auth: { mode: 'oauth', provider: 'github', requiredScopes: ['repo'] },
configFields: [
{

4
apps/sim/connectors/google-docs/google-docs.ts
View File

@@ -175,8 +175,8 @@ export const googleDocsConnector: ConnectorConfig = {
version: '1.0.0',
icon: GoogleDocsIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'google-docs',
requiredScopes: ['https://www.googleapis.com/auth/drive'],
},

4
apps/sim/connectors/google-drive/google-drive.ts
View File

@@ -187,8 +187,8 @@ export const googleDriveConnector: ConnectorConfig = {
version: '1.0.0',
icon: GoogleDriveIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'google-drive',
requiredScopes: ['https://www.googleapis.com/auth/drive'],
},

4
apps/sim/connectors/hubspot/hubspot.ts
View File

@@ -178,8 +178,8 @@ export const hubspotConnector: ConnectorConfig = {
version: '1.0.0',
icon: HubspotIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'hubspot',
requiredScopes: [
'crm.objects.contacts.read',

6
apps/sim/connectors/jira/jira.ts
View File

@@ -81,11 +81,7 @@ export const jiraConnector: ConnectorConfig = {
version: '1.0.0',
icon: JiraIcon,
oauth: {
required: true,
provider: 'jira',
requiredScopes: ['read:jira-work', 'offline_access'],
},
auth: { mode: 'oauth', provider: 'jira', requiredScopes: ['read:jira-work', 'offline_access'] },
configFields: [
{

6
apps/sim/connectors/linear/linear.ts
View File

@@ -190,11 +190,7 @@ export const linearConnector: ConnectorConfig = {
version: '1.0.0',
icon: LinearIcon,
oauth: {
required: true,
provider: 'linear',
requiredScopes: ['read'],
},
auth: { mode: 'oauth', provider: 'linear', requiredScopes: ['read'] },
configFields: [
{

6
apps/sim/connectors/notion/notion.ts
View File

@@ -177,11 +177,7 @@ export const notionConnector: ConnectorConfig = {
version: '1.0.0',
icon: NotionIcon,
oauth: {
required: true,
provider: 'notion',
requiredScopes: [],
},
auth: { mode: 'oauth', provider: 'notion', requiredScopes: [] },
configFields: [
{

6
apps/sim/connectors/onedrive/onedrive.ts
View File

@@ -156,11 +156,7 @@ export const onedriveConnector: ConnectorConfig = {
version: '1.0.0',
icon: MicrosoftOneDriveIcon,
oauth: {
required: true,
provider: 'onedrive',
requiredScopes: ['Files.Read'],
},
auth: { mode: 'oauth', provider: 'onedrive', requiredScopes: ['Files.Read'] },
configFields: [
{

2
apps/sim/connectors/registry.ts
View File

@@ -2,6 +2,7 @@ import { airtableConnector } from '@/connectors/airtable'
import { asanaConnector } from '@/connectors/asana'
import { confluenceConnector } from '@/connectors/confluence'
import { dropboxConnector } from '@/connectors/dropbox'
import { firefliesConnector } from '@/connectors/fireflies'
import { githubConnector } from '@/connectors/github'
import { googleDocsConnector } from '@/connectors/google-docs'
import { googleDriveConnector } from '@/connectors/google-drive'
@@ -22,6 +23,7 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
asana: asanaConnector,
confluence: confluenceConnector,
dropbox: dropboxConnector,
fireflies: firefliesConnector,
github: githubConnector,
google_docs: googleDocsConnector,
google_drive: googleDriveConnector,

6
apps/sim/connectors/salesforce/salesforce.ts
View File

@@ -185,11 +185,7 @@ export const salesforceConnector: ConnectorConfig = {
version: '1.0.0',
icon: SalesforceIcon,
oauth: {
required: true,
provider: 'salesforce',
requiredScopes: ['api', 'refresh_token'],
},
auth: { mode: 'oauth', provider: 'salesforce', requiredScopes: ['api', 'refresh_token'] },
configFields: [
{

6
apps/sim/connectors/sharepoint/sharepoint.ts
View File

@@ -297,11 +297,7 @@ export const sharepointConnector: ConnectorConfig = {
version: '1.0.0',
icon: MicrosoftSharepointIcon,
oauth: {
required: true,
provider: 'sharepoint',
requiredScopes: ['Sites.Read.All'],
},
auth: { mode: 'oauth', provider: 'sharepoint', requiredScopes: ['Sites.Read.All'] },
configFields: [
{

4
apps/sim/connectors/slack/slack.ts
View File

@@ -245,8 +245,8 @@ export const slackConnector: ConnectorConfig = {
version: '1.0.0',
icon: SlackIcon,
oauth: {
required: true,
auth: {
mode: 'oauth',
provider: 'slack',
requiredScopes: ['channels:read', 'channels:history', 'users:read'],
},

17
apps/sim/connectors/types.ts
View File

@@ -1,5 +1,14 @@
import type { OAuthService } from '@/lib/oauth/types'
/**
* Authentication configuration for a connector.
* OAuth connectors reuse the existing credential system.
* API key connectors store an encrypted key in the `encryptedApiKey` column.
*/
export type ConnectorAuthConfig =
| { mode: 'oauth'; provider: OAuthService; requiredScopes?: string[] }
| { mode: 'apiKey'; label?: string; placeholder?: string }
/**
* A single document fetched from an external source.
*/
@@ -74,12 +83,8 @@ export interface ConnectorConfig {
/** Icon component for the connector */
icon: React.ComponentType<{ className?: string }>
/** OAuth configuration (same pattern as ToolConfig.oauth) */
oauth: {
required: true
provider: OAuthService
requiredScopes?: string[]
}
/** Authentication configuration */
auth: ConnectorAuthConfig
/** Source configuration fields rendered in the add-connector UI */
configFields: ConnectorConfigField[]

6
apps/sim/connectors/webflow/webflow.ts
View File

@@ -84,11 +84,7 @@ export const webflowConnector: ConnectorConfig = {
version: '1.0.0',
icon: WebflowIcon,
oauth: {
required: true,
provider: 'webflow',
requiredScopes: ['sites:read', 'cms:read'],
},
auth: { mode: 'oauth', provider: 'webflow', requiredScopes: ['sites:read', 'cms:read'] },
configFields: [
{

6
apps/sim/connectors/wordpress/wordpress.ts
View File

@@ -96,11 +96,7 @@ export const wordpressConnector: ConnectorConfig = {
version: '1.0.0',
icon: WordpressIcon,
oauth: {
required: true,
provider: 'wordpress',
requiredScopes: ['global'],
},
auth: { mode: 'oauth', provider: 'wordpress', requiredScopes: ['global'] },
configFields: [
{

5
apps/sim/hooks/queries/kb/connectors.ts
View File

@@ -8,7 +8,7 @@ export interface ConnectorData {
id: string
knowledgeBaseId: string
connectorType: string
credentialId: string
credentialId: string | null
sourceConfig: Record<string, unknown>
syncMode: string
syncIntervalMinutes: number
@@ -109,7 +109,8 @@ export function useConnectorDetail(knowledgeBaseId?: string, connectorId?: strin
export interface CreateConnectorParams {
knowledgeBaseId: string
connectorType: string
credentialId: string
credentialId?: string
apiKey?: string
sourceConfig: Record<string, unknown>
syncIntervalMinutes?: number
}

12
apps/sim/lib/copilot/tools/server/knowledge/knowledge-base.ts
View File

@@ -555,21 +555,27 @@ export const knowledgeBaseServerTool: BaseServerTool<KnowledgeBaseArgs, Knowledg
if (!args.connectorType) {
return { success: false, message: 'connectorType is required for add_connector' }
}
if (!args.credentialId) {
if (!args.credentialId && !args.apiKey) {
return {
success: false,
message:
'credentialId is required for add_connector. Read environment/credentials.json to find credential IDs.',
'Either credentialId (for OAuth connectors) or apiKey (for API key connectors) is required for add_connector.',
}
}
const createBody: Record<string, unknown> = {
connectorType: args.connectorType,
credentialId: args.credentialId,
sourceConfig: args.sourceConfig ?? {},
syncIntervalMinutes: args.syncIntervalMinutes ?? 1440,
}
if (args.credentialId) {
createBody.credentialId = args.credentialId
}
if (args.apiKey) {
createBody.apiKey = args.apiKey
}
if (args.disabledTagIds?.length) {
;(createBody.sourceConfig as Record<string, unknown>).disabledTagIds =
args.disabledTagIds

4
apps/sim/lib/copilot/tools/shared/schemas.ts
View File

@@ -70,8 +70,10 @@ export const KnowledgeBaseArgsSchema = z.object({
tagFieldType: z.enum(['text', 'number', 'date', 'boolean']).optional(),
/** Connector type from registry, e.g. "confluence" (required for add_connector) */
connectorType: z.string().optional(),
/** OAuth credential ID from environment/credentials.json (required for add_connector) */
/** OAuth credential ID from environment/credentials.json (required for OAuth connectors) */
credentialId: z.string().optional(),
/** API key for API key-based connectors (required for API key connectors) */
apiKey: z.string().optional(),
/** Connector-specific config matching the schema in knowledgebases/connectors/{type}.json */
sourceConfig: z.record(z.unknown()).optional(),
/** Sync interval: 60, 360, 1440, 10080, or 0 for manual only (optional for add_connector, defaults to 1440) */

12
apps/sim/lib/copilot/vfs/serializers.ts
View File

@@ -192,7 +192,7 @@ interface SerializableConnectorConfig {
name: string
description: string
version: string
oauth: { provider: string; requiredScopes?: string[] }
auth: { mode: string; provider?: string; requiredScopes?: string[] }
configFields: SerializableConfigField[]
tagDefinitions?: SerializableTagDef[]
supportsIncrementalSync?: boolean
@@ -209,10 +209,7 @@ export function serializeConnectorSchema(connector: SerializableConnectorConfig)
name: connector.name,
description: connector.description,
version: connector.version,
oauth: {
provider: connector.oauth.provider,
requiredScopes: connector.oauth.requiredScopes ?? [],
},
auth: connector.auth,
configFields: connector.configFields.map((f) => {
const field: Record<string, unknown> = {
id: f.id,
@@ -241,8 +238,9 @@ export function serializeConnectorSchema(connector: SerializableConnectorConfig)
*/
export function serializeConnectorOverview(connectors: SerializableConnectorConfig[]): string {
const rows = connectors.map((c) => {
const scopes = c.oauth.requiredScopes?.length ? c.oauth.requiredScopes.join(', ') : '(none)'
return `| ${c.id} | ${c.name} | ${c.oauth.provider} | ${scopes} |`
const provider = c.auth.provider ?? c.auth.mode
const scopes = c.auth.requiredScopes?.length ? c.auth.requiredScopes.join(', ') : '(none)'
return `| ${c.id} | ${c.name} | ${provider} | ${scopes} |`
})
return [

7
apps/sim/lib/copilot/vfs/workspace-vfs.ts
View File

@@ -197,7 +197,7 @@ function getStaticComponentFiles(): Map<string, string> {
name: c.name,
description: c.description,
version: c.version,
oauth: { provider: c.oauth.provider, requiredScopes: c.oauth.requiredScopes },
auth: c.auth,
configFields: c.configFields,
tagDefinitions: c.tagDefinitions,
supportsIncrementalSync: c.supportsIncrementalSync,
@@ -1091,10 +1091,7 @@ export class WorkspaceVFS {
.limit(5)
if (execRows.length > 0) {
this.files.set(
`jobs/${safeName}/executions.json`,
serializeRecentExecutions(execRows)
)
this.files.set(`jobs/${safeName}/executions.json`, serializeRecentExecutions(execRows))
}
} catch (err) {
logger.warn('Failed to load job execution logs', {

46
apps/sim/lib/knowledge/connectors/sync-engine.ts
View File

@@ -7,6 +7,7 @@ import {
} from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { and, eq, inArray, isNull, ne } from 'drizzle-orm'
import { decryptApiKey } from '@/lib/api-key/crypto'
import { getInternalApiBaseUrl } from '@/lib/core/utils/urls'
import { isTriggerAvailable, processDocumentAsync } from '@/lib/knowledge/documents/service'
import { StorageService } from '@/lib/uploads'
@@ -14,7 +15,12 @@ import { deleteFile } from '@/lib/uploads/core/storage-service'
import { refreshAccessTokenIfNeeded } from '@/app/api/auth/oauth/utils'
import { knowledgeConnectorSync } from '@/background/knowledge-connector-sync'
import { CONNECTOR_REGISTRY } from '@/connectors/registry'
import type { DocumentTags, ExternalDocument, SyncResult } from '@/connectors/types'
import type {
ConnectorAuthConfig,
DocumentTags,
ExternalDocument,
SyncResult,
} from '@/connectors/types'
const logger = createLogger('ConnectorSyncEngine')
@@ -70,6 +76,35 @@ export async function dispatchSync(
}
}
/**
* Resolves an access token for a connector based on its auth mode.
* OAuth connectors refresh via the credential system; API key connectors
* decrypt the key stored in the dedicated `encryptedApiKey` column.
*/
async function resolveAccessToken(
connector: { credentialId: string | null; encryptedApiKey: string | null },
connectorConfig: { auth: ConnectorAuthConfig },
userId: string
): Promise<string | null> {
if (connectorConfig.auth.mode === 'apiKey') {
if (!connector.encryptedApiKey) {
throw new Error('API key connector is missing encrypted API key')
}
const { decrypted } = await decryptApiKey(connector.encryptedApiKey)
return decrypted
}
if (!connector.credentialId) {
throw new Error('OAuth connector is missing credential ID')
}
return refreshAccessTokenIfNeeded(
connector.credentialId,
userId,
`sync-${connector.credentialId}`
)
}
/**
* Execute a sync for a given knowledge connector.
*
@@ -116,12 +151,9 @@ export async function executeSync(
}
const userId = kbRows[0].userId
const sourceConfig = connector.sourceConfig as Record<string, unknown>
const accessToken = await refreshAccessTokenIfNeeded(
connector.credentialId,
userId,
`sync-${connectorId}`
)
const accessToken = await resolveAccessToken(connector, connectorConfig, userId)
if (!accessToken) {
throw new Error('Failed to obtain access token')
@@ -146,8 +178,6 @@ export async function executeSync(
startedAt: new Date(),
})
const sourceConfig = connector.sourceConfig as Record<string, unknown>
try {
const externalDocs: ExternalDocument[] = []
let cursor: string | undefined

2
packages/db/migrations/0165_short_thunderbird.sql Normal file
View File

@@ -0,0 +1,2 @@
ALTER TABLE "knowledge_connector" ALTER COLUMN "credential_id" DROP NOT NULL;--> statement-breakpoint
ALTER TABLE "knowledge_connector" ADD COLUMN "encrypted_api_key" text;

1284
packages/db/migrations/meta/0164_snapshot.json
View File

File diff suppressed because it is too large Load Diff

12991
packages/db/migrations/meta/0165_snapshot.json Normal file
View File

File diff suppressed because it is too large Load Diff

9
packages/db/migrations/meta/_journal.json
View File

@@ -1149,6 +1149,13 @@
"when": 1772824148549,
"tag": "0164_mean_roulette",
"breakpoints": true
},
{
"idx": 165,
"version": "7",
"when": 1772842945935,
"tag": "0165_short_thunderbird",
"breakpoints": true
}
]
}
}

3
packages/db/schema.ts
View File

@@ -2455,7 +2455,8 @@ export const knowledgeConnector = pgTable(
.notNull()
.references(() => knowledgeBase.id, { onDelete: 'cascade' }),
connectorType: text('connector_type').notNull(),
credentialId: text('credential_id').notNull(),
credentialId: text('credential_id'),
encryptedApiKey: text('encrypted_api_key'),
sourceConfig: json('source_config').notNull(),
syncMode: text('sync_mode').notNull().default('full'),
syncIntervalMinutes: integer('sync_interval_minutes').notNull().default(1440),