github/sim
1
1
Fork 0
mirror of https://github.com/simstudioai/sim.git synced 2026-04-06 03:00:16 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #692 from simstudioai/staging

Browse Source 
v0.2.15: fix + improvement
This commit is contained in:
Vikhyath Mondreti
2025-07-15 04:11:49 -07:00
committed by GitHub
parent 39444afcdc 0c5e70fc23
commit b223e45de9
26 changed files with 1028 additions and 442 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
apps/docs/content/docs/blocks/agent.mdx
View File

@@ -81,22 +81,16 @@ Control the creativity and randomness of responses:
<Tabs items={['Low (0-0.3)', 'Medium (0.3-0.7)', 'High (0.7-2.0)']}>
<Tab>
<p>
More deterministic, focused responses. Best for factual tasks, customer support, and
situations where accuracy is critical.
</p>
More deterministic, focused responses. Best for factual tasks, customer support, and
situations where accuracy is critical.
</Tab>
<Tab>
<p>
Balanced creativity and focus. Suitable for general purpose applications that require both
accuracy and some creativity.
</p>
Balanced creativity and focus. Suitable for general purpose applications that require both
accuracy and some creativity.
</Tab>
<Tab>
<p>
More creative, varied responses. Ideal for creative writing, brainstorming, and generating
diverse ideas.
</p>
More creative, varied responses. Ideal for creative writing, brainstorming, and generating
diverse ideas.
</Tab>
</Tabs>

175
apps/docs/content/docs/blocks/loop.mdx Normal file
View File

@@ -0,0 +1,175 @@
---
title: Loop
description: Create iterative workflows with loops that execute blocks repeatedly
---
import { Callout } from 'fumadocs-ui/components/callout'
import { Step, Steps } from 'fumadocs-ui/components/steps'
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
import { ThemeImage } from '@/components/ui/theme-image'
The Loop block is a container block in Sim Studio that allows you to execute a group of blocks repeatedly. Loops enable iterative processing in your workflows.
<ThemeImage
lightSrc="/static/light/loop-light.png"
darkSrc="/static/dark/loop-dark.png"
alt="Loop Block"
width={300}
height={175}
/>
<Callout type="info">
Loop blocks are container nodes that can hold other blocks inside them. The blocks inside a loop will execute multiple times based on your configuration.
</Callout>
## Overview
The Loop block enables you to:
<Steps>
<Step>
<strong>Iterate over collections</strong>: Process arrays or objects one item at a time
</Step>
<Step>
<strong>Repeat operations</strong>: Execute blocks a fixed number of times
</Step>
</Steps>
## Configuration Options
### Loop Type
Choose between two types of loops:
<Tabs items={['For Loop', 'ForEach Loop']}>
<Tab>
A numeric loop that executes a fixed number of times. Use this when you need to repeat an operation a specific number of times.
```
Example: Run 5 times
- Iteration 1
- Iteration 2
- Iteration 3
- Iteration 4
- Iteration 5
```
</Tab>
<Tab>
A collection-based loop that iterates over each item in an array or object. Use this when you need to process a collection of items.
```
Example: Process ["apple", "banana", "orange"]
- Iteration 1: Process "apple"
- Iteration 2: Process "banana"
- Iteration 3: Process "orange"
```
</Tab>
</Tabs>
## How to Use Loops
### Creating a Loop
1. Drag a Loop block from the toolbar onto your canvas
2. Configure the loop type and parameters
3. Drag other blocks inside the loop container
4. Connect the blocks as needed
### Accessing Results
After a loop completes, you can access aggregated results:
- **`<loop.results>`**: Array of results from all loop iterations
## Example Use Cases
### Processing API Results
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Scenario: Process multiple customer records</h4>
<ol className="list-decimal pl-5 text-sm">
<li>API block fetches customer list</li>
<li>ForEach loop iterates over each customer</li>
<li>Inside loop: Agent analyzes customer data</li>
<li>Inside loop: Function stores analysis results</li>
</ol>
</div>
### Iterative Content Generation
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Scenario: Generate multiple variations</h4>
<ol className="list-decimal pl-5 text-sm">
<li>Set For loop to 5 iterations</li>
<li>Inside loop: Agent generates content variation</li>
<li>Inside loop: Evaluator scores the content</li>
<li>After loop: Function selects best variation</li>
</ol>
</div>
## Advanced Features
### Limitations
<Callout type="warning">
Container blocks (Loops and Parallels) cannot be nested inside each other. This means:
- You cannot place a Loop block inside another Loop block
- You cannot place a Parallel block inside a Loop block
- You cannot place any container block inside another container block
If you need multi-dimensional iteration, consider restructuring your workflow to use sequential loops or process data in stages.
</Callout>
<Callout type="info">
Loops execute sequentially, not in parallel. If you need concurrent execution, use the Parallel block instead.
</Callout>
## Inputs and Outputs
<Tabs items={['Configuration', 'Variables', 'Results']}>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>Loop Type</strong>: Choose between 'for' or 'forEach'
</li>
<li>
<strong>Iterations</strong>: Number of times to execute (for loops)
</li>
<li>
<strong>Collection</strong>: Array or object to iterate over (forEach loops)
</li>
</ul>
</Tab>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>loop.currentItem</strong>: Current item being processed
</li>
<li>
<strong>loop.index</strong>: Current iteration number (0-based)
</li>
<li>
<strong>loop.items</strong>: Full collection (forEach loops)
</li>
</ul>
</Tab>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>loop.results</strong>: Array of all iteration results
</li>
<li>
<strong>Structure</strong>: Results maintain iteration order
</li>
<li>
<strong>Access</strong>: Available in blocks after the loop
</li>
</ul>
</Tab>
</Tabs>
## Best Practices
- **Set reasonable limits**: Keep iteration counts reasonable to avoid long execution times
- **Use ForEach for collections**: When processing arrays or objects, use ForEach instead of For loops
- **Handle errors gracefully**: Consider adding error handling inside loops for robust workflows

13
apps/docs/content/docs/blocks/meta.json
View File

@@ -1,4 +1,15 @@
{
"title": "Blocks",
"pages": ["agent", "api", "condition", "function", "evaluator", "router", "response", "workflow"]
"pages": [
"agent",
"api",
"condition",
"function",
"evaluator",
"router",
"response",
"workflow",
"loop",
"parallel"
]
}

210
apps/docs/content/docs/blocks/parallel.mdx Normal file
View File

@@ -0,0 +1,210 @@
---
title: Parallel
description: Execute multiple blocks concurrently for faster workflow processing
---
import { Callout } from 'fumadocs-ui/components/callout'
import { Step, Steps } from 'fumadocs-ui/components/steps'
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
import { ThemeImage } from '@/components/ui/theme-image'
The Parallel block is a container block in Sim Studio that allows you to execute multiple instances of blocks concurrently.
<ThemeImage
lightSrc="/static/light/parallel-light.png"
darkSrc="/static/dark/parallel-dark.png"
alt="Parallel Block"
width={300}
height={175}
/>
<Callout type="info">
Parallel blocks are container nodes that execute their contents multiple times simultaneously, unlike loops which execute sequentially.
</Callout>
## Overview
The Parallel block enables you to:
<Steps>
<Step>
<strong>Distribute work</strong>: Process multiple items concurrently
</Step>
<Step>
<strong>Speed up execution</strong>: Run independent operations simultaneously
</Step>
<Step>
<strong>Handle bulk operations</strong>: Process large datasets efficiently
</Step>
<Step>
<strong>Aggregate results</strong>: Collect outputs from all parallel executions
</Step>
</Steps>
## Configuration Options
### Parallel Type
Choose between two types of parallel execution:
<Tabs items={['Count-based', 'Collection-based']}>
<Tab>
Execute a fixed number of parallel instances. Use this when you need to run the same operation multiple times concurrently.
```
Example: Run 5 parallel instances
- Instance 1 ┐
- Instance 2 ├─ All execute simultaneously
- Instance 3 │
- Instance 4 │
- Instance 5 ┘
```
</Tab>
<Tab>
Distribute a collection across parallel instances. Each instance processes one item from the collection simultaneously.
```
Example: Process ["task1", "task2", "task3"] in parallel
- Instance 1: Process "task1" ┐
- Instance 2: Process "task2" ├─ All execute simultaneously
- Instance 3: Process "task3" ┘
```
</Tab>
</Tabs>
## How to Use Parallel Blocks
### Creating a Parallel Block
1. Drag a Parallel block from the toolbar onto your canvas
2. Configure the parallel type and parameters
3. Drag a single block inside the parallel container
4. Connect the block as needed
### Accessing Results
After a parallel block completes, you can access aggregated results:
- **`<parallel.results>`**: Array of results from all parallel instances
## Example Use Cases
### Batch API Processing
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Scenario: Process multiple API calls simultaneously</h4>
<ol className="list-decimal pl-5 text-sm">
<li>Parallel block with collection of API endpoints</li>
<li>Inside parallel: API block calls each endpoint</li>
<li>After parallel: Process all responses together</li>
</ol>
</div>
### Multi-Model AI Processing
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Scenario: Get responses from multiple AI models</h4>
<ol className="list-decimal pl-5 text-sm">
<li>Count-based parallel set to 3 instances</li>
<li>Inside parallel: Agent configured with different model per instance</li>
<li>After parallel: Compare and select best response</li>
</ol>
</div>
## Advanced Features
### Result Aggregation
Results from all parallel instances are automatically collected:
```javascript
// In a Function block after the parallel
const allResults = input.parallel.results;
// Returns: [result1, result2, result3, ...]
```
### Instance Isolation
Each parallel instance runs independently:
- Separate variable scopes
- No shared state between instances
- Failures in one instance don't affect others
### Limitations
<Callout type="warning">
Container blocks (Loops and Parallels) cannot be nested inside each other. This means:
- You cannot place a Loop block inside a Parallel block
- You cannot place another Parallel block inside a Parallel block
- You cannot place any container block inside another container block
</Callout>
<Callout type="warning">
Parallel blocks can only contain a single block. You cannot have multiple blocks connected to each other inside a parallel - only the first block would execute in that case.
</Callout>
<Callout type="info">
While parallel execution is faster, be mindful of:
- API rate limits when making concurrent requests
- Memory usage with large datasets
- Maximum of 20 concurrent instances to prevent resource exhaustion
</Callout>
## Parallel vs Loop
Understanding when to use each:
| Feature | Parallel | Loop |
|---------|----------|------|
| Execution | Concurrent | Sequential |
| Speed | Faster for independent operations | Slower but ordered |
| Order | No guaranteed order | Maintains order |
| Use case | Independent operations | Dependent operations |
| Resource usage | Higher | Lower |
## Inputs and Outputs
<Tabs items={['Configuration', 'Variables', 'Results']}>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>Parallel Type</strong>: Choose between 'count' or 'collection'
</li>
<li>
<strong>Count</strong>: Number of instances to run (count-based)
</li>
<li>
<strong>Collection</strong>: Array or object to distribute (collection-based)
</li>
</ul>
</Tab>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>parallel.currentItem</strong>: Item for this instance
</li>
<li>
<strong>parallel.index</strong>: Instance number (0-based)
</li>
<li>
<strong>parallel.items</strong>: Full collection (collection-based)
</li>
</ul>
</Tab>
<Tab>
<ul className="list-disc space-y-2 pl-6">
<li>
<strong>parallel.results</strong>: Array of all instance results
</li>
<li>
<strong>Access</strong>: Available in blocks after the parallel
</li>
</ul>
</Tab>
</Tabs>
## Best Practices
- **Independent operations only**: Ensure operations don't depend on each other
- **Handle rate limits**: Add delays or throttling for API-heavy workflows
- **Error handling**: Each instance should handle its own errors gracefully

7
apps/docs/content/docs/execution/index.mdx
View File

@@ -26,13 +26,8 @@ import {
Sim Studio provides a powerful execution engine that brings your workflows to life. Understanding how execution works will help you design more effective workflows and troubleshoot any issues that arise.
<div>
<video autoPlay loop muted playsInline className="w-full" src="/loops.mp4"></video>
</div>
<Callout type="info">
The execution engine handles everything from block execution order to data flow, error handling,
and loop management. It ensures your workflows run efficiently and predictably.
The execution engine handles everything from block execution order to data flow and error handling. It ensures your workflows run efficiently and predictably.
</Callout>
## Execution Documentation

214
apps/docs/content/docs/execution/loops.mdx
View File

@@ -1,214 +0,0 @@
---
title: Loops
description: Creating iterative processes with loops in Sim Studio
---
import { Callout } from 'fumadocs-ui/components/callout'
import { Step, Steps } from 'fumadocs-ui/components/steps'
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
Loops are a powerful feature in Sim Studio that allow you to create iterative processes, implement feedback mechanisms, and build more sophisticated workflows.
<div>
<video autoPlay loop muted playsInline className="w-full" src="/loops.mp4"></video>
</div>
## What Are Loops?
Loops in Sim Studio allow a group of blocks to execute repeatedly, with each iteration building on the results of the previous one. This enables:
- **Iterative Refinement**: Progressively improve outputs through multiple passes
- **Batch Processing**: Process collections of items one at a time
- **Feedback Mechanisms**: Create systems that learn from their own outputs
- **Conditional Processing**: Continue execution until specific criteria are met
<Callout type="info">
Loops are particularly powerful for AI workflows, allowing you to implement techniques like
chain-of-thought reasoning, recursive refinement, and multi-step problem solving.
</Callout>
## Creating Loops
To create a loop in your workflow:
<Steps>
<Step>
<strong>Select Blocks</strong>: Choose the blocks you want to include in the loop
</Step>
<Step>
<strong>Create Loop</strong>: Use the "Create Loop" option in the editor
</Step>
<Step>
<strong>Configure Loop Settings</strong>: Set iteration limits and conditions
</Step>
<Step>
<strong>Create Feedback Connections</strong>: Connect outputs from later blocks back to earlier
blocks
</Step>
</Steps>
## Loop Configuration
When configuring a loop, you can set several important parameters:
### Iteration Limits
- **Maximum Iterations**: The maximum number of times the loop can execute (default: 5)
- **Minimum Iterations**: The minimum number of times the loop must execute before checking conditions
<Callout type="warning">
Always set a reasonable maximum iteration limit to prevent infinite loops. The default limit of 5
iterations is a good starting point for most workflows.
</Callout>
### Loop Conditions
Loops can continue based on different types of conditions:
<Tabs items={['Conditional Block', 'Function Block', 'Fixed Iterations']}>
<Tab>
Use a Condition block to determine whether the loop should continue:
```javascript
// Example condition in a Condition block
function shouldContinueLoop() {
// Get the current score from an evaluator block
const score = input.evaluatorBlock.score;
// Continue looping if score is below threshold
return score < 0.8;
}
```
The loop will continue executing as long as the condition returns true and the maximum iteration limit hasn't been reached.
</Tab>
<Tab>
Use a Function block to implement complex loop conditions:
```javascript
// Example condition in a Function block
function processAndCheckContinuation() {
// Process data from previous blocks
const currentResult = input.agentBlock.content;
const previousResults = input.memoryBlock.results || [];
// Store results for comparison
const allResults = [...previousResults, currentResult];
// Check if we've converged (results not changing significantly)
const shouldContinue = previousResults.length === 0 ||
currentResult !== previousResults[previousResults.length - 1];
return {
results: allResults,
shouldContinue: shouldContinue
};
}
```
Connect this Function block's output to a Condition block to control the loop.
</Tab>
<Tab>
Execute the loop for a fixed number of iterations:
```
// Set in loop configuration
Minimum Iterations: 3
Maximum Iterations: 3
```
This will run the loop exactly 3 times, regardless of any conditions.
</Tab>
</Tabs>
## Loop Execution
When a workflow with loops executes, the loop manager handles the iteration process:
1. **First Pass**: All blocks in the loop execute normally
2. **Iteration Check**: The system checks if another iteration should occur
3. **State Reset**: If continuing, block states within the loop are reset
4. **Next Iteration**: The loop blocks execute again with updated inputs
5. **Termination**: The loop stops when either:
- The maximum iteration count is reached
- A loop condition evaluates to false (after minimum iterations)
## Loop Use Cases
Loops enable powerful workflow patterns:
### Iterative Refinement
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Example: Content Refinement</h4>
<div className="mb-2 text-sm text-gray-600 dark:text-gray-400">
Create a loop where an Agent block generates content, an Evaluator block assesses it, and a
Function block decides whether to continue refining.
</div>
<ol className="list-decimal pl-5 text-sm">
<li>Agent generates initial content</li>
<li>Evaluator scores the content</li>
<li>Function analyzes score and provides feedback</li>
<li>Loop back to Agent with feedback for improvement</li>
<li>Continue until quality threshold is reached</li>
</ol>
</div>
### Batch Processing
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Example: Data Processing Pipeline</h4>
<div className="mb-2 text-sm text-gray-600 dark:text-gray-400">
Process a collection of items one at a time through a series of blocks.
</div>
<ol className="list-decimal pl-5 text-sm">
<li>Function block extracts the next item from a collection</li>
<li>Processing blocks operate on the single item</li>
<li>Results are accumulated in a Memory block</li>
<li>Loop continues until all items are processed</li>
</ol>
</div>
### Recursive Problem Solving
<div className="mb-4 rounded-md border p-4">
<h4 className="font-medium">Example: Multi-step Reasoning</h4>
<div className="mb-2 text-sm text-gray-600 dark:text-gray-400">
Implement a recursive approach to complex problem solving.
</div>
<ol className="list-decimal pl-5 text-sm">
<li>Agent analyzes the current problem state</li>
<li>Function block implements a step in the solution</li>
<li>Condition block checks if the problem is solved</li>
<li>Loop continues until solution is found or maximum steps reached</li>
</ol>
</div>
## Best Practices for Loops
To use loops effectively in your workflows:
- **Set Appropriate Limits**: Always configure reasonable iteration limits
- **Use Memory Blocks**: Store state between iterations with Memory blocks
- **Include Exit Conditions**: Define clear conditions for when loops should terminate
- **Monitor Performance**: Watch for performance impacts with many iterations
- **Test Thoroughly**: Verify that loops terminate as expected in all scenarios
<Callout type="warning">
Loops with many blocks or complex operations can impact performance. Consider optimizing
individual blocks if your loops need many iterations.
</Callout>
## Loop Debugging
When debugging loops in your workflows:
- **Check Iteration Counts**: Verify the loop is executing the expected number of times
- **Inspect Block Inputs/Outputs**: Look at how data changes between iterations
- **Review Loop Conditions**: Ensure conditions are evaluating as expected
- **Use Console Logging**: Add console.log statements in Function blocks to track loop progress
- **Monitor Memory Usage**: Watch for growing data structures that might cause performance issues
By mastering loops, you can create much more sophisticated and powerful workflows in Sim Studio.

2
apps/docs/content/docs/execution/meta.json
View File

@@ -1,4 +1,4 @@
{
"title": "Execution",
"pages": ["basics", "loops", "advanced"]
"pages": ["basics", "advanced"]
}

40
apps/docs/content/docs/tools/gmail.mdx
View File

@@ -102,6 +102,46 @@ Draft emails using Gmail
| `threadId` | string |
| `labelIds` | string |
### `gmail_read`
Read emails from Gmail
#### Input
| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessToken` | string | Yes | Access token for Gmail API |
| `messageId` | string | No | ID of the message to read |
| `folder` | string | No | Folder/label to read emails from |
| `unreadOnly` | boolean | No | Only retrieve unread messages |
| `maxResults` | number | No | Maximum number of messages to retrieve \(default: 1, max: 10\) |
#### Output
| Parameter | Type |
| --------- | ---- |
| `content` | string |
| `metadata` | string |
### `gmail_search`
Search emails in Gmail
#### Input
| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `accessToken` | string | Yes | Access token for Gmail API |
| `query` | string | Yes | Search query for emails |
| `maxResults` | number | No | Maximum number of results to return \(default: 1, max: 10\) |
#### Output
| Parameter | Type |
| --------- | ---- |
| `content` | string |
| `metadata` | string |
## Block Configuration

BIN
apps/docs/public/static/dark/loop-dark.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

BIN
apps/docs/public/static/dark/parallel-dark.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 102 KiB

BIN
apps/docs/public/static/light/loop-light.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

BIN
apps/docs/public/static/light/parallel-light.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

27
apps/sim/app/api/knowledge/[id]/documents/route.test.ts
View File

@@ -30,6 +30,8 @@ describe('Knowledge Base Documents API Route', () => {
from: vi.fn().mockReturnThis(),
where: vi.fn().mockReturnThis(),
orderBy: vi.fn().mockReturnThis(),
limit: vi.fn().mockReturnThis(),
offset: vi.fn().mockReturnThis(),
insert: vi.fn().mockReturnThis(),
values: vi.fn().mockReturnThis(),
update: vi.fn().mockReturnThis(),
@@ -99,7 +101,12 @@ describe('Knowledge Base Documents API Route', () => {
it('should retrieve documents successfully for authenticated user', async () => {
mockAuth$.mockAuthenticatedUser()
mockCheckKnowledgeBaseAccess.mockResolvedValue({ hasAccess: true })
mockDbChain.orderBy.mockResolvedValue([mockDocument])
// Mock the count query (first query)
mockDbChain.where.mockResolvedValueOnce([{ count: 1 }])
// Mock the documents query (second query)
mockDbChain.offset.mockResolvedValue([mockDocument])
const req = createMockRequest('GET')
const { GET } = await import('./route')
@@ -108,8 +115,8 @@ describe('Knowledge Base Documents API Route', () => {
expect(response.status).toBe(200)
expect(data.success).toBe(true)
expect(data.data).toHaveLength(1)
expect(data.data[0].id).toBe('doc-123')
expect(data.data.documents).toHaveLength(1)
expect(data.data.documents[0].id).toBe('doc-123')
expect(mockDbChain.select).toHaveBeenCalled()
expect(mockCheckKnowledgeBaseAccess).toHaveBeenCalledWith('kb-123', 'user-123')
})
@@ -117,7 +124,12 @@ describe('Knowledge Base Documents API Route', () => {
it('should filter disabled documents by default', async () => {
mockAuth$.mockAuthenticatedUser()
mockCheckKnowledgeBaseAccess.mockResolvedValue({ hasAccess: true })
mockDbChain.orderBy.mockResolvedValue([mockDocument])
// Mock the count query (first query)
mockDbChain.where.mockResolvedValueOnce([{ count: 1 }])
// Mock the documents query (second query)
mockDbChain.offset.mockResolvedValue([mockDocument])
const req = createMockRequest('GET')
const { GET } = await import('./route')
@@ -130,7 +142,12 @@ describe('Knowledge Base Documents API Route', () => {
it('should include disabled documents when requested', async () => {
mockAuth$.mockAuthenticatedUser()
mockCheckKnowledgeBaseAccess.mockResolvedValue({ hasAccess: true })
mockDbChain.orderBy.mockResolvedValue([mockDocument])
// Mock the count query (first query)
mockDbChain.where.mockResolvedValueOnce([{ count: 1 }])
// Mock the documents query (second query)
mockDbChain.offset.mockResolvedValue([mockDocument])
const url = 'http://localhost:3000/api/knowledge/kb-123/documents?includeDisabled=true'
const req = new Request(url, { method: 'GET' }) as any

36
apps/sim/app/api/knowledge/[id]/documents/route.ts
View File

@@ -1,5 +1,5 @@
import crypto from 'node:crypto'
import { and, desc, eq, inArray, isNull } from 'drizzle-orm'
import { and, desc, eq, inArray, isNull, sql } from 'drizzle-orm'
import { type NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { getSession } from '@/lib/auth'
@@ -210,6 +210,9 @@ export async function GET(req: NextRequest, { params }: { params: Promise<{ id:
const url = new URL(req.url)
const includeDisabled = url.searchParams.get('includeDisabled') === 'true'
const search = url.searchParams.get('search')
const limit = Number.parseInt(url.searchParams.get('limit') || '50')
const offset = Number.parseInt(url.searchParams.get('offset') || '0')
// Build where conditions
const whereConditions = [
@@ -222,6 +225,23 @@ export async function GET(req: NextRequest, { params }: { params: Promise<{ id:
whereConditions.push(eq(document.enabled, true))
}
// Add search condition if provided
if (search) {
whereConditions.push(
// Search in filename
sql`LOWER(${document.filename}) LIKE LOWER(${`%${search}%`})`
)
}
// Get total count for pagination
const totalResult = await db
.select({ count: sql<number>`COUNT(*)` })
.from(document)
.where(and(...whereConditions))
const total = totalResult[0]?.count || 0
const hasMore = offset + limit < total
const documents = await db
.select({
id: document.id,
@@ -250,14 +270,24 @@ export async function GET(req: NextRequest, { params }: { params: Promise<{ id:
.from(document)
.where(and(...whereConditions))
.orderBy(desc(document.uploadedAt))
.limit(limit)
.offset(offset)
logger.info(
`[${requestId}] Retrieved ${documents.length} documents for knowledge base ${knowledgeBaseId}`
`[${requestId}] Retrieved ${documents.length} documents (${offset}-${offset + documents.length} of ${total}) for knowledge base ${knowledgeBaseId}`
)
return NextResponse.json({
success: true,
data: documents,
data: {
documents,
pagination: {
total,
limit,
offset,
hasMore,
},
},
})
} catch (error) {
logger.error(`[${requestId}] Error fetching documents`, error)

2
apps/sim/app/api/knowledge/utils.test.ts
View File

@@ -176,7 +176,7 @@ describe('Knowledge Utils', () => {
{}
)
expect(dbOps.order).toEqual(['insert', 'updateDoc', 'updateKb'])
expect(dbOps.order).toEqual(['insert', 'updateDoc'])
expect(dbOps.updatePayloads[0]).toMatchObject({
processingStatus: 'completed',

10
apps/sim/app/api/knowledge/utils.ts
View File

@@ -1,5 +1,5 @@
import crypto from 'crypto'
import { and, eq, isNull, sql } from 'drizzle-orm'
import { and, eq, isNull } from 'drizzle-orm'
import { processDocument } from '@/lib/documents/document-processor'
import { retryWithExponentialBackoff } from '@/lib/documents/utils'
import { env } from '@/lib/env'
@@ -522,14 +522,6 @@ export async function processDocumentAsync(
processingError: null,
})
.where(eq(document.id, documentId))
await tx
.update(knowledgeBase)
.set({
tokenCount: sql`${knowledgeBase.tokenCount} + ${processed.metadata.tokenCount}`,
updatedAt: now,
})
.where(eq(knowledgeBase.id, knowledgeBaseId))
})
})(),
TIMEOUTS.OVERALL_PROCESSING,

2
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/[documentId]/document.tsx
View File

@@ -117,7 +117,7 @@ export function Document({
setError(null)
const cachedDocuments = getCachedDocuments(knowledgeBaseId)
const cachedDoc = cachedDocuments?.find((d) => d.id === documentId)
const cachedDoc = cachedDocuments?.documents?.find((d) => d.id === documentId)
if (cachedDoc) {
setDocument(cachedDoc)

190
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/base.tsx
View File

@@ -1,9 +1,11 @@
'use client'
import { useEffect, useState } from 'react'
import { useCallback, useEffect, useState } from 'react'
import { format } from 'date-fns'
import {
AlertCircle,
ChevronLeft,
ChevronRight,
Circle,
CircleOff,
FileText,
@@ -25,6 +27,7 @@ import {
} from '@/components/ui/alert-dialog'
import { Button } from '@/components/ui/button'
import { Checkbox } from '@/components/ui/checkbox'
import { SearchHighlight } from '@/components/ui/search-highlight'
import { Tooltip, TooltipContent, TooltipTrigger } from '@/components/ui/tooltip'
import { createLogger } from '@/lib/logs/console-logger'
import { ActionBar } from '@/app/workspace/[workspaceId]/knowledge/[id]/components/action-bar/action-bar'
@@ -40,6 +43,9 @@ import { UploadModal } from './components/upload-modal/upload-modal'
const logger = createLogger('KnowledgeBase')
// Constants
const DOCUMENTS_PER_PAGE = 50
interface KnowledgeBaseProps {
id: string
knowledgeBaseName?: string
@@ -118,6 +124,22 @@ export function KnowledgeBase({
const { removeKnowledgeBase } = useKnowledgeStore()
const params = useParams()
const workspaceId = params.workspaceId as string
const [searchQuery, setSearchQuery] = useState('')
// Memoize the search query setter to prevent unnecessary re-renders
const handleSearchChange = useCallback((newQuery: string) => {
setSearchQuery(newQuery)
setCurrentPage(1) // Reset to page 1 when searching
}, [])
const [selectedDocuments, setSelectedDocuments] = useState<Set<string>>(new Set())
const [showDeleteDialog, setShowDeleteDialog] = useState(false)
const [showUploadModal, setShowUploadModal] = useState(false)
const [isDeleting, setIsDeleting] = useState(false)
const [isBulkOperating, setIsBulkOperating] = useState(false)
const [currentPage, setCurrentPage] = useState(1)
const {
knowledgeBase,
isLoading: isLoadingKnowledgeBase,
@@ -125,27 +147,52 @@ export function KnowledgeBase({
} = useKnowledgeBase(id)
const {
documents,
pagination,
isLoading: isLoadingDocuments,
error: documentsError,
updateDocument,
refreshDocuments,
} = useKnowledgeBaseDocuments(id)
} = useKnowledgeBaseDocuments(id, {
search: searchQuery || undefined,
limit: DOCUMENTS_PER_PAGE,
offset: (currentPage - 1) * DOCUMENTS_PER_PAGE,
})
const isSidebarCollapsed =
mode === 'expanded' ? !isExpanded : mode === 'collapsed' || mode === 'hover'
const [searchQuery, setSearchQuery] = useState('')
const [selectedDocuments, setSelectedDocuments] = useState<Set<string>>(new Set())
const [showDeleteDialog, setShowDeleteDialog] = useState(false)
const [showUploadModal, setShowUploadModal] = useState(false)
const [isDeleting, setIsDeleting] = useState(false)
const [isBulkOperating, setIsBulkOperating] = useState(false)
const router = useRouter()
const knowledgeBaseName = knowledgeBase?.name || passedKnowledgeBaseName || 'Knowledge Base'
const error = knowledgeBaseError || documentsError
// Pagination calculations
const totalPages = Math.ceil(pagination.total / pagination.limit)
const hasNextPage = currentPage < totalPages
const hasPrevPage = currentPage > 1
// Navigation functions
const goToPage = useCallback(
(page: number) => {
if (page >= 1 && page <= totalPages) {
setCurrentPage(page)
}
},
[totalPages]
)
const nextPage = useCallback(() => {
if (hasNextPage) {
setCurrentPage((prev) => prev + 1)
}
}, [hasNextPage])
const prevPage = useCallback(() => {
if (hasPrevPage) {
setCurrentPage((prev) => prev - 1)
}
}, [hasPrevPage])
// Auto-refresh documents when there are processing documents
useEffect(() => {
const hasProcessingDocuments = documents.some(
@@ -220,10 +267,8 @@ export function KnowledgeBase({
await Promise.allSettled(markFailedPromises)
}
// Filter documents based on search query
const filteredDocuments = documents.filter((doc) =>
doc.filename.toLowerCase().includes(searchQuery.toLowerCase())
)
// Calculate pagination info for display
const totalItems = pagination?.total || 0
const handleToggleEnabled = async (docId: string) => {
const document = documents.find((doc) => doc.id === docId)
@@ -366,14 +411,13 @@ export function KnowledgeBase({
const handleSelectAll = (checked: boolean) => {
if (checked) {
setSelectedDocuments(new Set(filteredDocuments.map((doc) => doc.id)))
setSelectedDocuments(new Set(documents.map((doc) => doc.id)))
} else {
setSelectedDocuments(new Set())
}
}
const isAllSelected =
filteredDocuments.length > 0 && selectedDocuments.size === filteredDocuments.length
const isAllSelected = documents.length > 0 && selectedDocuments.size === documents.length
const handleDocumentClick = (docId: string) => {
// Find the document to get its filename
@@ -621,20 +665,35 @@ export function KnowledgeBase({
{/* Main Content */}
<div className='flex-1 overflow-auto'>
<div className='px-6 pb-6'>
{/* Search and Create Section */}
<div className='mb-4 flex items-center justify-between pt-1'>
<SearchInput
value={searchQuery}
onChange={setSearchQuery}
placeholder='Search documents...'
/>
{/* Search and Filters Section */}
<div className='mb-4 space-y-3 pt-1'>
<div className='flex items-center justify-between'>
<SearchInput
value={searchQuery}
onChange={handleSearchChange}
placeholder='Search documents...'
/>
<div className='flex items-center gap-3'>
{/* Add Documents Button */}
<PrimaryButton onClick={handleAddDocuments}>
<Plus className='h-3.5 w-3.5' />
Add Documents
</PrimaryButton>
<div className='flex items-center gap-3'>
{/* Clear Search Button */}
{searchQuery && (
<button
onClick={() => {
setSearchQuery('')
setCurrentPage(1)
}}
className='text-muted-foreground text-sm hover:text-foreground'
>
Clear search
</button>
)}
{/* Add Documents Button */}
<PrimaryButton onClick={handleAddDocuments}>
<Plus className='h-3.5 w-3.5' />
Add Documents
</PrimaryButton>
</div>
</div>
</div>
@@ -714,7 +773,7 @@ export function KnowledgeBase({
<col className='w-[14%]' />
</colgroup>
<tbody>
{filteredDocuments.length === 0 && !isLoadingDocuments ? (
{documents.length === 0 && !isLoadingDocuments ? (
<tr className='border-b transition-colors hover:bg-accent/30'>
{/* Select column */}
<td className='px-4 py-3'>
@@ -726,7 +785,7 @@ export function KnowledgeBase({
<div className='flex items-center gap-2'>
<FileText className='h-6 w-5 text-muted-foreground' />
<span className='text-muted-foreground text-sm italic'>
{documents.length === 0
{totalItems === 0
? 'No documents yet'
: 'No documents match your search'}
</span>
@@ -793,7 +852,7 @@ export function KnowledgeBase({
</tr>
))
) : (
filteredDocuments.map((doc) => {
documents.map((doc) => {
const isSelected = selectedDocuments.has(doc.id)
const statusDisplay = getStatusDisplay(doc)
// const processingTime = getProcessingTime(doc)
@@ -834,7 +893,10 @@ export function KnowledgeBase({
<Tooltip>
<TooltipTrigger asChild>
<span className='block truncate text-sm' title={doc.filename}>
{doc.filename}
<SearchHighlight
text={doc.filename}
searchQuery={searchQuery}
/>
</span>
</TooltipTrigger>
<TooltipContent side='top'>{doc.filename}</TooltipContent>
@@ -998,6 +1060,64 @@ export function KnowledgeBase({
</tbody>
</table>
</div>
{/* Pagination Controls */}
{totalPages > 1 && (
<div className='flex items-center justify-center border-t bg-background px-6 py-4'>
<div className='flex items-center gap-1'>
<Button
variant='ghost'
size='sm'
onClick={prevPage}
disabled={!hasPrevPage || isLoadingDocuments}
className='h-8 w-8 p-0'
>
<ChevronLeft className='h-4 w-4' />
</Button>
{/* Page numbers - show a few around current page */}
<div className='mx-4 flex items-center gap-6'>
{Array.from({ length: Math.min(totalPages, 5) }, (_, i) => {
let page: number
if (totalPages <= 5) {
page = i + 1
} else if (currentPage <= 3) {
page = i + 1
} else if (currentPage >= totalPages - 2) {
page = totalPages - 4 + i
} else {
page = currentPage - 2 + i
}
if (page < 1 || page > totalPages) return null
return (
<button
key={page}
onClick={() => goToPage(page)}
disabled={isLoadingDocuments}
className={`font-medium text-sm transition-colors hover:text-foreground disabled:cursor-not-allowed disabled:opacity-50 ${
page === currentPage ? 'text-foreground' : 'text-muted-foreground'
}`}
>
{page}
</button>
)
})}
</div>
<Button
variant='ghost'
size='sm'
onClick={nextPage}
disabled={!hasNextPage || isLoadingDocuments}
className='h-8 w-8 p-0'
>
<ChevronRight className='h-4 w-4' />
</Button>
</div>
</div>
)}
</div>
</div>
</div>
@@ -1011,8 +1131,8 @@ export function KnowledgeBase({
<AlertDialogTitle>Delete Knowledge Base</AlertDialogTitle>
<AlertDialogDescription>
Are you sure you want to delete "{knowledgeBaseName}"? This will permanently delete
the knowledge base and all {documents.length} document
{documents.length === 1 ? '' : 's'} within it. This action cannot be undone.
the knowledge base and all {totalItems} document
{totalItems === 1 ? '' : 's'} within it. This action cannot be undone.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>

2
apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/workflow-block/components/sub-block/components/document-selector/document-selector.tsx
View File

@@ -87,7 +87,7 @@ export function DocumentSelector({
throw new Error(result.error || 'Failed to fetch documents')
}
const fetchedDocuments = result.data || []
const fetchedDocuments = result.data.documents || result.data || []
setDocuments(fetchedDocuments)
} catch (err) {
if ((err as Error).name === 'AbortError') return

129
apps/sim/blocks/blocks/gmail.ts
View File

@@ -21,8 +21,9 @@ export const GmailBlock: BlockConfig<GmailToolResponse> = {
layout: 'full',
options: [
{ label: 'Send Email', id: 'send_gmail' },
// { label: 'Read Email', id: 'read_gmail' },
{ label: 'Read Email', id: 'read_gmail' },
{ label: 'Draft Email', id: 'draft_gmail' },
{ label: 'Search Email', id: 'search_gmail' },
],
},
// Gmail Credentials
@@ -67,70 +68,62 @@ export const GmailBlock: BlockConfig<GmailToolResponse> = {
condition: { field: 'operation', value: ['send_gmail', 'draft_gmail'] },
},
// Read Email Fields - Add folder selector
// {
// id: 'folder',
// title: 'Label',
// type: 'folder-selector',
// layout: 'full',
// provider: 'google-email',
// serviceId: 'gmail',
// requiredScopes: [
// // 'https://www.googleapis.com/auth/gmail.readonly',
// 'https://www.googleapis.com/auth/gmail.labels',
// ],
// placeholder: 'Select Gmail label/folder',
// condition: { field: 'operation', value: 'read_gmail' },
// },
// {
// id: 'unreadOnly',
// title: 'Unread Only',
// type: 'switch',
// layout: 'full',
// condition: { field: 'operation', value: 'read_gmail' },
// },
// {
// id: 'maxResults',
// title: 'Number of Emails',
// type: 'short-input',
// layout: 'full',
// placeholder: 'Number of emails to retrieve (default: 1, max: 10)',
// condition: { field: 'operation', value: 'read_gmail' },
// },
// {
// id: 'messageId',
// title: 'Message ID',
// type: 'short-input',
// layout: 'full',
// placeholder: 'Enter message ID to read (optional)',
// condition: {
// field: 'operation',
// value: 'read_gmail',
// and: {
// field: 'folder',
// value: '',
// },
// },
// },
// // Search Fields
// {
// id: 'query',
// title: 'Search Query',
// type: 'short-input',
// layout: 'full',
// placeholder: 'Enter search terms',
// condition: { field: 'operation', value: 'search_gmail' },
// },
// {
// id: 'maxResults',
// title: 'Max Results',
// type: 'short-input',
// layout: 'full',
// placeholder: 'Maximum number of results (default: 10)',
// condition: { field: 'operation', value: 'search_gmail' },
// },
{
id: 'folder',
title: 'Label',
type: 'folder-selector',
layout: 'full',
provider: 'google-email',
serviceId: 'gmail',
requiredScopes: [
'https://www.googleapis.com/auth/gmail.readonly',
'https://www.googleapis.com/auth/gmail.labels',
],
placeholder: 'Select Gmail label/folder',
condition: { field: 'operation', value: 'read_gmail' },
},
{
id: 'unreadOnly',
title: 'Unread Only',
type: 'switch',
layout: 'full',
condition: { field: 'operation', value: 'read_gmail' },
},
{
id: 'messageId',
title: 'Message ID',
type: 'short-input',
layout: 'full',
placeholder: 'Enter message ID to read (optional)',
condition: {
field: 'operation',
value: 'read_gmail',
and: {
field: 'folder',
value: '',
},
},
},
// Search Fields
{
id: 'query',
title: 'Search Query',
type: 'short-input',
layout: 'full',
placeholder: 'Enter search terms',
condition: { field: 'operation', value: 'search_gmail' },
},
{
id: 'maxResults',
title: 'Max Results',
type: 'short-input',
layout: 'full',
placeholder: 'Maximum number of results (default: 10)',
condition: { field: 'operation', value: ['search_gmail', 'read_gmail'] },
},
],
tools: {
access: ['gmail_send', 'gmail_draft'],
access: ['gmail_send', 'gmail_draft', 'gmail_read', 'gmail_search'],
config: {
tool: (params) => {
switch (params.operation) {
@@ -138,6 +131,10 @@ export const GmailBlock: BlockConfig<GmailToolResponse> = {
return 'gmail_send'
case 'draft_gmail':
return 'gmail_draft'
case 'search_gmail':
return 'gmail_search'
case 'read_gmail':
return 'gmail_read'
default:
throw new Error(`Invalid Gmail operation: ${params.operation}`)
}
@@ -146,9 +143,9 @@ export const GmailBlock: BlockConfig<GmailToolResponse> = {
// Pass the credential directly from the credential field
const { credential, ...rest } = params
// Set default folder to INBOX if not specified
if (rest.operation === 'read_gmail' && !rest.folder) {
rest.folder = 'INBOX'
// Ensure folder is always provided for read_gmail operation
if (rest.operation === 'read_gmail') {
rest.folder = rest.folder || 'INBOX'
}
return {

33
apps/sim/components/ui/search-highlight.tsx
View File

@@ -1,7 +1,5 @@
'use client'
import { Fragment } from 'react'
interface SearchHighlightProps {
text: string
searchQuery: string
@@ -13,22 +11,18 @@ export function SearchHighlight({ text, searchQuery, className = '' }: SearchHig
return <span className={className}>{text}</span>
}
// Create a regex to find matches (case-insensitive)
// Handle multiple search terms separated by spaces
// Create regex pattern for all search terms
const searchTerms = searchQuery
.trim()
.split(/\s+/)
.filter((term) => term.length > 0)
.map((term) => term.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))
if (searchTerms.length === 0) {
return <span className={className}>{text}</span>
}
// Create regex pattern for all search terms
const escapedTerms = searchTerms.map((term) => term.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))
const regexPattern = `(${escapedTerms.join('|')})`
const regex = new RegExp(regexPattern, 'gi')
const regex = new RegExp(`(${searchTerms.join('|')})`, 'gi')
const parts = text.split(regex)
return (
@@ -36,18 +30,17 @@ export function SearchHighlight({ text, searchQuery, className = '' }: SearchHig
{parts.map((part, index) => {
if (!part) return null
const isMatch = regex.test(part)
const isMatch = searchTerms.some((term) => new RegExp(term, 'gi').test(part))
return (
<Fragment key={index}>
{isMatch ? (
<span className='rounded-sm bg-yellow-200 px-0.5 py-0.5 font-medium text-yellow-900 dark:bg-yellow-900/50 dark:text-yellow-200'>
{part}
</span>
) : (
part
)}
</Fragment>
return isMatch ? (
<span
key={index}
className='bg-yellow-200 text-yellow-900 dark:bg-yellow-900/50 dark:text-yellow-200'
>
{part}
</span>
) : (
<span key={index}>{part}</span>
)
})}
</span>

26
apps/sim/db/index.ts
View File

@@ -11,25 +11,23 @@ const connectionString = env.POSTGRES_URL ?? env.DATABASE_URL
/**
* Connection Pool Allocation Strategy
*
* Main App (this file): 3 connections per instance
* Socket Server Operations: 2 connections
* Socket Server Room Manager: 1 connection
* Main App: 25 connections per instance
* Socket Server: 3 connections total
*
* With ~3-4 Vercel serverless instances typically active:
* - Main app: 3 × 4 = 12 connections
* - Socket server: 2 + 1 = 3 connections
* - Buffer: 5 connections for spikes/other services
* - Total: ~20 connections (at capacity limit)
*
* This conservative allocation prevents pool exhaustion while maintaining performance.
* - Main app: 25 × 4 = 100 connections
* - Socket server: 3 connections
* - Buffer: 25 connections
* - Total: ~128 connections
* - Supabase limit: 128 connections (16XL instance)
*/
const postgresClient = postgres(connectionString, {
prepare: false, // Disable prefetch as it is not supported for "Transaction" pool mode
idle_timeout: 20, // Reduce idle timeout to 20 seconds to free up connections faster
connect_timeout: 30, // Increase connect timeout to 30 seconds to handle network issues
max: 2, // Further reduced limit to prevent Supabase connection exhaustion
onnotice: () => {}, // Disable notices to reduce noise
prepare: false,
idle_timeout: 20,
connect_timeout: 30,
max: 25,
onnotice: () => {},
})
const drizzleClient = drizzle(postgresClient, { schema })

66
apps/sim/hooks/use-knowledge.ts
View File

@@ -40,24 +40,33 @@ export function useKnowledgeBase(id: string) {
}
}
export function useKnowledgeBaseDocuments(knowledgeBaseId: string) {
// Constants
const MAX_DOCUMENTS_LIMIT = 10000
const DEFAULT_PAGE_SIZE = 50
export function useKnowledgeBaseDocuments(
knowledgeBaseId: string,
options?: { search?: string; limit?: number; offset?: number }
) {
const { getDocuments, getCachedDocuments, loadingDocuments, updateDocument, refreshDocuments } =
useKnowledgeStore()
const [error, setError] = useState<string | null>(null)
const documents = getCachedDocuments(knowledgeBaseId) || []
const documentsCache = getCachedDocuments(knowledgeBaseId)
const allDocuments = documentsCache?.documents || []
const isLoading = loadingDocuments.has(knowledgeBaseId)
// Load all documents on initial mount
useEffect(() => {
if (!knowledgeBaseId || documents.length > 0 || isLoading) return
if (!knowledgeBaseId || allDocuments.length > 0 || isLoading) return
let isMounted = true
const loadData = async () => {
const loadAllDocuments = async () => {
try {
setError(null)
await getDocuments(knowledgeBaseId)
await getDocuments(knowledgeBaseId, { limit: MAX_DOCUMENTS_LIMIT })
} catch (err) {
if (isMounted) {
setError(err instanceof Error ? err.message : 'Failed to load documents')
@@ -65,28 +74,59 @@ export function useKnowledgeBaseDocuments(knowledgeBaseId: string) {
}
}
loadData()
loadAllDocuments()
return () => {
isMounted = false
}
}, [knowledgeBaseId, documents.length, isLoading]) // Removed getDocuments from dependencies
}, [knowledgeBaseId, allDocuments.length, isLoading, getDocuments])
const refreshDocumentsData = async () => {
// Client-side filtering and pagination
const { documents, pagination } = useMemo(() => {
let filteredDocs = allDocuments
// Apply search filter
if (options?.search) {
const searchLower = options.search.toLowerCase()
filteredDocs = filteredDocs.filter((doc) => doc.filename.toLowerCase().includes(searchLower))
}
// Apply pagination
const offset = options?.offset || 0
const limit = options?.limit || DEFAULT_PAGE_SIZE
const total = filteredDocs.length
const paginatedDocs = filteredDocs.slice(offset, offset + limit)
return {
documents: paginatedDocs,
pagination: {
total,
limit,
offset,
hasMore: offset + limit < total,
},
}
}, [allDocuments, options?.search, options?.limit, options?.offset])
const refreshDocumentsData = useCallback(async () => {
try {
setError(null)
await refreshDocuments(knowledgeBaseId)
await refreshDocuments(knowledgeBaseId, { limit: MAX_DOCUMENTS_LIMIT })
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to refresh documents')
}
}
}, [knowledgeBaseId, refreshDocuments])
const updateDocumentLocal = (documentId: string, updates: Partial<DocumentData>) => {
updateDocument(knowledgeBaseId, documentId, updates)
}
const updateDocumentLocal = useCallback(
(documentId: string, updates: Partial<DocumentData>) => {
updateDocument(knowledgeBaseId, documentId, updates)
},
[knowledgeBaseId, updateDocument]
)
return {
documents,
pagination,
isLoading,
error,
refreshDocuments: refreshDocumentsData,

145
apps/sim/stores/knowledge/store.ts
View File

@@ -88,10 +88,24 @@ export interface ChunksCache {
lastFetchTime: number
}
export interface DocumentsPagination {
total: number
limit: number
offset: number
hasMore: boolean
}
export interface DocumentsCache {
documents: DocumentData[]
pagination: DocumentsPagination
searchQuery?: string
lastFetchTime: number
}
interface KnowledgeStore {
// State
knowledgeBases: Record<string, KnowledgeBaseData>
documents: Record<string, DocumentData[]> // knowledgeBaseId -> documents
documents: Record<string, DocumentsCache> // knowledgeBaseId -> documents cache
chunks: Record<string, ChunksCache> // documentId -> chunks cache
knowledgeBasesList: KnowledgeBaseData[]
@@ -104,14 +118,20 @@ interface KnowledgeStore {
// Actions
getKnowledgeBase: (id: string) => Promise<KnowledgeBaseData | null>
getDocuments: (knowledgeBaseId: string) => Promise<DocumentData[]>
getDocuments: (
knowledgeBaseId: string,
options?: { search?: string; limit?: number; offset?: number }
) => Promise<DocumentData[]>
getChunks: (
knowledgeBaseId: string,
documentId: string,
options?: { search?: string; limit?: number; offset?: number }
) => Promise<ChunkData[]>
getKnowledgeBasesList: () => Promise<KnowledgeBaseData[]>
refreshDocuments: (knowledgeBaseId: string) => Promise<DocumentData[]>
refreshDocuments: (
knowledgeBaseId: string,
options?: { search?: string; limit?: number; offset?: number }
) => Promise<DocumentData[]>
refreshChunks: (
knowledgeBaseId: string,
documentId: string,
@@ -133,7 +153,7 @@ interface KnowledgeStore {
// Getters
getCachedKnowledgeBase: (id: string) => KnowledgeBaseData | null
getCachedDocuments: (knowledgeBaseId: string) => DocumentData[] | null
getCachedDocuments: (knowledgeBaseId: string) => DocumentsCache | null
getCachedChunks: (documentId: string, options?: { search?: string }) => ChunksCache | null
// Loading state getters
@@ -235,18 +255,21 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
}
},
getDocuments: async (knowledgeBaseId: string) => {
getDocuments: async (
knowledgeBaseId: string,
options?: { search?: string; limit?: number; offset?: number }
) => {
const state = get()
// Return cached documents if they exist
// Return cached documents if they exist (no search-based caching since we do client-side filtering)
const cached = state.documents[knowledgeBaseId]
if (cached) {
return cached
if (cached && cached.documents.length > 0) {
return cached.documents
}
// Return empty array if already loading to prevent duplicate requests
if (state.loadingDocuments.has(knowledgeBaseId)) {
return []
return cached?.documents || []
}
try {
@@ -254,7 +277,14 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
loadingDocuments: new Set([...state.loadingDocuments, knowledgeBaseId]),
}))
const response = await fetch(`/api/knowledge/${knowledgeBaseId}/documents`)
// Build query parameters
const params = new URLSearchParams()
if (options?.search) params.set('search', options.search)
if (options?.limit) params.set('limit', options.limit.toString())
if (options?.offset) params.set('offset', options.offset.toString())
const url = `/api/knowledge/${knowledgeBaseId}/documents${params.toString() ? `?${params.toString()}` : ''}`
const response = await fetch(url)
if (!response.ok) {
throw new Error(`Failed to fetch documents: ${response.statusText}`)
@@ -266,12 +296,25 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
throw new Error(result.error || 'Failed to fetch documents')
}
const documents = result.data
const documents = result.data.documents || result.data // Handle both paginated and non-paginated responses
const pagination = result.data.pagination || {
total: documents.length,
limit: options?.limit || 50,
offset: options?.offset || 0,
hasMore: false,
}
const documentsCache: DocumentsCache = {
documents,
pagination,
searchQuery: options?.search,
lastFetchTime: Date.now(),
}
set((state) => ({
documents: {
...state.documents,
[knowledgeBaseId]: documents,
[knowledgeBaseId]: documentsCache,
},
loadingDocuments: new Set(
[...state.loadingDocuments].filter((loadingId) => loadingId !== knowledgeBaseId)
@@ -455,12 +498,15 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
}
},
refreshDocuments: async (knowledgeBaseId: string) => {
refreshDocuments: async (
knowledgeBaseId: string,
options?: { search?: string; limit?: number; offset?: number }
) => {
const state = get()
// Return empty array if already loading to prevent duplicate requests
if (state.loadingDocuments.has(knowledgeBaseId)) {
return state.documents[knowledgeBaseId] || []
return state.documents[knowledgeBaseId]?.documents || []
}
try {
@@ -468,7 +514,14 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
loadingDocuments: new Set([...state.loadingDocuments, knowledgeBaseId]),
}))
const response = await fetch(`/api/knowledge/${knowledgeBaseId}/documents`)
// Build query parameters - for refresh, always start from offset 0
const params = new URLSearchParams()
if (options?.search) params.set('search', options.search)
if (options?.limit) params.set('limit', options.limit.toString())
params.set('offset', '0') // Always start fresh on refresh
const url = `/api/knowledge/${knowledgeBaseId}/documents${params.toString() ? `?${params.toString()}` : ''}`
const response = await fetch(url)
if (!response.ok) {
throw new Error(`Failed to fetch documents: ${response.statusText}`)
@@ -480,10 +533,16 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
throw new Error(result.error || 'Failed to fetch documents')
}
const serverDocuments = result.data
const serverDocuments = result.data.documents || result.data
const pagination = result.data.pagination || {
total: serverDocuments.length,
limit: options?.limit || 50,
offset: 0,
hasMore: false,
}
set((state) => {
const currentDocuments = state.documents[knowledgeBaseId] || []
const currentDocuments = state.documents[knowledgeBaseId]?.documents || []
// Create a map of server documents by filename for quick lookup
const serverDocumentsByFilename = new Map()
@@ -535,10 +594,17 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
// Add any remaining temporary documents that don't have server equivalents
const finalDocuments = [...mergedDocuments, ...filteredCurrentDocs]
const documentsCache: DocumentsCache = {
documents: finalDocuments,
pagination,
searchQuery: options?.search,
lastFetchTime: Date.now(),
}
return {
documents: {
...state.documents,
[knowledgeBaseId]: finalDocuments,
[knowledgeBaseId]: documentsCache,
},
loadingDocuments: new Set(
[...state.loadingDocuments].filter((loadingId) => loadingId !== knowledgeBaseId)
@@ -638,17 +704,20 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
updateDocument: (knowledgeBaseId: string, documentId: string, updates: Partial<DocumentData>) => {
set((state) => {
const documents = state.documents[knowledgeBaseId]
if (!documents) return state
const documentsCache = state.documents[knowledgeBaseId]
if (!documentsCache) return state
const updatedDocuments = documents.map((doc) =>
const updatedDocuments = documentsCache.documents.map((doc) =>
doc.id === documentId ? { ...doc, ...updates } : doc
)
return {
documents: {
...state.documents,
[knowledgeBaseId]: updatedDocuments,
[knowledgeBaseId]: {
...documentsCache,
documents: updatedDocuments,
},
},
}
})
@@ -677,7 +746,8 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
addPendingDocuments: (knowledgeBaseId: string, newDocuments: DocumentData[]) => {
set((state) => {
const existingDocuments = state.documents[knowledgeBaseId] || []
const existingDocumentsCache = state.documents[knowledgeBaseId]
const existingDocuments = existingDocumentsCache?.documents || []
const existingIds = new Set(existingDocuments.map((doc) => doc.id))
const uniqueNewDocuments = newDocuments.filter((doc) => !existingIds.has(doc.id))
@@ -689,15 +759,29 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
const updatedDocuments = [...existingDocuments, ...uniqueNewDocuments]
const documentsCache: DocumentsCache = {
documents: updatedDocuments,
pagination: {
...(existingDocumentsCache?.pagination || {
limit: 50,
offset: 0,
hasMore: false,
}),
total: updatedDocuments.length,
},
searchQuery: existingDocumentsCache?.searchQuery,
lastFetchTime: Date.now(),
}
return {
documents: {
...state.documents,
[knowledgeBaseId]: updatedDocuments,
[knowledgeBaseId]: documentsCache,
},
}
})
logger.info(
`Added ${newDocuments.filter((doc) => !get().documents[knowledgeBaseId]?.some((existing) => existing.id === doc.id)).length} pending documents for knowledge base: ${knowledgeBaseId}`
`Added ${newDocuments.filter((doc) => !get().documents[knowledgeBaseId]?.documents?.some((existing) => existing.id === doc.id)).length} pending documents for knowledge base: ${knowledgeBaseId}`
)
},
@@ -731,10 +815,10 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
removeDocument: (knowledgeBaseId: string, documentId: string) => {
set((state) => {
const documents = state.documents[knowledgeBaseId]
if (!documents) return state
const documentsCache = state.documents[knowledgeBaseId]
if (!documentsCache) return state
const updatedDocuments = documents.filter((doc) => doc.id !== documentId)
const updatedDocuments = documentsCache.documents.filter((doc) => doc.id !== documentId)
// Also clear chunks for the removed document
const newChunks = { ...state.chunks }
@@ -743,7 +827,10 @@ export const useKnowledgeStore = create<KnowledgeStore>((set, get) => ({
return {
documents: {
...state.documents,
[knowledgeBaseId]: updatedDocuments,
[knowledgeBaseId]: {
...documentsCache,
documents: updatedDocuments,
},
},
chunks: newChunks,
}

2
apps/sim/tools/gmail/read.ts
View File

@@ -31,7 +31,7 @@ export const gmailReadTool: ToolConfig<GmailReadParams, GmailToolResponse> = {
folder: {
type: 'string',
required: false,
visibility: 'user-or-llm',
visibility: 'user-only',
description: 'Folder/label to read emails from',
},
unreadOnly: {

121
apps/sim/tools/gmail/search.ts
View File

@@ -52,25 +52,77 @@ export const gmailSearchTool: ToolConfig<GmailSearchParams, GmailToolResponse> =
}),
},
transformResponse: async (response) => {
transformResponse: async (response, params) => {
const data = await response.json()
if (!response.ok) {
throw new Error(data.error?.message || 'Failed to search emails')
}
return {
success: true,
output: {
content: `Found ${data.messages?.length || 0} messages`,
metadata: {
results:
data.messages?.map((msg: any) => ({
if (!data.messages || data.messages.length === 0) {
return {
success: true,
output: {
content: 'No messages found matching your search query.',
metadata: {
results: [],
},
},
}
}
try {
// Fetch full message details for each result
const messagePromises = data.messages.map(async (msg: any) => {
const messageResponse = await fetch(`${GMAIL_API_BASE}/messages/${msg.id}?format=full`, {
headers: {
Authorization: `Bearer ${params?.accessToken || ''}`,
'Content-Type': 'application/json',
},
})
if (!messageResponse.ok) {
throw new Error(`Failed to fetch details for message ${msg.id}`)
}
return await messageResponse.json()
})
const messages = await Promise.all(messagePromises)
// Process all messages and create a summary
const processedMessages = messages.map(processMessageForSummary)
return {
success: true,
output: {
content: createMessagesSummary(processedMessages),
metadata: {
results: processedMessages.map((msg) => ({
id: msg.id,
threadId: msg.threadId,
})) || [],
subject: msg.subject,
from: msg.from,
date: msg.date,
snippet: msg.snippet,
})),
},
},
},
}
} catch (error: any) {
console.error('Error fetching message details:', error)
return {
success: true,
output: {
content: `Found ${data.messages.length} messages but couldn't retrieve all details: ${error.message || 'Unknown error'}`,
metadata: {
results: data.messages.map((msg: any) => ({
id: msg.id,
threadId: msg.threadId,
})),
},
},
}
}
},
@@ -87,3 +139,52 @@ export const gmailSearchTool: ToolConfig<GmailSearchParams, GmailToolResponse> =
return error.message || 'An unexpected error occurred while searching emails'
},
}
// Helper function to process a message for summary (without full content)
function processMessageForSummary(message: any): any {
if (!message || !message.payload) {
return {
id: message?.id || '',
threadId: message?.threadId || '',
subject: 'Unknown Subject',
from: 'Unknown Sender',
date: '',
snippet: message?.snippet || '',
}
}
const headers = message.payload.headers || []
const subject =
headers.find((h: any) => h.name.toLowerCase() === 'subject')?.value || 'No Subject'
const from = headers.find((h: any) => h.name.toLowerCase() === 'from')?.value || 'Unknown Sender'
const date = headers.find((h: any) => h.name.toLowerCase() === 'date')?.value || ''
return {
id: message.id,
threadId: message.threadId,
subject,
from,
date,
snippet: message.snippet || '',
}
}
// Helper function to create a summary of multiple messages
function createMessagesSummary(messages: any[]): string {
if (messages.length === 0) {
return 'No messages found.'
}
let summary = `Found ${messages.length} messages:\n\n`
messages.forEach((msg, index) => {
summary += `${index + 1}. Subject: ${msg.subject}\n`
summary += ` From: ${msg.from}\n`
summary += ` Date: ${msg.date}\n`
summary += ` Preview: ${msg.snippet}\n\n`
})
summary += `To read full content of a specific message, use the gmail_read tool with messageId: ${messages.map((m) => m.id).join(', ')}`
return summary
}