mirror of
https://github.com/simstudioai/sim.git
synced 2026-04-28 03:00:29 -04:00
239 lines
9.3 KiB
Plaintext
239 lines
9.3 KiB
Plaintext
Below is a high-level implementation guide for syncing our workflow state from localStorage (and our Zustand stores) to our Supabase/Postgres database using Drizzle ORM. We will combine three approaches:
|
||
1. Debounced Sync:
|
||
When the workflow state changes (e.g. blocks, edges, loops, etc.), we debounce the update so that many small changes are batched together into a single write. This minimizes rapid consecutive writes.
|
||
2. Periodic Sync (Auto-save):
|
||
We set up a timer (for example, every 30 seconds) to ensure that the state is synced—even if the debounce did not trigger (say, because changes stopped for a while).
|
||
3. Critical Events Sync (e.g. BeforeUnload):
|
||
We add an event listener (such as on beforeunload) to flush any unsaved changes when the user navigates away or closes the tab.
|
||
> Note on Security:
|
||
> To keep our implementation open source friendly and secure, we will not expose any database credentials or secrets on the client side. Instead, we’ll create an API endpoint (using Next.js API Routes) that will perform the actual database update using our Drizzle ORM connection. This pattern keeps our sensitive configuration on the server and ensures our client code only makes secure HTTP requests. We’re also careful to associate workflow records with authenticated users (via Better Auth) once we have the user flow in place.
|
||
---
|
||
|
||
Step-by-Step Plan
|
||
1. Define the Database Schema (if needed)
|
||
If you haven’t yet created a table to store workflows, you should create one via a migration. A possible table could look like:
|
||
Table Name: workflows
|
||
Columns:
|
||
id (text, primary key) – the workflow ID
|
||
user_id (text) – to associate with the current user (via Better Auth)
|
||
state (JSONB) – a JSON column that includes the parts of your state (e.g. blocks, edges, loops, lastSaved, etc.)
|
||
updated_at and created_at (timestamps)
|
||
> We’re using Supabase/Postgres, so ensure you create the migration and run it using Drizzle’s migration tools (or Supabase dashboard).
|
||
|
||
|
||
2. Create a DB Endpoint for Workflow Sync
|
||
Because the client must not talk directly to the database, create an API route (e.g. /api/workflows/sync) that accepts a workflow state update. This API route will:
|
||
Validate the incoming data.
|
||
Check the authenticated user (so that workflows are attached to a user).
|
||
Use Drizzle ORM to upsert (insert or update) the workflow row.
|
||
A simplified version might look like this:
|
||
```
|
||
import { NextResponse } from 'next/server'
|
||
import { db } from '@/db'
|
||
import { workflow } from '@/db/schema'
|
||
import { z } from 'zod'
|
||
|
||
// Define the schema for incoming data
|
||
const WorkflowSyncSchema = z.object({
|
||
id: z.string(),
|
||
userId: z.string(),
|
||
state: z.any(),
|
||
})
|
||
|
||
export async function POST(request: Request) {
|
||
try {
|
||
const body = await request.json()
|
||
const { id, userId, state } = WorkflowSyncSchema.parse(body)
|
||
|
||
// Upsert the workflow (using your preferred upsert method)
|
||
await db.insert(workflow).values({ id, userId, state, updatedAt: new Date() })
|
||
.onConflictDoUpdate({
|
||
target: [workflow.id],
|
||
set: { state, updatedAt: new Date() },
|
||
})
|
||
|
||
return NextResponse.json({ success: true })
|
||
} catch (error) {
|
||
console.error('Workflow sync error:', error)
|
||
return NextResponse.json({ error: 'Sync failed' }, { status: 500 })
|
||
}
|
||
}
|
||
```
|
||
> Security note:
|
||
> Ensure that authentication middleware (e.g. Better Auth) protects this endpoint. Do not trust client-sent user IDs without verification.
|
||
|
||
3. Implement the Client-Side Sync Functions
|
||
a. Debounced Sync Hook
|
||
Create a custom hook (for example, useDebouncedWorkflowSync) that watches your workflow state (using Zustand selectors) and uses a debounce function (like the one from lodash.debounce) to call your API endpoint.
|
||
```
|
||
import { useEffect } from 'react'
|
||
import debounce from 'lodash.debounce'
|
||
import { useWorkflowStore } from '@/stores/workflow/store'
|
||
import { useWorkflowRegistry } from '@/stores/workflow/registry/store'
|
||
|
||
export function useDebouncedWorkflowSync() {
|
||
const workflowState = useWorkflowStore((state) => ({
|
||
blocks: state.blocks,
|
||
edges: state.edges,
|
||
loops: state.loops,
|
||
lastSaved: state.lastSaved,
|
||
}))
|
||
const { activeWorkflowId } = useWorkflowRegistry()
|
||
|
||
useEffect(() => {
|
||
if (!activeWorkflowId) return
|
||
|
||
const syncWorkflow = async () => {
|
||
try {
|
||
await fetch('/api/workflows/sync', {
|
||
method: 'POST',
|
||
headers: {
|
||
'Content-Type': 'application/json',
|
||
},
|
||
body: JSON.stringify({
|
||
id: activeWorkflowId,
|
||
// Once you have authentication in place, set userId appropriately.
|
||
userId: 'current-authenticated-user-id',
|
||
state: workflowState,
|
||
}),
|
||
})
|
||
} catch (err) {
|
||
console.error('Debounced sync error:', err)
|
||
}
|
||
}
|
||
|
||
const debouncedSync = debounce(syncWorkflow, 2000)
|
||
debouncedSync()
|
||
|
||
return () => debouncedSync.cancel()
|
||
}, [
|
||
workflowState.blocks,
|
||
workflowState.edges,
|
||
workflowState.loops,
|
||
workflowState.lastSaved,
|
||
activeWorkflowId,
|
||
])
|
||
}
|
||
```
|
||
|
||
|
||
b. Periodic Sync Hook
|
||
Set up another hook that, on an interval (say, every 30 seconds), calls the same API endpoint. This ensures that even if the user pauses making changes, the latest state is still pushed to the database.
|
||
```
|
||
import { useEffect } from 'react'
|
||
import { useWorkflowStore } from '@/stores/workflow/store'
|
||
import { useWorkflowRegistry } from '@/stores/workflow/registry/store'
|
||
|
||
export function usePeriodicWorkflowSync(intervalMs = 30000) {
|
||
const workflowState = useWorkflowStore((state) => ({
|
||
blocks: state.blocks,
|
||
edges: state.edges,
|
||
loops: state.loops,
|
||
lastSaved: state.lastSaved,
|
||
}))
|
||
const { activeWorkflowId } = useWorkflowRegistry()
|
||
|
||
useEffect(() => {
|
||
if (!activeWorkflowId) return
|
||
|
||
const syncWorkflow = async () => {
|
||
try {
|
||
await fetch('/api/workflows/sync', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify({
|
||
id: activeWorkflowId,
|
||
userId: 'current-authenticated-user-id',
|
||
state: workflowState,
|
||
}),
|
||
})
|
||
} catch (err) {
|
||
console.error('Periodic sync error:', err)
|
||
}
|
||
}
|
||
|
||
const interval = setInterval(syncWorkflow, intervalMs)
|
||
|
||
return () => clearInterval(interval)
|
||
}, [workflowState, activeWorkflowId, intervalMs])
|
||
}
|
||
```
|
||
|
||
c. Critical Event Sync (On Unload)
|
||
Finally, add a hook that listens for the beforeunload event and immediately syncs any unsaved changes.
|
||
```
|
||
import { useEffect } from 'react'
|
||
import { useWorkflowStore } from '@/stores/workflow/store'
|
||
import { useWorkflowRegistry } from '@/stores/workflow/registry/store'
|
||
|
||
export function useSyncOnUnload() {
|
||
const workflowState = useWorkflowStore((state) => ({
|
||
blocks: state.blocks,
|
||
edges: state.edges,
|
||
loops: state.loops,
|
||
lastSaved: state.lastSaved,
|
||
}))
|
||
const { activeWorkflowId } = useWorkflowRegistry()
|
||
|
||
useEffect(() => {
|
||
const handleBeforeUnload = async () => {
|
||
if (!activeWorkflowId) return
|
||
|
||
try {
|
||
await fetch('/api/workflows/sync', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify({
|
||
id: activeWorkflowId,
|
||
userId: 'current-authenticated-user-id',
|
||
state: workflowState,
|
||
}),
|
||
// Use the keepalive option (if supported) to try flushing even during unload.
|
||
keepalive: true,
|
||
})
|
||
} catch (err) {
|
||
console.error('Sync on unload error:', err)
|
||
}
|
||
}
|
||
|
||
window.addEventListener('beforeunload', handleBeforeUnload)
|
||
return () => window.removeEventListener('beforeunload', handleBeforeUnload)
|
||
}, [workflowState, activeWorkflowId])
|
||
}
|
||
```
|
||
|
||
|
||
> Fallback for Unsynced Changes:
|
||
> Since we are already persisting the state in localStorage (and, in-memory, via Zustand), even if one of these sync methods fails, nothing is lost. When the application reloads, we can rehydrate the workflow state from localStorage.
|
||
4. Integrate the Hooks into Your Application
|
||
Within your main workflow component (or a top-level component that deals with workflow state), call these hooks:
|
||
```
|
||
import { useEffect } from 'react'
|
||
import { useDebouncedWorkflowSync } from '@/hooks/useDebouncedWorkflowSync'
|
||
import { usePeriodicWorkflowSync } from '@/hooks/usePeriodicWorkflowSync'
|
||
import { useSyncOnUnload } from '@/hooks/useSyncOnUnload'
|
||
|
||
export default function WorkflowContent() {
|
||
// Your workflow component code...
|
||
|
||
// Start the syncing hooks:
|
||
useDebouncedWorkflowSync()
|
||
usePeriodicWorkflowSync()
|
||
useSyncOnUnload()
|
||
|
||
// ...
|
||
}
|
||
```
|
||
|
||
---
|
||
Summary
|
||
Define / Create the DB Table: Create a workflow table on Supabase/Postgres and use Drizzle ORM (with proper migrations) to manage this schema.
|
||
API Endpoint: Build a secure API route (/api/workflows/sync) that uses Drizzle ORM to upsert workflow state. Secure access by verifying the authenticated user.
|
||
Client Sync Hooks:
|
||
Debounced Sync: Batches rapid changes.
|
||
Periodic Sync: Ensures regular saves.
|
||
BeforeUnload Sync: Catches any unsaved changes as the user leaves.
|
||
Local Fallback: Since our state is preserved in localStorage (and memory) the application is resilient even if a write fails.
|
||
This approach follows best practices, is secure (by keeping sensitive operations server-side), and aligns well with Next.js, Zustand, and your current codebase structure.
|
||
---
|
||
Let me know if you’d like to move on to implementing one or more of these parts in code. |