Files
AutoGPT/autogpt_platform/frontend
Nicholas Tindle 901e9eba5d feat(frontend): Enhanced output rendering system for agent runs (#10819)
## Summary
Introduces a modular, extensible output renderer system supporting
multiple content types (text, code, images, videos, JSON, markdown) for
agent run outputs. The system includes smart clipboard operations,
concatenated downloads, and rich markdown rendering with LaTeX math and
video embedding support.

## Changes 🏗️

### Core Output Rendering System
- **Added extensible renderer architecture**
(`output-renderers/types.ts`)
  - Plugin-based system with priority ordering
  - Registry pattern for automatic renderer selection
  - Support for custom metadata and MIME types

### Output Renderers
- **TextRenderer**: Plain text with proper formatting and line breaks
- **CodeRenderer**: Syntax-highlighted code blocks with language
detection
- **JSONRenderer**: Collapsible, formatted JSON with syntax highlighting
- **ImageRenderer**: Image display with support for URLs, data URIs, and
file uploads
- **VideoRenderer**: Embedded video player for YouTube, Vimeo, and
direct video files
- **MarkdownRenderer**: Rich markdown with:
  - GitHub Flavored Markdown (tables, task lists, strikethrough)
- LaTeX math rendering via KaTeX (inline `$...$` and display `$$...$$`)
  - Syntax highlighting via highlight.js
  - Video embedding (YouTube/Vimeo URLs auto-convert to embeds)
  - Clickable heading anchors
  - Dark mode support

### User Interface Components
- **OutputItem**: Individual output display with renderer selection
- **OutputActions**: Hover-based action buttons for:
  - Copy to clipboard with smart MIME type detection
- Download with intelligent concatenation (text files merge, binaries
separate)
  - Share functionality (placeholder for future implementation)
- **AgentRunOutputView**: Main output view component with feature flag
integration

### Clipboard & Download Features
- Smart clipboard operations using native ClipboardItem API
- MIME type detection and browser capability checking
- Fallback strategies for unsupported content types
- Concatenated downloads for text-based outputs
- Individual downloads for binary content

### Feature Flag Integration
- Added `ENABLE_ENHANCED_OUTPUT_HANDLING` flag to LaunchDarkly
- Backwards compatible with existing output display
- Graceful fallback for disabled feature flag

### Styling & UX
- Max width constraints (950px card, 660px content)
- Hover-based action buttons for clean interface
- Dark mode support across all renderers
- Responsive design for various content types
- Loading states and error handling

## Test Plan 📋

### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:

### Test Scenarios:
- [x] **Basic Output Rendering**
  - [x] Execute agent with text output - verify proper formatting
  - [x] Execute agent with JSON output - verify collapsible tree view
  - [x] Execute agent with code output - verify syntax highlighting
  
- [x] **Rich Content**
  - [x] Test markdown rendering with headers, lists, tables
  - [x] Test LaTeX math expressions (inline and display)
  - [x] Test code blocks within markdown
  - [x] Test task lists and strikethrough
  
- [x] **Media Handling**
  - [x] Upload and display PNG/JPEG images
  - [x] Test video URL embedding (YouTube/Vimeo)
  - [x] Test direct video file playback
  
- [x] **Clipboard Operations**
  - [x] Copy plain text output
  - [x] Copy formatted code
  - [x] Copy JSON data
  - [x] Copy markdown content
  - [x] Verify fallback for unsupported MIME types
  
- [x] **Download Functionality**
  - [x] Download single text output
  - [x] Download multiple text outputs (verify concatenation)
  - [x] Download mixed content (verify separate files)
  - [x] Download images and binary content
  
- [x] **Feature Flag**
  - [x] Enable flag - verify enhanced rendering
  - [x] Disable flag - verify fallback to original view
  - [x] Check backwards compatibility
 
  
- [x] **Edge Cases**
  - [x] Large JSON objects (performance)
  - [x] Very long text outputs
  - [x] Mixed content types in single run
  - [x] Malformed markdown
  - [x] Invalid video URLs

## Dependencies Added
- `react-markdown` (9.0.3) - Already present
- `remark-gfm` (4.0.1) - GitHub Flavored Markdown
- `remark-math` (6.0.0) - LaTeX math support
- `rehype-katex` (7.0.1) - Math rendering
- `katex` (0.16.22) - Math typesetting
- `rehype-highlight` (7.0.2) - Syntax highlighting
- `highlight.js` (11.11.1) - Highlighting library
- `rehype-slug` (6.0.0) - Heading anchors
- `rehype-autolink-headings` (7.1.0) - Clickable headings

## Notes
- Mermaid diagram support was attempted but removed due to compatibility
issues
- Share functionality is stubbed out for future implementation
- PNG file upload rendering issue has logging in place for debugging
- All components follow existing UI patterns and use Tailwind CSS

## Screenshots
<img width="1656" height="1250" alt="image"
src="https://github.com/user-attachments/assets/af7542fe-db89-4521-aaf5-19e33d48a409"
/>
## Related Issues
- Implements SECRT-1209

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Tindle <ntindle@users.noreply.github.com>
2025-09-04 13:37:26 +00:00
..
2025-06-05 15:02:27 +00:00

This is the frontend for AutoGPT's next generation

🧢 Getting Started

This project uses pnpm as the package manager via corepack. Corepack is a Node.js tool that automatically manages package managers without requiring global installations.

Prerequisites

Make sure you have Node.js 16.10+ installed. Corepack is included with Node.js by default.

⚠️ Migrating from yarn

This project was previously using yarn1, make sure to clean up the old files if you set it up previously with yarn:

rm -f yarn.lock && rm -rf node_modules

Then follow the setup steps below.

Setup

1. Enable corepack (run this once on your system):

corepack enable

This enables corepack to automatically manage pnpm based on the packageManager field in package.json.

2. Install dependencies:

pnpm i

3. Start the development server:

Running the Front-end & Back-end separately

We recommend this approach if you are doing active development on the project. First spin up the Back-end:

# on `autogpt_platform`
docker compose --profile local up deps_backend -d
# on `autogpt_platform/backend`
poetry run app

Then start the Front-end:

# on `autogpt_platform/frontend`
pnpm dev

Open http://localhost:3000 with your browser to see the result. If the server starts on http://localhost:3001 it means the Front-end is already running via Docker. You have to kill the container then or do docker compose down.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

Running both the Front-end and Back-end via Docker

If you run:

# on `autogpt_platform`
docker compose up -d

It will spin up the Back-end and Front-end via Docker. The Front-end will start on port 3000. This might not be what you want when actively contributing to the Front-end as you won't have direct/easy access to the Next.js dev server.

Subsequent Runs

For subsequent development sessions, you only need to run:

pnpm dev

Every time a new Front-end dependency is added by you or others, you will need to run pnpm i to install the new dependencies.

Available Scripts

  • pnpm dev - Start development server
  • pnpm build - Build for production
  • pnpm start - Start production server
  • pnpm lint - Run ESLint and Prettier checks
  • pnpm format - Format code with Prettier
  • pnpm types - Run TypeScript type checking
  • pnpm test - Run Playwright tests
  • pnpm test-ui - Run Playwright tests with UI
  • pnpm fetch:openapi - Fetch OpenAPI spec from backend
  • pnpm generate:api-client - Generate API client from OpenAPI spec
  • pnpm generate:api - Fetch OpenAPI spec and generate API client

This project uses next/font to automatically optimize and load Inter, a custom Google Font.

🔄 Data Fetching Strategy

Note

You don't need to run the OpenAPI commands below to run the Front-end. You will only need to run them when adding or modifying endpoints on the Backend API and wanting to use those on the Frontend.

This project uses an auto-generated API client powered by Orval, which creates type-safe API clients from OpenAPI specifications.

How It Works

  1. Backend Requirements: Each API endpoint needs a summary and tag in the OpenAPI spec
  2. Operation ID Generation: FastAPI generates operation IDs using the pattern {method}{tag}{summary}
  3. Spec Fetching: The OpenAPI spec is fetched from http://localhost:8006/openapi.json and saved to the frontend
  4. Spec Transformation: The OpenAPI spec is cleaned up using a custom transformer (see autogpt_platform/frontend/src/app/api/transformers)
  5. Client Generation: Auto-generated client includes TypeScript types, API endpoints, and Zod schemas, organized by tags

API Client Commands

# Fetch OpenAPI spec from backend and generate client
pnpm generate:api

# Only fetch the OpenAPI spec
pnpm fetch:openapi

# Only generate the client (after spec is fetched)
pnpm generate:api-client

Using the Generated Client

The generated client provides React Query hooks for both queries and mutations:

Queries (GET requests)

import { useGetV1GetNotificationPreferences } from "@/app/api/__generated__/endpoints/auth/auth";

const { data, isLoading, isError } = useGetV1GetNotificationPreferences({
  query: {
    select: (res) => res.data,
    // Other React Query options
  },
});

Mutations (POST, PUT, DELETE requests)

import { useDeleteV2DeleteStoreSubmission } from "@/app/api/__generated__/endpoints/store/store";
import { getGetV2ListMySubmissionsQueryKey } from "@/app/api/__generated__/endpoints/store/store";
import { useQueryClient } from "@tanstack/react-query";

const queryClient = useQueryClient();

const { mutateAsync: deleteSubmission } = useDeleteV2DeleteStoreSubmission({
  mutation: {
    onSuccess: () => {
      // Invalidate related queries to refresh data
      queryClient.invalidateQueries({
        queryKey: getGetV2ListMySubmissionsQueryKey(),
      });
    },
  },
});

// Usage
await deleteSubmission({
  submissionId: submission_id,
});

Server Actions

For server-side operations, you can also use the generated client functions directly:

import { postV1UpdateNotificationPreferences } from "@/app/api/__generated__/endpoints/auth/auth";

// In a server action
const preferences = {
  email: "user@example.com",
  preferences: {
    AGENT_RUN: true,
    ZERO_BALANCE: false,
    // ... other preferences
  },
  daily_limit: 0,
};

await postV1UpdateNotificationPreferences(preferences);

Server-Side Prefetching

For server-side components, you can prefetch data on the server and hydrate it in the client cache. This allows immediate access to cached data when queries are called:

import { getQueryClient } from "@/lib/tanstack-query/getQueryClient";
import {
  prefetchGetV2ListStoreAgentsQuery,
  prefetchGetV2ListStoreCreatorsQuery
} from "@/app/api/__generated__/endpoints/store/store";
import { HydrationBoundary, dehydrate } from "@tanstack/react-query";

// In your server component
const queryClient = getQueryClient();

await Promise.all([
  prefetchGetV2ListStoreAgentsQuery(queryClient, {
    featured: true,
  }),
  prefetchGetV2ListStoreAgentsQuery(queryClient, {
    sorted_by: "runs",
  }),
  prefetchGetV2ListStoreCreatorsQuery(queryClient, {
    featured: true,
    sorted_by: "num_agents",
  }),
]);

return (
  <HydrationBoundary state={dehydrate(queryClient)}>
    <MainMarkeplacePage />
  </HydrationBoundary>
);

This pattern improves performance by serving pre-fetched data from the server while maintaining the benefits of client-side React Query features.

Configuration

The Orval configuration is located in autogpt_platform/frontend/orval.config.ts. It generates two separate clients:

  1. autogpt_api_client: React Query hooks for client-side data fetching
  2. autogpt_zod_schema: Zod schemas for validation

For more details, see the Orval documentation or check the configuration file.

🚩 Feature Flags

This project uses LaunchDarkly for feature flags, allowing us to control feature rollouts and A/B testing.

Using Feature Flags

Check if a feature is enabled

import { Flag, useGetFlag } from "@/services/feature-flags/use-get-flag";

function MyComponent() {
  const isAgentActivityEnabled = useGetFlag(Flag.AGENT_ACTIVITY);

  if (!isAgentActivityEnabled) {
    return null; // Hide feature
  }

  return <div>Feature is enabled!</div>;
}

Protect entire components

import { withFeatureFlag } from "@/services/feature-flags/with-feature-flag";

const MyFeaturePage = withFeatureFlag(MyPageComponent, "my-feature-flag");

Testing with Feature Flags

For local development or running Playwright tests locally, use mocked feature flags by setting NEXT_PUBLIC_PW_TEST=true in your .env file. This bypasses LaunchDarkly and uses the mock values defined in the code.

Adding New Flags

  1. Add the flag to the Flag enum in use-get-flag.ts
  2. Add the flag type to FlagValues type
  3. Add mock value to mockFlags for testing
  4. Configure the flag in LaunchDarkly dashboard

🚚 Deploy

TODO

📙 Storybook

Storybook is a powerful development environment for UI components. It allows you to build UI components in isolation, making it easier to develop, test, and document your components independently from your main application.

Purpose in the Development Process

  1. Component Development: Develop and test UI components in isolation.
  2. Visual Testing: Easily spot visual regressions.
  3. Documentation: Automatically document components and their props.
  4. Collaboration: Share components with your team or stakeholders for feedback.

How to Use Storybook

  1. Start Storybook: Run the following command to start the Storybook development server:

    pnpm storybook
    

    This will start Storybook on port 6006. Open http://localhost:6006 in your browser to view your component library.

  2. Build Storybook: To build a static version of Storybook for deployment, use:

    pnpm build-storybook
    
  3. Running Storybook Tests: Storybook tests can be run using:

    pnpm test-storybook
    
  4. Writing Stories: Create .stories.tsx files alongside your components to define different states and variations of your components.

By integrating Storybook into our development workflow, we can streamline UI development, improve component reusability, and maintain a consistent design system across the project.

🔭 Tech Stack

Core Framework & Language

  • Next.js - React framework with App Router
  • React - UI library for building user interfaces
  • TypeScript - Typed JavaScript for better developer experience

Styling & UI Components

Development & Testing

Backend & Services

  • Supabase - Backend-as-a-Service (database, auth, storage)
  • Sentry - Error monitoring and performance tracking

Package Management

  • pnpm - Fast, disk space efficient package manager
  • Corepack - Node.js package manager management

Additional Libraries

Development Tools

  • NEXT_PUBLIC_REACT_QUERY_DEVTOOL - Enable React Query DevTools. Set to true to enable.