- name:Run integration test evaluation for DelegatorAgent (Haiku)
env:
SANDBOX_FORCE_REBUILD_RUNTIME:True
run:|
poetry run ./evaluation/integration_tests/scripts/run_infer.sh llm.eval HEAD DelegatorAgent '' 30 $N_PROCESSES "t01_fix_simple_typo,t02_add_bash_hello" 'delegator_haiku_run'
# Find and export the delegator test results
REPORT_FILE_DELEGATOR_HAIKU=$(find evaluation/evaluation_outputs/outputs/integration_tests/DelegatorAgent/*haiku*_maxiter_30_N* -name "report.md" -type f | head -n 1)
- name:Run integration test evaluation for DelegatorAgent (DeepSeek)
- name:Run integration test evaluation for VisualBrowsingAgent (DeepSeek)
env:
SANDBOX_FORCE_REBUILD_RUNTIME:True
run:|
poetry run ./evaluation/integration_tests/scripts/run_infer.sh llm.eval HEAD DelegatorAgent '' 30 $N_PROCESSES "t01_fix_simple_typo,t02_add_bash_hello" 'delegator_deepseek_run'
poetry run ./evaluation/integration_tests/scripts/run_infer.sh llm.eval HEAD VisualBrowsingAgent '' 15 $N_PROCESSES "t05_simple_browsing,t06_github_pr_browsing.py" 'visualbrowsing_deepseek_run'
# Find and export the delegator test results
REPORT_FILE_DELEGATOR_DEEPSEEK=$(find evaluation/evaluation_outputs/outputs/integration_tests/DelegatorAgent/deepseek*_maxiter_30_N* -name "report.md" -type f | head -n 1)
# Find and export the visual browsing agent test results
REPORT_FILE_VISUALBROWSING_DEEPSEEK=$(find evaluation/evaluation_outputs/outputs/integration_tests/VisualBrowsingAgent/deepseek*_maxiter_15_N* -name "report.md" -type f | head -n 1)
cd evaluation/evaluation_outputs/outputs # Change to the outputs directory
tar -czvf ../../../integration_tests_${TIMESTAMP}.tar.gz integration_tests/CodeActAgent/* integration_tests/DelegatorAgent/* # Only include the actual result directories
tar -czvf ../../../integration_tests_${TIMESTAMP}.tar.gz integration_tests/CodeActAgent/* integration_tests/VisualBrowsingAgent/* # Only include the actual result directories
The core AI entity in OpenHands that can perform software development tasks by interacting with tools, browsing the web, and modifying code.
#### Agent Controller
A component that manages the agent's lifecycle, handles its state, and coordinates interactions between the agent and various tools.
#### Agent Delegation
The ability of an agent to hand off specific tasks to other specialized agents for better task completion.
#### Agent Hub
A central registry of different agent types and their capabilities, allowing for easy agent selection and instantiation.
#### Agent Skill
A specific capability or function that an agent can perform, such as file manipulation, web browsing, or code editing.
#### Agent State
The current context and status of an agent, including its memory, active tools, and ongoing tasks.
#### CodeAct Agent
[A generalist agent in OpenHands](https://arxiv.org/abs/2407.16741) designed to perform tasks by editing and executing code.
### Browser
A system for web-based interactions and tasks.
#### Browser Gym
A testing and evaluation environment for browser-based agent interactions and tasks.
#### Web Browser Tool
A tool that enables agents to interact with web pages and perform web-based tasks.
### Commands
Terminal and execution related functionality.
#### Bash Session
A persistent terminal session that maintains state and history for bash command execution.
This uses tmux under the hood.
### Configuration
System-wide settings and options.
#### Agent Configuration
Settings that define an agent's behavior, capabilities, and limitations, including available tools and runtime settings.
#### Configuration Options
Settings that control various aspects of OpenHands behavior, including runtime, security, and agent settings.
#### LLM Config
Configuration settings for language models used by agents, including model selection and parameters.
#### LLM Draft Config
Settings for draft mode operations with language models, typically used for faster, lower-quality responses.
#### Runtime Configuration
Settings that define how the runtime environment should be set up and operated.
#### Security Options
Configuration settings that control security features and restrictions.
### Conversation
A sequence of interactions between a user and an agent, including messages, actions, and their results.
#### Conversation Info
Metadata about a conversation, including its status, participants, and timeline.
#### Conversation Manager
A component that handles the creation, storage, and retrieval of conversations.
#### Conversation Metadata
Additional information about conversations, such as tags, timestamps, and related resources.
#### Conversation Status
The current state of a conversation, including whether it's active, completed, or failed.
#### Conversation Store
A storage system for maintaining conversation history and related data.
### Events
#### Event
Every Conversation comprises a series of Events. Each Event is either an Action or an Observation.
#### Event Stream
A continuous flow of events that represents the ongoing activities and interactions in the system.
#### Action
A specific operation or command that an agent executes through available tools, such as running a command or editing a file.
#### Observation
The response or result returned by a tool after an agent's action, providing feedback about the action's outcome.
### Interface
Different ways to interact with OpenHands.
#### CLI Mode
A command-line interface mode for interacting with OpenHands agents without a graphical interface.
#### GUI Mode
A graphical user interface mode for interacting with OpenHands agents through a web interface.
#### Headless Mode
A mode of operation where OpenHands runs without a user interface, suitable for automation and scripting.
### Agent Memory
The system that decides which parts of the Event Stream (i.e. the conversation history) should be passed into each LLM prompt.
#### Memory Store
A storage system for maintaining agent memory and context across sessions.
#### Condenser
A component that processes and summarizes conversation history to maintain context while staying within token limits.
#### Truncation
A very simple Condenser strategy. Reduces conversation history or content to stay within token limits.
### Microagent
A specialized prompt that enhances OpenHands with domain-specific knowledge, repository-specific context, and task-specific workflows.
#### Microagent Registry
A central repository of available microagents and their configurations.
#### Public Microagent
A general-purpose microagent available to all OpenHands users, triggered by specific keywords.
#### Repository Microagent
A type of microagent that provides repository-specific context and guidelines, stored in the `.openhands/microagents/` directory.
### Prompt
Components for managing and processing prompts.
#### Prompt Caching
A system for caching and reusing common prompts to improve performance.
#### Prompt Manager
A component that handles the loading, processing, and management of prompts used by agents, including microagents.
#### Response Parsing
The process of interpreting and structuring responses from language models and tools.
### Runtime
The execution environment where agents perform their tasks, which can be local, remote, or containerized.
#### Action Execution Server
A REST API that receives agent actions (e.g. bash commands, python code, browsing actions), executes them in the runtime environment, and returns the results.
#### Action Execution Client
A component that handles the execution of actions in the runtime environment, managing the communication between the agent and the runtime.
#### Docker Runtime
A containerized runtime environment that provides isolation and reproducibility for agent operations.
#### E2B Runtime
A specialized runtime environment built on E2B for secure and isolated code execution.
#### Local Runtime
A runtime environment that executes on the local machine, suitable for development and testing.
#### Modal Runtime
A runtime environment built on Modal for scalable and distributed agent operations.
#### Remote Runtime
A sandboxed environment that executes code and commands remotely, providing isolation and security for agent operations.
#### Runtime Builder
A component that builds a Docker image for the Action Execution Server based on a user-specified base image.
### Security
Security-related components and features.
#### Security Analyzer
A component that checks agent actions for potential security risks.
@@ -113,6 +113,20 @@ individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
### Slack and Discord Etiquettes
These Slack and Discord etiquette guidelines are designed to foster an inclusive, respectful, and productive environment for all community members. By following these best practices, we ensure effective communication and collaboration while minimizing disruptions. Let’s work together to build a supportive and welcoming community!
- Communicate respectfully and professionally, avoiding sarcasm or harsh language, and remember that tone can be difficult to interpret in text.
- Use threads for specific discussions to keep channels organized and easier to follow.
- Tag others only when their input is critical or urgent, and use @here, @channel or @everyone sparingly to minimize disruptions.
- Be patient, as open-source contributors and maintainers often have other commitments and may need time to respond.
- Post questions or discussions in the most relevant channel (e.g., for [slack - #general](https://app.slack.com/client/T06P212QSEA/C06P5NCGSFP) for general topics, [slack - #questions](https://openhands-ai.slack.com/archives/C06U8UTKSAD) for queries/questions, [discord - #general](https://discord.com/channels/1222935860639563850/1222935861386018885)).
- When asking for help or raising issues, include necessary details like links, screenshots, or clear explanations to provide context.
- Keep discussions in public channels whenever possible to allow others to benefit from the conversation, unless the matter is sensitive or private.
- Always adhere to [our standards](https://github.com/All-Hands-AI/OpenHands/blob/main/CODE_OF_CONDUCT.md#our-standards) to ensure a welcoming and collaborative environment.
- If you choose to mute a channel, consider setting up alerts for topics that still interest you to stay engaged. For Slack, Go to Settings → Notifications → My Keywords to add specific keywords that will notify you when mentioned. For example, if you're here for discussions about LLMs, mute the channel if it’s too busy, but set notifications to alert you only when “LLMs” appears in messages. Also for Discord, go to the channel notifications and choose the option that best describes your need.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
@@ -5,7 +5,7 @@ Otherwise, you can clone the OpenHands project directly.
## Start the Server for Development
### 1. Requirements
* Linux, Mac OS, or [WSL on Windows](https://learn.microsoft.com/en-us/windows/wsl/install) [Ubuntu <= 22.04]
* Linux, Mac OS, or [WSL on Windows](https://learn.microsoft.com/en-us/windows/wsl/install) [Ubuntu >= 22.04]
* [Docker](https://docs.docker.com/engine/install/) (For those on MacOS, make sure to allow the default Docker socket to be used from advanced settings!)
These are the procedures and guidelines on how issues are triaged in this repo by the maintainers.
## General
*Most issues must be tagged with **enhancement** or **bug**.
* Issues may be tagged with what it relates to (**backend**, **frontend**, **agent quality**, etc.).
*All issues must be tagged with **enhancement**, **bug** or **troubleshooting/help**.
* Issues may be tagged with what it relates to (**agent quality**, **frontend**, **resolver**, etc.).
## Severity
* **Low**: Minor issues or affecting single user.
* **Medium**: Affecting multiple users.
* **High**: High visibility issues or affecting many users.
* **Critical**: Affecting all users or potential security issues.
## Effort
@@ -18,8 +19,14 @@ These are the procedures and guidelines on how issues are triaged in this repo b
## Not Enough Information
* User is asked to provide more information (logs, how to reproduce, etc.) when the issue is not clear.
* If an issue is unclear and the author does not provide more information or respond to a request, the issue may be closed as **not planned** (Usually after a week).
* If an issue is unclear and the author does not provide more information or respond to a request,
the issue may be closed as **not planned** (Usually after a week).
## Multiple Requests/Fixes in One Issue
* These issues will be narrowed down to one request/fix so the issue is more easily tracked and fixed.
* Issues may be broken down into multiple issues if required.
## Stale and Auto Closures
* In order to keep a maintainable backlog, issues that have no activity within 30 days are automatically marked as **Stale**.
* If issues marked as **Stale** continue to have no activity for 7 more days, they will automatically be closed as not planned.
* Issues may be reopened by maintainers if deemed important.
You'll find OpenHands running at [http://localhost:3000](http://localhost:3000)!
@@ -69,7 +69,7 @@ run OpenHands in a scriptable [headless mode](https://docs.all-hands.dev/modules
interact with it via a [friendly CLI](https://docs.all-hands.dev/modules/usage/how-to/cli-mode),
or run it on tagged issues with [a github action](https://docs.all-hands.dev/modules/usage/how-to/github-action).
Visit [Installation](https://docs.all-hands.dev/modules/usage/installation) for more information and setup instructions.
Visit [Running OpenHands](https://docs.all-hands.dev/modules/usage/installation) for more information and setup instructions.
> [!CAUTION]
> OpenHands is meant to be run by a single user on their local workstation.
@@ -96,7 +96,7 @@ troubleshooting resources, and advanced configuration options.
OpenHands is a community-driven project, and we welcome contributions from everyone. We do most of our communication
through Slack, so this is the best place to start, but we also are happy to have you contact us on Discord or Github:
- [Join our Slack workspace](https://join.slack.com/t/openhands-ai/shared_invite/zt-2wkh4pklz-w~h_DVDtEe9H5kyQlcNxVw) - Here we talk about research, architecture, and future development.
- [Join our Slack workspace](https://join.slack.com/t/openhands-ai/shared_invite/zt-2ngejmfw6-9gW4APWOC9XUp1n~SiQ6iw) - Here we talk about research, architecture, and future development.
- [Join our Discord server](https://discord.gg/ESHStjSjD4) - This is a community-run server for general discussion, questions, and feedback.
- [Read or post Github Issues](https://github.com/All-Hands-AI/OpenHands/issues) - Check out the issues we're working on, or add your own ideas.
#- SANDBOX_USER_ID=${SANDBOX_USER_ID:-1234} # enable this only if you want a specific non-root sandbox user but you will have to manually adjust permissions of openhands-state for this user
Le Runtime EventStream d'OpenHands est le composant principal qui permet l'exécution sécurisée et flexible des actions des agents d'IA.
Le Runtime Docker d'OpenHands est le composant principal qui permet l'exécution sécurisée et flexible des actions des agents d'IA.
Il crée un environnement en bac à sable (sandbox) en utilisant Docker, où du code arbitraire peut être exécuté en toute sécurité sans risquer le système hôte.
## Pourquoi avons-nous besoin d'un runtime en bac à sable ?
> Assurez-vous que ```sandbox_base_container_image``` est défini sur le nom de votre image personnalisée précédente.
> Assurez-vous que ```base_container_image``` est défini sur le nom de votre image personnalisée précédente.
## Exécution
@@ -82,20 +83,17 @@ dockerfile_content = (
## Dépannage / Erreurs
### Erreur: ```useradd: UID 1000 est non unique```
Si vous voyez cette erreur dans la sortie de la console, il s'agit du fait que OpenHands essaie de créer le utilisateur openhands dans le sandbox avec un ID d'utilisateur de 1000, cependant cet ID d'utilisateur est déjà utilisé dans l'image (pour une raison inconnue). Pour résoudre ce problème, changez la valeur du champ sandbox_user_id dans le fichier config.toml en une valeur différente:
Si vous voyez cette erreur dans la sortie de la console, il s'agit du fait que OpenHands essaie de créer le utilisateur openhands dans le sandbox avec un ID d'utilisateur de 1000, cependant cet ID d'utilisateur est déjà utilisé dans l'image (pour une raison inconnue). Pour résoudre ce problème, changez la valeur du champ user_id dans le fichier config.toml en une valeur différente:
Si vous voyez un message d'erreur indiquant que le port est utilisé ou indisponible, essayez de supprimer toutes les containers docker en cours d'exécution (exécutez `docker ps` et `docker rm` des containers concernés) puis ré-exécutez ```make run```
## Discuter
Pour d'autres problèmes ou questions rejoignez le [Slack](https://join.slack.com/t/openhands-ai/shared_invite/zt-2wkh4pklz-w~h_DVDtEe9H5kyQlcNxVw) ou le [Discord](https://discord.gg/ESHStjSjD4) et demandez!
@@ -44,12 +44,13 @@ Tout d'abord, assurez-vous de pouvoir exécuter OpenHands en suivant les instruc
### Spécifier l'Image de Base du Sandbox
Dans le fichier `config.toml` dans le répertoire OpenHands, définissez `sandbox_base_container_image` sur l'image que vous souhaitez utiliser. Cela peut être une image que vous avez déjà extraite ou une que vous avez construite :
Dans le fichier `config.toml` dans le répertoire OpenHands, définissez `base_container_image` sur l'image que vous souhaitez utiliser. Cela peut être une image que vous avez déjà extraite ou une que vous avez construite :
Vous pouvez également exécuter OpenHands en mode [headless scriptable](https://docs.all-hands.dev/modules/usage/how-to/headless-mode), en tant que [CLI interactive](https://docs.all-hands.dev/modules/usage/how-to/cli-mode), ou en utilisant l'[Action GitHub OpenHands](https://docs.all-hands.dev/modules/usage/how-to/github-action).
The OpenHands EventStream Runtime is the core component that enables secure and flexible execution of AI agent's action.
The OpenHands Docker Runtime is the core component that enables secure and flexible execution of AI agent's action.
It creates a sandboxed environment using Docker, where arbitrary code can be run safely without risking the host system.
## Why do we need a sandboxed runtime?
@@ -54,14 +54,13 @@ graph TD
6. Action Execution: The runtime client receives actions from the backend, executes them in the sandboxed environment, and sends back observations
7. Observation Return: The action execution server sends execution results back to the OpenHands backend as observations
The role of the client:
- It acts as an intermediary between the OpenHands backend and the sandboxed environment
- It executes various types of actions (shell commands, file operations, Python code, etc.) safely within the container
- It manages the state of the sandboxed environment, including the current working directory and loaded plugins
- It formats and returns observations to the backend, ensuring a consistent interface for processing results
## How OpenHands builds and maintains OH Runtime images
OpenHands' approach to building and managing runtime images ensures efficiency, consistency, and flexibility in creating and maintaining Docker images for both production and development environments.
@@ -78,16 +77,15 @@ Tags may be in one of 2 formats:
@@ -55,6 +55,11 @@ The core configuration options are defined in the `[core]` section of the `confi
- Default: `"./trajectories"`
- Description: Path to store trajectories (can be a folder or a file). If it's a folder, the trajectories will be saved in a file named with the session id name and .json extension, in that folder.
-`replay_trajectory_path`
- Type: `str`
- Default: `""`
- Description: Path to load a trajectory and replay. If given, must be a path to the trajectory file in JSON format. The actions in the trajectory file would be replayed first before any user instruction is executed.
### File Store
-`file_store_path`
- Type: `str`
@@ -121,7 +126,7 @@ The core configuration options are defined in the `[core]` section of the `confi
-`runtime`
- Type: `str`
- Default: `"eventstream"`
- Default: `"docker"`
- Description: Runtime environment
-`default_agent`
@@ -192,21 +197,6 @@ For development setups, you can also define custom named LLM configurations. See
- Default: `""`
- Description: Custom LLM provider
### Embeddings
-`embedding_base_url`
- Type: `str`
- Default: `""`
- Description: Embedding API base URL
-`embedding_deployment_name`
- Type: `str`
- Default: `""`
- Description: Embedding deployment name
-`embedding_model`
- Type: `str`
- Default: `"local"`
- Description: Embedding model to use
### Message Handling
-`max_message_chars`
@@ -291,23 +281,6 @@ For development setups, you can also define custom named LLM configurations. See
The agent configuration options are defined in the `[agent]` and `[agent.<agent_name>]` sections of the `config.toml` file.
### Microagent Configuration
-`micro_agent_name`
- Type: `str`
- Default: `""`
- Description: Name of the micro agent to use for this agent
### Memory Configuration
-`memory_enabled`
- Type: `bool`
- Default: `false`
- Description: Whether long-term memory (embeddings) is enabled
-`memory_max_threads`
- Type: `int`
- Default: `3`
- Description: The maximum number of threads indexing at the same time for embeddings
### LLM Configuration
-`llm_config`
- Type: `str`
@@ -335,8 +308,18 @@ The agent configuration options are defined in the `[agent]` and `[agent.<agent_
- Default: `false`
- Description: Whether Jupyter is enabled in the action space
-`enable_search_engine`
- Type: `bool`
- Default: `false`
- Description: Whether the search engine tool is enabled in the action space. See [Search Configuration](./search/search-configuration.md) for details.
-`enable_history_truncation`
- Type: `bool`
- Default: `true`
- Description: Whether history should be truncated to continue the session when hitting LLM context length limit
### Microagent Usage
-`use_microagents`
-`enable_prompt_extensions`
- Type: `bool`
- Default: `true`
- Description: Whether to use microagents at all
@@ -375,6 +358,11 @@ To use these with the docker command, pass in `-e SANDBOX_<option>`. Example: `-
- Default: `false`
- Description: Use host network
-`runtime_binding_address`
- Type: `str`
- Default: `0.0.0.0`
- Description: The binding address for the runtime ports. It specifies which network interface on the host machine Docker should bind the runtime ports to.
@@ -39,12 +39,13 @@ You can provide custom directions for OpenHands by following the [README for the
### Custom configurations
Github resolver will automatically check for valid [repository secrets](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions?tool=webui#creating-secrets-for-a-repository) or [repository variables](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#creating-configuration-variables-for-a-repository) to customize its behavior.
GitHub resolver will automatically check for valid [repository secrets](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions?tool=webui#creating-secrets-for-a-repository) or [repository variables](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#creating-configuration-variables-for-a-repository) to customize its behavior.
OpenHands provides a user-friendly Graphical User Interface (GUI) mode for interacting with the AI assistant.
This mode offers an intuitive way to set up the environment, manage settings, and communicate with the AI.
OpenHands provides a Graphical User Interface (GUI) mode for interacting with the AI assistant.
## Installation and Setup
@@ -14,104 +11,95 @@ This mode offers an intuitive way to set up the environment, manage settings, an
### Initial Setup
1. Upon first launch, you'll see a settings modal.
2. Select an `LLM Provider` and `LLM Model` from the dropdown menus.
1. Upon first launch, you'll see a settings page.
2. Select an `LLM Provider` and `LLM Model` from the dropdown menus. If the required model does not exist in the list,
toggle `Advanced` options and enter it with the correct prefix in the `Custom Model` text box.
3. Enter the corresponding `API Key` for your chosen provider.
4. Click "Save" to apply the settings.
4. Click `Save Changes` to apply the settings.
### GitHub Token Setup
OpenHands automatically exports a `GITHUB_TOKEN` to the shell environment if it is available. This can happen in two ways:
- **Locally (OSS)**: The user directly inputs their GitHub token.
- **Online (SaaS)**: The token is obtained through GitHub OAuth authentication.
#### Setting Up a Local GitHub Token
1.**Generate a Personal Access Token (PAT)**:
- Go to GitHub Settings > Developer Settings > Personal Access Tokens > Tokens (classic).
- Click "Generate new token (classic)".
- **Local Installation**: The user directly inputs their GitHub token.
<details>
<summary>Setting Up a GitHub Token</summary>
1.**Generate a Personal Access Token (PAT)**:
- On GitHub, go to Settings > Developer Settings > Personal Access Tokens > Tokens (classic).
- Click `Generate new token (classic)`.
- Required scopes:
-`repo` (Full control of private repositories)
-`workflow` (Update GitHub Action workflows)
-`read:org` (Read organization data)
2.**Enter Token in OpenHands**:
- Click the Settings button (gear icon).
- Navigate to the `GitHub Settings` section.
- Paste your token in the `GitHub Token` field.
- Click `Save Changes` to apply the changes.
</details>
2.**Enter Token in OpenHands**:
- Click the Settings button (gear icon) in the top right.
- Navigate to the "GitHub" section.
- Paste your token in the "GitHub Token" field.
- Click "Save" to apply the changes.
<details>
<summary>Organizational Token Policies</summary>
#### Organizational Token Policies
If you're working with organizational repositories, additional setup may be required:
If you're working with organizational repositories, additional setup may be required:
1.**Check Organization Requirements**:
1.**Check Organization Requirements**:
- Organization admins may enforce specific token policies.
- Some organizations require tokens to be created with SSO enabled.
- Review your organization's [token policy settings](https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/setting-a-personal-access-token-policy-for-your-organization).
2.**Verify Organization Access**:
2.**Verify Organization Access**:
- Go to your token settings on GitHub.
- Look for the organization under "Organization access".
- If required, click "Enable SSO" next to your organization.
- Look for the organization under `Organization access`.
- If required, click `Enable SSO` next to your organization.
- Complete the SSO authorization process.
</details>
#### OAuth Authentication (Online Mode)
<details>
<summary>Troubleshooting</summary>
When using OpenHands in online mode, the GitHub OAuth flow:
Common issues and solutions:
1. Requests the following permissions:
- **Token Not Recognized**:
- Ensure the token is properly saved in settings.
- Check that the token hasn't expired.
- Verify the token has the required scopes.
- Try regenerating the token.
- **Organization Access Denied**:
- Check if SSO is required but not enabled.
- Verify organization membership.
- Contact organization admin if token policies are blocking access.
- **Verifying Token Works**:
- The app will show a green checkmark if the token is valid.
- Try accessing a repository to confirm permissions.
- Check the browser console for any error messages.
</details>
- **OpenHands Cloud**: The token is obtained through GitHub OAuth authentication.
<details>
<summary>OAuth Authentication</summary>
When using OpenHands Cloud, the GitHub OAuth flow requests the following permissions:
- Repository access (read/write)
- Workflow management
- Organization read access
2. Authentication steps:
- Click "Sign in with GitHub" when prompted.
To authenticate OpenHands:
- Click `Sign in with GitHub` when prompted.
- Review the requested permissions.
- Authorize OpenHands to access your GitHub account.
- If using an organization, authorize organization access if prompted.
#### Troubleshooting
Common issues and solutions:
- **Token Not Recognized**:
- Ensure the token is properly saved in settings.
- Check that the token hasn't expired.
- Verify the token has the required scopes.
- Try regenerating the token.
- **Organization Access Denied**:
- Check if SSO is required but not enabled.
- Verify organization membership.
- Contact organization admin if token policies are blocking access.
- **Verifying Token Works**:
- The app will show a green checkmark if the token is valid.
- Try accessing a repository to confirm permissions.
- Check the browser console for any error messages.
- Use the "Test Connection" button in settings if available.
</details>
### Advanced Settings
1.Toggle `Advanced Options` to access additional settings.
1.Inside the Settings page, toggle `Advanced` options to access additional settings.
2. Use the `Custom Model` text box to manually enter a model if it's not in the list.
3. Specify a `Base URL` if required by your LLM provider.
### Main Interface
The main interface consists of several key components:
- **Chat Window**: The central area where you can view the conversation history with the AI assistant.
- **Input Box**: Located at the bottom of the screen, use this to type your messages or commands to the AI.
- **Send Button**: Click this to send your message to the AI.
- **Settings Button**: A gear icon that opens the settings modal, allowing you to adjust your configuration at any time.
- **Workspace Panel**: Displays the files and folders in your workspace, allowing you to navigate and view files, or the agent's past commands or web browsing history.
### Interacting with the AI
1. Type your question, request, or task description in the input box.
1. Type your prompt in the input box.
2. Click the send button or press Enter to submit your message.
3. The AI will process your input and provide a response in the chat window.
4. You can continue the conversation by asking follow-up questions or providing additional information.
- Docker version 26.0.0+ or Docker Desktop 4.31.0+.
- You must be using Linux or Mac OS.
- If you are on Windows, you must use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install).
- MacOS with [Docker Desktop support](https://docs.docker.com/desktop/setup/install/mac-install/#system-requirements)
- Linux
- Windows with [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) and [Docker Desktop support](https://docs.docker.com/desktop/setup/install/windows-install/#system-requirements)
## Start the app
A system with a modern processor and a minimum of **4GB RAM** is recommended to run OpenHands.
## Prerequisites
<details>
<summary>MacOS</summary>
**Docker Desktop**
1. [Install Docker Desktop on Mac](https://docs.docker.com/desktop/setup/install/mac-install).
2. Open Docker Desktop, go to `Settings > Advanced` and ensure `Allow the default Docker socket to be used` is enabled.
</details>
<details>
<summary>Linux</summary>
:::note
Tested with Ubuntu 22.04.
:::
**Docker Desktop**
1. [Install Docker Desktop on Linux](https://docs.docker.com/desktop/setup/install/linux/).
Now you're ready to [get started with OpenHands](./getting-started).
## Versions
The command above pulls the most recent stable release of OpenHands. You have other options as well:
- For a specific release, use `docker.all-hands.dev/all-hands-ai/openhands:$VERSION`, replacing $VERSION with the version number.
- We use semver, and release major, minor, and patch tags. So `0.9` will automatically point to the latest `0.9.x` release, and `0` will point to the latest `0.x.x` release.
- For the most up-to-date development version, you can use `docker.all-hands.dev/all-hands-ai/openhands:main`. This version is unstable and is recommended for testing or development purposes only.
The [docker command above](./installation#start-the-app) pulls the most recent stable release of OpenHands. You have other options as well:
- For a specific release, replace $VERSION in `openhands:$VERSION` and `runtime:$VERSION`, with the version number.
We use SemVer so `0.9` will automatically point to the latest `0.9.x` release, and `0` will point to the latest `0.x.x` release.
- For the most up-to-date development version, replace $VERSION in `openhands:$VERSION` and `runtime:$VERSION`, with `main`.
This version is unstable and is recommended for testing or development purposes only.
You can choose the tag that best suits your needs based on stability requirements and desired features.
@@ -25,23 +25,17 @@ You will need your ChatGPT deployment name which can be found on the deployments
<deployment-name> below.
:::
1. Enable `Advanced Options`
1. Enable `Advanced` options
2. Set the following:
-`Custom Model` to azure/<deployment-name>
-`Base URL` to your Azure API Base URL (e.g. `https://example-endpoint.openai.azure.com`)
-`API Key` to your Azure API key
## Embeddings
OpenHands uses llama-index for embeddings. You can find their documentation on Azure [here](https://docs.llamaindex.ai/en/stable/api_reference/embeddings/azure_openai/).
### Azure OpenAI Configuration
When running OpenHands, set the following environment variables using `-e` in the
When running OpenHands, set the following environment variable using `-e` in the
[docker run command](/modules/usage/installation#start-the-app):
```
LLM_EMBEDDING_MODEL="azureopenai"
LLM_EMBEDDING_DEPLOYMENT_NAME="<your-embedding-deployment-name>" # e.g. "TextEmbedding...<etc>"
LLM_API_VERSION="<api-version>" # e.g. "2024-02-15-preview"
- Code generation uses GPT-4 with a higher token limit for generating larger code blocks
- The default configuration remains available for other tasks
# Custom Configurations with Reserved Names
OpenHands can use custom LLM configurations named with reserved names, for specific use cases. If you specify the model and other settings under the reserved names, then OpenHands will load and them for a specific purpose. As of now, one such configuration is implemented: draft editor.
## Draft Editor Configuration
The `draft_editor` configuration is a group of settings you can provide, to specify the model to use for preliminary drafting of code edits, for any tasks that involve editing and refining code. You need to provide it under the section `[llm.draft_editor]`.
For example, you can define in `config.toml` a draft editor like this:
```toml
[llm.draft_editor]
model="gpt-4"
temperature=0.2
top_p=0.95
presence_penalty=0.0
frequency_penalty=0.0
```
This configuration:
- Uses GPT-4 for high-quality edits and suggestions
- Sets a low temperature (0.2) to maintain consistency while allowing some flexibility
- Uses a high top_p value (0.95) to consider a wide range of token options
- Disables presence and frequency penalties to maintain focus on the specific edits needed
Use this configuration when you want to let an LLM draft edits before making them. In general, it may be useful to:
- Review and suggest code improvements
- Refine existing content while maintaining its core meaning
- Make precise, focused changes to code or text
:::note
Custom LLM configurations are only available when using OpenHands in development mode, via `main.py` or `cli.py`. When running via `docker run`, please use the standard configuration options.
@@ -10,7 +10,7 @@ OpenHands uses LiteLLM to make calls to Google's chat models. You can find their
When running OpenHands, you'll need to set the following in the OpenHands UI through the Settings:
-`LLM Provider` to `Gemini`
-`LLM Model` to the model you will be using.
If the model is not in the list, toggle `Advanced Options`, and enter it in `Custom Model` (e.g. gemini/<model-name> like `gemini/gemini-1.5-pro`).
If the model is not in the list, toggle `Advanced` options, and enter it in `Custom Model` (e.g. gemini/<model-name> like `gemini/gemini-1.5-pro`).
@@ -8,7 +8,7 @@ When running OpenHands, you'll need to set the following in the OpenHands UI thr
-`LLM Provider` to `Groq`
-`LLM Model` to the model you will be using. [Visit here to see the list of
models that Groq hosts](https://console.groq.com/docs/models). If the model is not in the list, toggle
`Advanced Options`, and enter it in `Custom Model` (e.g. groq/<model-name> like `groq/llama3-70b-8192`).
`Advanced` options, and enter it in `Custom Model` (e.g. groq/<model-name> like `groq/llama3-70b-8192`).
-`API key` to your Groq API key. To find or create your Groq API Key, [see here](https://console.groq.com/keys).
@@ -17,7 +17,7 @@ models that Groq hosts](https://console.groq.com/docs/models). If the model is n
The Groq endpoint for chat completion is [mostly OpenAI-compatible](https://console.groq.com/docs/openai). Therefore, you can access Groq models as you
would access any OpenAI-compatible endpoint. In the OpenHands UI through the Settings:
1. Enable `Advanced Options`
1. Enable `Advanced` options
2. Set the following:
-`Custom Model` to the prefix `openai/` + the model you will be using (e.g. `openai/llama3-70b-8192`)
@@ -8,5 +8,5 @@ When running OpenHands, you'll need to set the following in the OpenHands UI thr
*`LLM Provider` to `OpenRouter`
*`LLM Model` to the model you will be using.
[Visit here to see a full list of OpenRouter models](https://openrouter.ai/models).
If the model is not in the list, toggle `Advanced Options`, and enter it in `Custom Model` (e.g. openrouter/<model-name> like `openrouter/anthropic/claude-3.5-sonnet`).
If the model is not in the list, toggle `Advanced` options, and enter it in `Custom Model` (e.g. openrouter/<model-name> like `openrouter/anthropic/claude-3.5-sonnet`).
@@ -56,11 +59,12 @@ any files that are mounted into its workspace.
This setup can cause some issues with file permissions (hence the `SANDBOX_USER_ID` variable)
but seems to work well on most systems.
## All Hands Runtime
The All Hands Runtime is currently in beta. You can request access by joining
the #remote-runtime-limited-beta channel on Slack ([see the README](https://github.com/All-Hands-AI/OpenHands?tab=readme-ov-file#-how-to-join-the-community) for an invite).
## OpenHands Remote Runtime
To use the All Hands Runtime, set the following environment variables when
OpenHands Remote Runtime is currently in beta (read [here](https://runtime.all-hands.dev/) for more details), it allows you to launch runtimes in parallel in the cloud.
Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!
To use the OpenHands Remote Runtime, set the following environment variables when
starting OpenHands:
```bash
@@ -84,3 +88,99 @@ docker run # ...
-e MODAL_API_TOKEN_ID="your-id" \
-e MODAL_API_TOKEN_SECRET="your-secret" \
```
## Daytona Runtime
Another option is using [Daytona](https://www.daytona.io/) as a runtime provider:
### Step 1: Retrieve Your Daytona API Key
1. Visit the [Daytona Dashboard](https://app.daytona.io/dashboard/keys).
2. Click **"Create Key"**.
3. Enter a name for your key and confirm the creation.
4. Once the key is generated, copy it.
### Step 2: Set Your API Key as an Environment Variable
Run the following command in your terminal, replacing `<your-api-key>` with the actual key you copied:
```bash
export DAYTONA_API_KEY="<your-api-key>"
```
This step ensures that OpenHands can authenticate with the Daytona platform when it runs.
### Step 3: Run OpenHands Locally Using Docker
To start the latest version of OpenHands on your machine, execute the following command in your terminal:
- Automatically pulls and runs the OpenHands container using Docker.
Once executed, OpenHands should be running locally and ready for use.
For more details and manual initialization, view the entire [README.md](https://github.com/All-Hands-AI/OpenHands/blob/main/openhands/runtime/impl/daytona/README.md)
## Local Runtime
The Local Runtime allows the OpenHands agent to execute actions directly on your local machine without using Docker. This runtime is primarily intended for controlled environments like CI pipelines or testing scenarios where Docker is not available.
:::caution
**Security Warning**: The Local Runtime runs without any sandbox isolation. The agent can directly access and modify files on your machine. Only use this runtime in controlled environments or when you fully understand the security implications.
:::
### Prerequisites
Before using the Local Runtime, ensure you have the following dependencies installed:
1. You have followed the [Development setup instructions](https://github.com/All-Hands-AI/OpenHands/blob/main/Development.md).
2. tmux is available on your system.
### Configuration
To use the Local Runtime, besides required configurations like the model, API key, you'll need to set the following options via environment variables or the [config.toml file](https://github.com/All-Hands-AI/OpenHands/blob/main/config.template.toml) when starting OpenHands:
- Via environment variables:
```bash
# Required
export RUNTIME=local
# Optional but recommended
export WORKSPACE_BASE=/path/to/your/workspace
```
- Via `config.toml`:
```toml
[core]
runtime = "local"
workspace_base = "/path/to/your/workspace"
```
If `WORKSPACE_BASE` is not set, the runtime will create a temporary directory for the agent to work in.
### Example Usage
Here's an example of how to start OpenHands with the Local Runtime in Headless Mode:
```bash
# Set the runtime type to local
export RUNTIME=local
# Optionally set a workspace directory
export WORKSPACE_BASE=/path/to/your/project
# Start OpenHands
poetry run python -m openhands.core.main -t "write a bash script that prints hi"
```
### Use Cases
The Local Runtime is particularly useful for:
- CI/CD pipelines where Docker is not available.
- Testing and development of OpenHands itself.
- Environments where container usage is restricted.
- Scenarios where direct file system access is required.
OpenHands provides a search engine capability that allows agents to perform web searches using the Brave Search API. This guide explains how to configure and use the search feature.
## Overview
The search engine feature enables agents to:
- Execute web search queries programmatically
- Get structured results including web pages, news, videos, and FAQs
- Avoid CAPTCHA challenges that often occur when using browser-based search
## Configuration
### Enabling Search
To enable the search engine feature, set the following in your `config.toml`:
```toml
[agent]
enable_search_engine=true
```
Or when using Docker, set the environment variable:
```bash
-e AGENT_ENABLE_SEARCH_ENGINE=true
```
### API Key Setup
The search feature requires a Brave Search API key. You can obtain one from the [Brave Search API Dashboard](https://api.search.brave.com/app/keys).
Set the API key in your `config.toml`:
```toml
[search]
enabled=true
api_key="your-api-key-here"
```
Or when using Docker:
```bash
-e SEARCH_ENABLED=true
-e SEARCH_API_KEY="your-api-key-here"
```
## Search Results
When a search is performed, the results are returned in a structured format that includes:
- Web search results
- News articles
- Video content
- FAQ entries
- Discussion threads
- Infoboxes (when available)
- Location information (when relevant)
Each result type includes:
- Title
- URL (when applicable)
- Description or snippet
- Additional metadata specific to the result type
## Usage Example
When the search feature is enabled, agents can use the `search_engine` tool to perform searches. For example:
```python
# The agent can make a tool call like this:
{
"name":"search_engine",
"arguments":{
"query":"latest developments in AI"
}
}
```
The search results will be returned in a markdown-formatted structure that's easy for the agent to parse and understand.
## Best Practices
1.**Query Formulation**
- Keep queries focused and specific
- Include relevant keywords
- Avoid overly complex or compound queries
2.**Rate Limiting**
- Be mindful of API rate limits
- Cache results when appropriate
- Implement retries with exponential backoff for failed requests
3.**Error Handling**
- Handle API errors gracefully
- Provide meaningful feedback when searches fail
- Have fallback strategies when search is unavailable
## Troubleshooting
Common issues and solutions:
1.**Search Not Working**
- Verify `enable_search_engine` is set to `true`
- Confirm the Brave API key is correctly set
- Check API key permissions and quotas
2.**No Results**
- Verify the query is not empty
- Try reformulating the search query
- Check for any API response errors
3.**Rate Limiting**
- Monitor API usage
- Implement caching if needed
- Consider upgrading API tier if limits are consistently hit
<ahref="https://docs.all-hands.dev/modules/usage/getting-started"><imgsrc="https://img.shields.io/badge/Documentation-000?logo=googledocs&logoColor=FFE165&style=for-the-badge"alt="Check out the documentation"/></a>
<ahref="https://arxiv.org/abs/2407.16741"><imgsrc="https://img.shields.io/badge/Paper%20on%20Arxiv-000?logoColor=FFE165&logo=arxiv&style=for-the-badge"alt="Paper on Arxiv"/></a>
@@ -20,6 +20,8 @@ To evaluate an agent, you can provide the agent's name to the `run_infer.py` pro
### Evaluating Different LLMs
OpenHands in development mode uses `config.toml` to keep track of most configuration.
**IMPORTANT: For evaluation, only the LLM section in `config.toml` will be used. Other configurations, such as `save_trajectory_path`, are not applied during evaluation.**
Here's an example configuration file you can use to define and use multiple LLMs:
```toml
@@ -40,6 +42,8 @@ api_key = "XXX"
temperature=0.0
```
For other configurations specific to evaluation, such as `save_trajectory_path`, these are typically set in the `get_config` function of the respective `run_infer.py` file for each benchmark.
## Supported Benchmarks
The OpenHands evaluation harness supports a wide variety of benchmarks across [software engineering](#software-engineering), [web browsing](#web-browsing), [miscellaneous assistance](#misc-assistance), and [real-world](#real-world) tasks.
@@ -56,9 +56,10 @@ You can update the arguments in the script
./evaluation/benchmarks/aider_bench/scripts/run_infer.sh eval_gpt35_turbo HEAD CodeActAgent 1001"1,3,10"
```
### Run Inference on `RemoteRuntime` (experimental)
### Run Inference on `RemoteRuntime`
This is in beta. Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!
This is in limited beta. Contact Xingyao over slack if you want to try this out!
This folder contains the evaluation harness that we built on top of the original [Commit0](https://commit-0.github.io/) ([paper](TBD)).
This folder contains the evaluation harness that we built on top of the original [Commit0](https://commit-0.github.io/) ([paper](https://arxiv.org/abs/2412.01769v1)).
The evaluation consists of three steps:
@@ -23,10 +23,10 @@ Make sure your Docker daemon is running, and you have ample disk space (at least
When the `run_infer.sh` script is started, it will automatically pull the `lite` split in Commit0. For example, for instance ID `commit-0/minitorch`, it will try to pull our pre-build docker image `wentingzhao/minitorch` from DockerHub. This image will be used create an OpenHands runtime image where the agent will operate on.
./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 16 100 8 wentingzhao/commit0_combined test
./evaluation/benchmarks/commit0/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 16 100 8 wentingzhao/commit0_combined test
```
where `model_config` is mandatory, and the rest are optional.
@@ -48,26 +48,25 @@ default, it is set to 1.
- `dataset`, a huggingface dataset name. e.g. `wentingzhao/commit0_combined`, specifies which dataset to evaluate on.
- `dataset_split`, split for the huggingface dataset. Notice only `test` is supported for Commit0.
Note that the `USE_INSTANCE_IMAGE` environment variable is always set to `true` for Commit0.
Let's say you'd like to run 10 instances using `llm.eval_sonnet` and CodeActAgent,
then your command would be:
```bash
./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
./evaluation/benchmarks/commit0/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
```
### Run Inference on `RemoteRuntime` (experimental)
### Run Inference on `RemoteRuntime`
This is in beta. Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!
This is in limited beta. Contact Xingyao over slack if you want to try this out!
# Example - This runs evaluation on CodeActAgent for 10 instances on "wentingzhao/commit0_combined"'s test set, with max 30 iteration per instances, with 1 number of workers running in parallel
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.