chore: temporary fix to get sonnet 3.7 working again (#7140 )

Release 0.28.0
rename commit0_bench to commit0 (#7124 )
2026-04-29 03:00:45 -04:00 · 2025-03-06 15:32:27 -05:00 · 2025-03-06 09:24:58 -05:00 · 2025-03-06 02:55:39 +00:00 · 2025-03-05 21:58:34 +00:00 · 2025-03-05 20:15:02 +00:00
69 changed files with 1028 additions and 449 deletions
@@ -41,8 +41,10 @@ jobs:
    steps:
      - name: Checkout
        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3.4.0
+        uses: docker/setup-qemu-action@v3.6.0
        with:
          image: tonistiigi/binfmt:latest
      - name: Login to GHCR
@@ -90,8 +92,10 @@ jobs:
    steps:
      - name: Checkout
        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3.4.0
+        uses: docker/setup-qemu-action@v3.6.0
        with:
          image: tonistiigi/binfmt:latest
      - name: Login to GHCR
@@ -154,6 +158,8 @@ jobs:
        base_image: ['nikolaik']
    steps:
      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
      - name: Cache Poetry dependencies
        uses: actions/cache@v4
        with:
@@ -100,7 +100,7 @@ poetry run pytest ./tests/unit/test_*.py
 To reduce build time (e.g., if no changes were made to the client-runtime component), you can use an existing Docker container image by
 setting the SANDBOX_RUNTIME_CONTAINER_IMAGE environment variable to the desired Docker image.

-Example: `export SANDBOX_RUNTIME_CONTAINER_IMAGE=ghcr.io/all-hands-ai/runtime:0.27-nikolaik`
+Example: `export SANDBOX_RUNTIME_CONTAINER_IMAGE=ghcr.io/all-hands-ai/runtime:0.28-nikolaik`

 ## Develop inside Docker container

@@ -43,17 +43,17 @@ See the [Running OpenHands](https://docs.all-hands.dev/modules/usage/installatio
 system requirements and more information.

 ```bash
-docker pull docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik
+docker pull docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik

 docker run -it --rm --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27
+    docker.all-hands.dev/all-hands-ai/openhands:0.28
 ```

 You'll find OpenHands running at [http://localhost:3000](http://localhost:3000)!
@@ -11,7 +11,7 @@ services:
      - BACKEND_HOST=${BACKEND_HOST:-"0.0.0.0"}
      - SANDBOX_API_HOSTNAME=host.docker.internal
      #
-      - SANDBOX_RUNTIME_CONTAINER_IMAGE=${SANDBOX_RUNTIME_CONTAINER_IMAGE:-ghcr.io/all-hands-ai/runtime:0.27-nikolaik}
+      - SANDBOX_RUNTIME_CONTAINER_IMAGE=${SANDBOX_RUNTIME_CONTAINER_IMAGE:-ghcr.io/all-hands-ai/runtime:0.28-nikolaik}
      - SANDBOX_USER_ID=${SANDBOX_USER_ID:-1234}
      - WORKSPACE_MOUNT_PATH=${WORKSPACE_BASE:-$PWD/workspace}
    ports:
@@ -7,7 +7,7 @@ services:
    image: openhands:latest
    container_name: openhands-app-${DATE:-}
    environment:
-      - SANDBOX_RUNTIME_CONTAINER_IMAGE=${SANDBOX_RUNTIME_CONTAINER_IMAGE:-docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik}
+      - SANDBOX_RUNTIME_CONTAINER_IMAGE=${SANDBOX_RUNTIME_CONTAINER_IMAGE:-docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik}
      #- SANDBOX_USER_ID=${SANDBOX_USER_ID:-1234} # enable this only if you want a specific non-root sandbox user but you will have to manually adjust permissions of openhands-state for this user
      - WORKSPACE_MOUNT_PATH=${WORKSPACE_BASE:-$PWD/workspace}
    ports:
@@ -52,7 +52,7 @@ LLM_API_KEY="sk_test_12345"
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -61,7 +61,7 @@ docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.cli
 ```

@@ -46,7 +46,7 @@ LLM_API_KEY="sk_test_12345"
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -56,6 +56,6 @@ docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.main -t "write a bash script that prints hi" --no-auto-continue
 ```
@@ -13,16 +13,16 @@
 La façon la plus simple d'exécuter OpenHands est avec Docker.

 ```bash
-docker pull docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik
+docker pull docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik

 docker run -it --rm --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27
+    docker.all-hands.dev/all-hands-ai/openhands:0.28
 ```

 Vous pouvez également exécuter OpenHands en mode [headless scriptable](https://docs.all-hands.dev/modules/usage/how-to/headless-mode), en tant que [CLI interactive](https://docs.all-hands.dev/modules/usage/how-to/cli-mode), ou en utilisant l'[Action GitHub OpenHands](https://docs.all-hands.dev/modules/usage/how-to/github-action).
@@ -13,7 +13,7 @@ C'est le Runtime par défaut qui est utilisé lorsque vous démarrez OpenHands.

 ```
 docker run # ...
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -v /var/run/docker.sock:/var/run/docker.sock \
    # ...
 ```
@@ -50,7 +50,7 @@ LLM_API_KEY="sk_test_12345"
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -59,7 +59,7 @@ docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.cli
 ```

@@ -47,7 +47,7 @@ LLM_API_KEY="sk_test_12345"
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -57,6 +57,6 @@ docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.main -t "write a bash script that prints hi" --no-auto-continue
 ```
@@ -11,16 +11,16 @@
 在 Docker 中运行 OpenHands 是最简单的方式。

 ```bash
-docker pull docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik
+docker pull docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik

 docker run -it --rm --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27
+    docker.all-hands.dev/all-hands-ai/openhands:0.28
 ```

 你也可以在可脚本化的[无头模式](https://docs.all-hands.dev/modules/usage/how-to/headless-mode)下运行 OpenHands，作为[交互式 CLI](https://docs.all-hands.dev/modules/usage/how-to/cli-mode)，或使用 [OpenHands GitHub Action](https://docs.all-hands.dev/modules/usage/how-to/github-action)。
@@ -11,7 +11,7 @@

 ```
 docker run # ...
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -v /var/run/docker.sock:/var/run/docker.sock \
    # ...
 ```
@@ -35,7 +35,7 @@ To run OpenHands in CLI mode with Docker:
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -45,7 +45,7 @@ docker run -it \
    -v ~/.openhands-state:/.openhands-state \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.cli
 ```

@@ -32,7 +32,7 @@ To run OpenHands in Headless mode with Docker:
 ```bash
 docker run -it \
    --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
@@ -43,7 +43,7 @@ docker run -it \
    -v ~/.openhands-state:/.openhands-state \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27 \
+    docker.all-hands.dev/all-hands-ai/openhands:0.28 \
    python -m openhands.core.main -t "write a bash script that prints hi"
 ```

@@ -58,17 +58,17 @@ A system with a modern processor and a minimum of **4GB RAM** is recommended to
 The easiest way to run OpenHands is in Docker.

 ```bash
-docker pull docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik
+docker pull docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik

 docker run -it --rm --pull=always \
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
-    docker.all-hands.dev/all-hands-ai/openhands:0.27
+    docker.all-hands.dev/all-hands-ai/openhands:0.28
 ```

 You'll find OpenHands running at http://localhost:3000!
@@ -16,7 +16,7 @@ some flags being passed to `docker run` that make this possible:

 ```
 docker run # ...
-    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.27-nikolaik \
+    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.28-nikolaik \
    -v /var/run/docker.sock:/var/run/docker.sock \
    # ...
 ```
@@ -24,7 +24,7 @@
        "@docusaurus/module-type-aliases": "^3.5.1",
        "@docusaurus/tsconfig": "^3.7.0",
        "@docusaurus/types": "^3.5.1",
-        "typescript": "~5.7.3"
+        "typescript": "~5.8.2"
      },
      "engines": {
        "node": ">=18.0"
@@ -17638,9 +17638,9 @@
      }
    },
    "node_modules/typescript": {
-      "version": "5.7.3",
-      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.7.3.tgz",
-      "integrity": "sha512-84MVSjMEHP+FQRPy3pX9sTVV/INIex71s9TL2Gm5FG/WG1SqXeKyZ0k7/blY/4FdOzI12CBy1vGc4og/eus0fw==",
+      "version": "5.8.2",
+      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.8.2.tgz",
+      "integrity": "sha512-aJn6wq13/afZp/jT9QZmwEjDqqvSGp1VT5GVg+f/t6/oVyrgXM6BY1h9BRh/O5p3PlUPAe+WuiEZOmb/49RqoQ==",
      "license": "Apache-2.0",
      "bin": {
        "tsc": "bin/tsc",
@@ -31,7 +31,7 @@
    "@docusaurus/module-type-aliases": "^3.5.1",
    "@docusaurus/tsconfig": "^3.7.0",
    "@docusaurus/types": "^3.5.1",
-    "typescript": "~5.7.3"
+    "typescript": "~5.8.2"
  },
  "browserslist": {
    "production": [
@@ -56,9 +56,10 @@ You can update the arguments in the script
 ./evaluation/benchmarks/aider_bench/scripts/run_infer.sh eval_gpt35_turbo HEAD CodeActAgent 100 1 "1,3,10"
 ```

-### Run Inference on `RemoteRuntime` (experimental)
+### Run Inference on `RemoteRuntime`
+
+This is in beta. Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!

-This is in limited beta. Contact Xingyao over slack if you want to try this out!

 ```bash
 ./evaluation/benchmarks/aider_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [eval-num-workers] [eval_ids]
@@ -23,10 +23,10 @@ Make sure your Docker daemon is running, and you have ample disk space (at least
 When the `run_infer.sh` script is started, it will automatically pull the `lite` split in Commit0. For example, for instance ID `commit-0/minitorch`, it will try to pull our pre-build docker image `wentingzhao/minitorch` from DockerHub. This image will be used create an OpenHands runtime image where the agent will operate on.

 ```bash
-./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh [repo_split] [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]
+./evaluation/benchmarks/commit0/scripts/run_infer.sh [repo_split] [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]

 # Example
-./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 16 100 8 wentingzhao/commit0_combined test
+./evaluation/benchmarks/commit0/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 16 100 8 wentingzhao/commit0_combined test
 ```

 where `model_config` is mandatory, and the rest are optional.
@@ -48,26 +48,25 @@ default, it is set to 1.
 - `dataset`, a huggingface dataset name. e.g. `wentingzhao/commit0_combined`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. Notice only `test` is supported for Commit0.

-Note that the `USE_INSTANCE_IMAGE` environment variable is always set to `true` for Commit0.
-
 Let's say you'd like to run 10 instances using `llm.eval_sonnet` and CodeActAgent,

 then your command would be:

 ```bash
-./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
+./evaluation/benchmarks/commit0/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
 ```

-### Run Inference on `RemoteRuntime` (experimental)
+### Run Inference on `RemoteRuntime`
+
+This is in beta. Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!

-This is in limited beta. Contact Xingyao over slack if you want to try this out!

 ```bash
-./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh [repo_split] [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]
+./evaluation/benchmarks/commit0/scripts/run_infer.sh [repo_split] [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]

 # Example - This runs evaluation on CodeActAgent for 10 instances on "wentingzhao/commit0_combined"'s test set, with max 30 iteration per instances, with 1 number of workers running in parallel
 ALLHANDS_API_KEY="YOUR-API-KEY" RUNTIME=remote SANDBOX_REMOTE_RUNTIME_API_URL="https://runtime.eval.all-hands.dev" EVAL_DOCKER_IMAGE_PREFIX="docker.io/wentingzhao" \
-./evaluation/benchmarks/commit0_bench/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
+./evaluation/benchmarks/commit0/scripts/run_infer.sh lite llm.eval_sonnet HEAD CodeActAgent 10 30 1 wentingzhao/commit0_combined test
 ```

 To clean-up all existing runtime you've already started, run:
@@ -39,7 +39,6 @@ from openhands.utils.async_utils import call_async_from_sync
 from openhands.utils.shutdown_listener import sleep_if_should_continue

 USE_HINT_TEXT = os.environ.get('USE_HINT_TEXT', 'false').lower() == 'true'
-USE_INSTANCE_IMAGE = os.environ.get('USE_INSTANCE_IMAGE', 'false').lower() == 'true'
 RUN_WITH_BROWSING = os.environ.get('RUN_WITH_BROWSING', 'false').lower() == 'true'

 AGENT_CLS_TO_FAKE_USER_RESPONSE_FN = {
@@ -105,7 +104,6 @@ def get_config(
    instance: pd.Series,
    metadata: EvalMetadata,
 ) -> AppConfig:
-    assert USE_INSTANCE_IMAGE
    repo_name = instance['repo'].split('/')[1]
    base_container_image = get_instance_docker_image(repo_name)
    logger.info(
@@ -303,16 +301,6 @@ def complete_runtime(
    pytest_exit_code = obs.content.strip()
    # logger.info(f'Pytest exit code: {pytest_exit_code}')

-    # Read the test report
-    action = CmdRunAction(command='cat report.json')
-    action.set_hard_timeout(600)
-    logger.info(action, extra={'msg_type': 'ACTION'})
-    obs = runtime.run_action(action)
-    # logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-    assert_and_raise(
-        isinstance(obs, CmdOutputObservation),
-        f'Failed to read test report: {str(obs)}',
-    )
    # Get test IDs from instance
    repo_name = instance['repo'].split('/')[1]
    repo_name = repo_name.replace('.', '-')
@@ -323,8 +311,20 @@ def complete_runtime(
    # logger.info(obs, extra={'msg_type': 'OBSERVATION'})
    test_ids = obs.content.strip().split('\n')

+    # Read the test report
+    action = CmdRunAction(command='cat report.json')
+    action.set_hard_timeout(600)
+    logger.info(action, extra={'msg_type': 'ACTION'})
+    obs = runtime.run_action(action)
+    # logger.info(obs, extra={'msg_type': 'OBSERVATION'})
+    assert_and_raise(
+        isinstance(obs, CmdOutputObservation),
+        f'Failed to read test report: {str(obs)}',
+    )
+    json_report = obs.content.strip()
+
    try:
-        report = json.loads(obs.content)
+        report = json.loads(json_report)
        tests = {x['nodeid']: x['call'] for x in report['tests'] if 'call' in x}

        # Calculate test statistics
@@ -30,11 +30,6 @@ if [ -z "$MAX_ITER" ]; then
  MAX_ITER=100
 fi

-if [ -z "$USE_INSTANCE_IMAGE" ]; then
-  echo "USE_INSTANCE_IMAGE not specified, use default true"
-  USE_INSTANCE_IMAGE=true
-fi
-
 if [ -z "$RUN_WITH_BROWSING" ]; then
  echo "RUN_WITH_BROWSING not specified, use default false"
  RUN_WITH_BROWSING=false
@@ -56,8 +51,6 @@ if [ -z "$SPLIT" ]; then
  SPLIT="test"
 fi

-export USE_INSTANCE_IMAGE=$USE_INSTANCE_IMAGE
-echo "USE_INSTANCE_IMAGE: $USE_INSTANCE_IMAGE"
 export RUN_WITH_BROWSING=$RUN_WITH_BROWSING
 echo "RUN_WITH_BROWSING: $RUN_WITH_BROWSING"

@@ -91,7 +84,7 @@ fi

 function run_eval() {
  local eval_note=$1
-  COMMAND="poetry run python evaluation/benchmarks/commit0_bench/run_infer.py \
+  COMMAND="poetry run python evaluation/benchmarks/commit0/run_infer.py \
    --agent-cls $AGENT \
    --llm-config $MODEL_CONFIG \
    --max-iterations $MAX_ITER \
@@ -13,6 +13,7 @@
 # limitations under the License.
 # This file is modified from https://github.com/ShishirPatil/gorilla/blob/main/eval/eval-scripts/ast_eval_hf.py

+import tree_sitter_python as tspython
 from tree_sitter import Language, Parser


@@ -39,10 +40,9 @@ def get_all_sub_trees(root_node):


 # Parse the program into AST trees
-def ast_parse(candidate, lang='python'):
-    LANGUAGE = Language('evaluation/gorilla/my-languages.so', lang)
-    parser = Parser()
-    parser.set_language(LANGUAGE)
+def ast_parse(candidate):
+    LANGUAGE = Language(tspython.language())
+    parser = Parser(LANGUAGE)

    candidate_tree = parser.parse(bytes(candidate, 'utf8')).root_node
    return candidate_tree
@@ -13,6 +13,7 @@
 # limitations under the License.
 # This file is modified from https://github.com/ShishirPatil/gorilla/blob/main/eval/eval-scripts/ast_eval_tf.py

+import tree_sitter_python as tspython
 from tree_sitter import Language, Parser


@@ -39,10 +40,9 @@ def get_all_sub_trees(root_node):


 # Parse the program into AST trees
-def ast_parse(candidate, lang='python'):
-    LANGUAGE = Language('evaluation/gorilla/my-languages.so', lang)
-    parser = Parser()
-    parser.set_language(LANGUAGE)
+def ast_parse(candidate):
+    LANGUAGE = Language(tspython.language())
+    parser = Parser(LANGUAGE)

    candidate_tree = parser.parse(bytes(candidate, 'utf8')).root_node
    return candidate_tree
@@ -13,6 +13,7 @@
 # limitations under the License.
 # This file is modified from https://github.com/ShishirPatil/gorilla/blob/main/eval/eval-scripts/ast_eval_th.py

+import tree_sitter_python as tspython
 from tree_sitter import Language, Parser


@@ -39,10 +40,9 @@ def get_all_sub_trees(root_node):


 # Parse the program into AST trees
-def ast_parse(candidate, lang='python'):
-    LANGUAGE = Language('evaluation/gorilla/my-languages.so', lang)
-    parser = Parser()
-    parser.set_language(LANGUAGE)
+def ast_parse(candidate):
+    LANGUAGE = Language(tspython.language())
+    parser = Parser(LANGUAGE)

    candidate_tree = parser.parse(bytes(candidate, 'utf8')).root_node
    return candidate_tree
@@ -71,19 +71,19 @@ def fetch_data(url, filename):

 def get_data_for_hub(hub: str):
    if hub == 'hf':
-        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/eval/eval-data/questions/huggingface/questions_huggingface_0_shot.jsonl'
-        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/api/huggingface_api.jsonl'
-        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/apibench/huggingface_eval.json'
+        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/eval/eval-data/questions/huggingface/questions_huggingface_0_shot.jsonl'
+        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/api/huggingface_api.jsonl'
+        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/apibench/huggingface_eval.json'
        ast_eval = ast_eval_hf
    elif hub == 'torch':
-        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/eval/eval-data/questions/torchhub/questions_torchhub_0_shot.jsonl'
-        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/api/torchhub_api.jsonl'
-        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/apibench/torchhub_eval.json'
+        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/eval/eval-data/questions/torchhub/questions_torchhub_0_shot.jsonl'
+        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/api/torchhub_api.jsonl'
+        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/apibench/torchhub_eval.json'
        ast_eval = ast_eval_th
    elif hub == 'tf':
-        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/eval/eval-data/questions/tensorflowhub/questions_tensorflowhub_0_shot.jsonl'
-        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/api/tensorflowhub_api.jsonl'
-        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/main/data/apibench/tensorflow_eval.json'
+        question_data = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/eval/eval-data/questions/tensorflowhub/questions_tensorflowhub_0_shot.jsonl'
+        api_dataset = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/api/tensorflowhub_api.jsonl'
+        apibench = 'https://raw.githubusercontent.com/ShishirPatil/gorilla/refs/tags/v1.2/data/apibench/tensorflow_eval.json'
        ast_eval = ast_eval_tf

    question_data = fetch_data(question_data, 'question_data.jsonl')
@@ -19,9 +19,10 @@ Access with browser the above MiniWoB URLs and see if they load correctly.
 ./evaluation/benchmarks/miniwob/scripts/run_infer.sh llm.claude-35-sonnet-eval
 ```

-### Run Inference on `RemoteRuntime` (experimental)
+### Run Inference on `RemoteRuntime`
+
+This is in beta. Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!

-This is in limited beta. Contact Xingyao over slack if you want to try this out!

 ```bash
 ./evaluation/benchmarks/miniwob/scripts/run_infer.sh [model_config] [git-version] [agent] [note] [eval_limit] [num_workers]
@@ -2,11 +2,13 @@

 This folder contains the evaluation harness that we built on top of the original [SWE-Bench benchmark](https://www.swebench.com/) ([paper](https://arxiv.org/abs/2310.06770)).

+**UPDATE (2/18/2025): We now support running SWE-Gym using the same evaluation harness here. For more details, checkout [this README](./SWE-Gym.md).
+
 **UPDATE (7/1/2024): We now support the official SWE-Bench dockerized evaluation as announced [here](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md).**

 The evaluation consists of three steps:

-1. Environment setup: [install python environment](../../README.md#development-environment), [configure LLM config](../../README.md#configure-openhands-and-your-llm), and [pull docker](#openhands-swe-bench-instance-level-docker-support).
+1. Environment setup: [install python environment](../../README.md#development-environment) and [configure LLM config](../../README.md#configure-openhands-and-your-llm).
 2. [Run inference](#run-inference-on-swe-bench-instances): Generate a edit patch for each Github issue
 3. [Evaluate patches using SWE-Bench docker](#evaluate-generated-patches)

@@ -14,22 +16,21 @@ The evaluation consists of three steps:

 Please follow instruction [here](../../README.md#setup) to setup your local development environment and LLM.

-## OpenHands SWE-Bench Instance-level Docker Support
+## Run Inference (Rollout) on SWE-Bench Instances: Generate Patch from Problem Statement

-OpenHands now support using the [official evaluation docker](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md) for both **[inference](#run-inference-on-swe-bench-instances) and [evaluation](#evaluate-generated-patches)**.
-This is now the default behavior.
+### Running Locally with Docker

-## Run Inference on SWE-Bench Instances
+Make sure your Docker daemon is running, and you have ample disk space (at least 200-500GB, depends on the SWE-Bench set you are running on) for the instance-level docker image.

-Make sure your Docker daemon is running, and you have ample disk space (at least 200-500GB, depends on the SWE-Bench set you are running on) for the [instance-level docker image](#openhands-swe-bench-instance-level-docker-support).
-
-When the `run_infer.sh` script is started, it will automatically pull the relevant SWE-Bench images. For example, for instance ID `django_django-11011`, it will try to pull our pre-build docker image `sweb.eval.x86_64.django_s_django-11011` from DockerHub. This image will be used create an OpenHands runtime image where the agent will operate on.
+When the `run_infer.sh` script is started, it will automatically pull the relevant SWE-Bench images.
+For example, for instance ID `django_django-11011`, it will try to pull our pre-build docker image `sweb.eval.x86_64.django_s_django-11011` from DockerHub.
+This image will be used create an OpenHands runtime image where the agent will operate on.

 ```bash
 ./evaluation/benchmarks/swe_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]

 # Example
-./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 300 30 1 princeton-nlp/SWE-bench_Lite test
+./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 500 100 1 princeton-nlp/SWE-bench_Verified test
 ```

 where `model_config` is mandatory, and the rest are optional.
@@ -47,14 +48,16 @@ in order to use `eval_limit`, you must also set `agent`.
 default, it is set to 30.
 - `num_workers`, e.g. `3`, is the number of parallel workers to run the evaluation. By
 default, it is set to 1.
- `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench` or `princeton-nlp/SWE-bench_Lite`, specifies which dataset to evaluate on.
+- `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench`, `princeton-nlp/SWE-bench_Lite`, or `princeton-nlp/SWE-bench_Verified`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. e.g., `test`, `dev`. Default to `test`.

-There are also two optional environment variables you can set.
+> [!CAUTION]
+> Setting `num_workers` larger than 1 is not officially tested, YMMV.
+
+There is also one optional environment variable you can set.

 ```bash
 export USE_HINT_TEXT=true # if you want to use hint text in the evaluation. Default to false. Ignore this if you are not sure.
-export USE_INSTANCE_IMAGE=true # if you want to use instance-level docker images. Default to true
 ```

 Let's say you'd like to run 10 instances using `llm.eval_gpt4_1106_preview` and CodeActAgent,
@@ -65,9 +68,11 @@ then your command would be:
 ./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 10
 ```

-### Run Inference on `RemoteRuntime` (experimental)
+### Running in parallel with RemoteRuntime

-This is in limited beta. Contact Xingyao over slack if you want to try this out!
+OpenHands Remote Runtime is currently in beta (read [here](https://runtime.all-hands.dev/) for more details), it allows you to run rollout in parallel in the cloud, so you don't need a powerful machine to run evaluation.
+
+Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!

 ```bash
 ./evaluation/benchmarks/swe_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]
@@ -100,41 +105,14 @@ After running the inference, you will obtain a `output.jsonl` (by default it wil

 ## Evaluate Generated Patches

-### Download Docker Images
-
-**(Recommended for reproducibility)** If you have extra local space (e.g., 200GB), you can try pull the [instance-level docker images](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md#choosing-the-right-cache_level) we've prepared by running:
-
-```bash
-evaluation/benchmarks/swe_bench/scripts/docker/pull_all_eval_docker.sh instance
-```
-
-If you want to save disk space a bit (e.g., with ~50GB free disk space), while speeding up the image pre-build process, you can pull the environment-level docker images:
-
-```bash
-evaluation/benchmarks/swe_bench/scripts/docker/pull_all_eval_docker.sh env
-```
-
-If you want to evaluate on the full SWE-Bench test set:
-
-```bash
-evaluation/benchmarks/swe_bench/scripts/docker/pull_all_eval_docker.sh instance full
-```
-
-### Run evaluation
+### Run evaluation with official SWE-Bench harness (Recommend if you have local disk space)

 With `output.jsonl` file, you can run `eval_infer.sh` to evaluate generated patches, and produce a fine-grained report.
-
 **This evaluation is performed using the official dockerized evaluation announced [here](https://github.com/princeton-nlp/SWE-bench/blob/main/docs/20240627_docker/README.md).**

-> If you want to evaluate existing results, you should first run this to clone existing outputs
->
->```bash
->git clone https://huggingface.co/spaces/OpenHands/evaluation evaluation/evaluation_outputs
->```
+> [!NOTE]
+> This process will automatically download docker images from SWE-Bench official docker hub, please make sure you have enough disk space!

-NOTE, you should have already pulled the instance-level OR env-level docker images following [this section](#openhands-swe-bench-instance-level-docker-support).
-
-Then you can run the following:

 ```bash
 ./evaluation/benchmarks/swe_bench/scripts/eval_infer.sh $YOUR_OUTPUT_JSONL [instance_id] [dataset_name] [split]
@@ -163,9 +141,10 @@ The final results will be saved to `evaluation/evaluation_outputs/outputs/swe_be
 - `report.json`: a JSON file that contains keys like `"resolved_ids"` pointing to instance IDs that are resolved by the agent.
 - `logs/`: a directory of test logs

-### Run evaluation with `RemoteRuntime` (experimental)
+### Run evaluation with `RemoteRuntime`

-This is in limited beta. Contact Xingyao over slack if you want to try this out!
+OpenHands Remote Runtime is currently in beta (read [here](https://runtime.all-hands.dev/) for more details), it allows you to run rollout in parallel in the cloud, so you don't need a powerful machine to run evaluation.
+Fill out [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply if you want to try this out!

 ```bash
 ./evaluation/benchmarks/swe_bench/scripts/eval_infer_remote.sh [output.jsonl filepath] [num_workers]
@@ -180,35 +159,3 @@ To clean-up all existing runtimes that you've already started, run:
 ```bash
 ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/utils/scripts/cleanup_remote_runtime.sh
 ```
-
-## Visualize Results
-
-First you need to clone `https://huggingface.co/spaces/OpenHands/evaluation` and add your own running results from openhands into the `outputs` of the cloned repo.
-
-```bash
-git clone https://huggingface.co/spaces/OpenHands/evaluation
-```
-
-**(optional) setup streamlit environment with conda**:
-
-```bash
-cd evaluation
-conda create -n streamlit python=3.10
-conda activate streamlit
-pip install -r requirements.txt
-```
-
-**run the visualizer**:
-Then, in a separate Python environment with `streamlit` library, you can run the following:
-
-```bash
-# Make sure you are inside the cloned `evaluation` repo
-conda activate streamlit # if you follow the optional conda env setup above
-streamlit run app.py --server.port 8501 --server.address 0.0.0.0
-```
-
-Then you can access the SWE-Bench trajectory visualizer at `localhost:8501`.
-
-## Submit your evaluation results
-
-You can start your own fork of [our huggingface evaluation outputs](https://huggingface.co/spaces/OpenHands/evaluation) and submit a PR of your evaluation results following the guide [here](https://huggingface.co/docs/hub/en/repositories-pull-requests-discussions#pull-requests-and-discussions).
@@ -0,0 +1,128 @@
+<h1 align="center"> Training Software Engineering Agents and Verifiers with SWE-Gym </h1>
+
+<p align="center">
+  <a href="https://www.jiayipan.com/" style="text-decoration: none;">Jiayi Pan<sup>*,1</sup></a>,
+  <a href="https://xwang.dev/" style="text-decoration: none;">Xingyao Wang<sup>*,2</sup></a>,
+  <a href="https://www.phontron.com/" style="text-decoration: none;">Graham Neubig<sup>3</sup></a>,
+  <a href="https://www.cs.toronto.edu/~ndjaitly/" style="text-decoration: none;">Navdeep Jaitly<sup>4</sup></a>,
+  <a href="https://blender.cs.illinois.edu/hengji.html" style="text-decoration: none;">Heng Ji<sup>2</sup></a>,
+  <a href="https://www.alanesuhr.com/" style="text-decoration: none;">Alane Suhr<sup>^,1</sup></a>,
+  <a href="https://dreasysnail.github.io/" style="text-decoration: none;">Yizhe Zhang<sup>^,4</sup></a>
+</p>
+
+<p align="center">
+  <sup>1</sup>UC Berkeley, <sup>2</sup>UIUC, <sup>3</sup>CMU, <sup>4</sup>Apple </br>
+  <sub><sup>*</sup>Equal contribution, <sup>^</sup>Equal supervision</sub>
+</p>
+
+<p align="center">
+<a href="https://arxiv.org/abs/2412.21139">📃 Paper</a>
+•
+<a href="https://huggingface.co/SWE-Gym" >🤗 Data & Models</a>
+</p>
+
+We present **SWE-Gym**, the first environment for training real-world software engineering agents.
+We use it to train strong LM agents that achieve state-of-the-art open results on SWE-Bench, with early, promising scaling characteristics as we increase training and inference-time compute.
+
+<p align="center">
+  <img src="https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/teaser.jpg?raw=true" width="100%" alt="teaser">
+</p>
+
+---
+# Run SWE-Gym with OpenHands
+
+The process of running SWE-Gym is very similar to how you'd run SWE-Bench evaluation.
+
+
+1. First, clone OpenHands repo `git clone https://github.com/All-Hands-AI/OpenHands.git`
+2. Then setup the repo following [Development.md](https://github.com/All-Hands-AI/OpenHands/blob/main/Development.md)
+3. Then you can simply serve your own model as an OpenAI compatible endpoint, put those info in config.toml. You can do this by following instruction [here](../../README.md#setup).
+4. And then simply do the following to sample for 16x parallelism:
+
+```bash
+export ALLHANDS_API_KEY=ah-yourkey  # You don't need to set this when running these in local docker container
+./evaluation/benchmarks/swe_bench/scripts/rollout_swegym.sh llm.mymodel-temp05 'train-t05' 16
+```
+
+NOTE: SWE-Gym sampling with parallelism is currently only tested with AllHands RemoteRuntime (limited beta). Fill [this form](https://docs.google.com/forms/d/e/1FAIpQLSckVz_JFwg2_mOxNZjCtr7aoBFI2Mwdan3f75J_TrdMS1JV2g/viewform) to apply for access.
+
+
+5. When `rollout_swegym.sh` finishes, you will get a file called `output.with_completions.jsonl.gz`. Then you can use [`./scripts/swegym/convert_data.ipynb`](./scripts/swegym/convert_data.ipynb) to convert them into SFT data format.
+
+---
+# More info about SWE-Gym
+
+Progress in agents for software engineering has been limited by the lack of training environments that both include rigorous verification for reinforcement learning and cover the expansive tasks encountered in real-world repository-level engineering.
+
+We introduce SWE-Gym: An Open Environment for Training Software Engineering Agents & Verifiers.
+Our baselines achieve new open SOTA - 32%/26% on SWE-Bench Verified/Lite, with promising scaling trends.
+
+![SWE-Gym Scaling](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/scaling.jpg?raw=true)
+*SWE-Gym enables scalable improvements for software engineering agents at both training and inference time. Our current results is primarily bottlenecked by training and inference compute, rather than the size of our environment.*
+
+## SWE-Gym Environment
+
+We create SWE-Gym, the first environment for training SWE agents, with **2.4K real tasks from 11 Python repos** & a Lite split of 234 instances. SWE-Gym combines real-world Python tasks, repository context, executable environments, and test verification to train agents for solving software engineering problems.
+
+![SWE-Gym Repo Distribution](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/swe-gym.jpg?raw=true)
+
+
+## SWE-Gym trains LMs as agents
+
+When fine-tuned on less than 500 agent-environment interaction trajectories sampled from it from GPT-4o and Claude 3.5 Sonnet, we achieve **+14%** absolute gains on SWE-Bench Verified with an 32B LM-powered OpenHands agent.
+
+![OpenHands Performance diff before and after training](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/oh-agent.jpg?raw=true)
+
+
+## SWE-Gym enables self-improvement
+
+SWE-Gym is also effective across agent scaffolds. With rejection sampling fine-tuning and MoatlessTools scaffold, our 32B and 7B models achieve 20% and 10% respectively on SWE-Bench Lite through self-improvement.
+
+<p align="center">
+  <img src="https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/ml-agent.jpg?raw=true" width="80%" alt="Moatless self-improvement">
+</p>
+
+
+
+## SWE-Gym enables inference-time scaling
+
+SWE-Gym enables inference-time scaling through verifiers trained on agent trajectories.
+These verifiers identify most promising solutions via best-of-n selection, together with our learned agents, they achieve 32%/26% on SWE-Bench Verified/Lite, a new open SoTA.
+
+
+![Inference Time Scaling for Moatless Agent](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/inference-ml.jpg?raw=true)
+*Inference Time Scaling for Moatless Agent*
+
+![Inference Time Scaling for OpenHands Agent](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/inference-oh.jpg?raw=true)
+*Inference Time Scaling for OpenHands Agent*
+
+
+## Our baselines on SWE-Gym shows strong scaling trends
+
+Lastly, our ablations reveal strong scaling trends - performance is now bottlenecked by train and inference compute, rather than the size of our dataset. Pushing and improving these scaling trends further is an exciting direction for future work.
+
+![](https://github.com/SWE-Gym/SWE-Gym/blob/main/assets/images/scaling.jpg?raw=true)
+
+## Reproducing Results
+**The Dataset**
+
+To access SWE-Gym dataset, checkout our huggingface hub page [SWE-Gym](https://huggingface.co/SWE-Gym)
+
+The environment constants are currently saved at [SWE-Bench-Fork](https://github.com/SWE-Gym/SWE-Bench-Fork)
+
+We also have pre-built docker images for each instance under [xingyaoww/sweb.eval.x86_64](https://hub.docker.com/search?q=xingyaoww%2Fsweb.eval.x86_64.) prefix at docker hub.
+
+
+## 📚 Citation
+
+```bibtex
+@misc{pan2024trainingsoftwareengineeringagents,
+      title={Training Software Engineering Agents and Verifiers with SWE-Gym},
+      author={Jiayi Pan and Xingyao Wang and Graham Neubig and Navdeep Jaitly and Heng Ji and Alane Suhr and Yizhe Zhang},
+      year={2024},
+      eprint={2412.21139},
+      archivePrefix={arXiv},
+      primaryClass={cs.SE},
+      url={https://arxiv.org/abs/2412.21139},
+}
+```
@@ -6,17 +6,6 @@ import time
 from functools import partial

 import pandas as pd
-from swebench.harness.grading import get_eval_report
-from swebench.harness.run_evaluation import (
-    APPLY_PATCH_FAIL,
-    APPLY_PATCH_PASS,
-)
-from swebench.harness.test_spec.test_spec import (
-    SWEbenchInstance,
-    TestSpec,
-    make_test_spec,
-)
-from swebench.harness.utils import load_swebench_dataset
 from tqdm import tqdm

 from evaluation.benchmarks.swe_bench.resource.mapping import (
@@ -348,6 +337,31 @@ if __name__ == '__main__':
    )
    args, _ = parser.parse_known_args()

+    if 'SWE-Gym' in args.dataset:
+        from swegym.harness.grading import get_eval_report
+        from swegym.harness.run_evaluation import (
+            APPLY_PATCH_FAIL,
+            APPLY_PATCH_PASS,
+        )
+        from swegym.harness.test_spec import (
+            SWEbenchInstance,
+            TestSpec,
+            make_test_spec,
+        )
+        from swegym.harness.utils import load_swebench_dataset
+    else:  # Newer version of SWE-Bench have different import paths
+        from swebench.harness.grading import get_eval_report
+        from swebench.harness.run_evaluation import (
+            APPLY_PATCH_FAIL,
+            APPLY_PATCH_PASS,
+        )
+        from swebench.harness.test_spec.test_spec import (
+            SWEbenchInstance,
+            TestSpec,
+            make_test_spec,
+        )
+        from swebench.harness.utils import load_swebench_dataset
+
    # Load SWE-Bench dataset
    full_dataset: list[SWEbenchInstance] = load_swebench_dataset(
        args.dataset, args.split
@@ -44,7 +44,6 @@ from openhands.utils.async_utils import call_async_from_sync
 from openhands.utils.shutdown_listener import sleep_if_should_continue

 USE_HINT_TEXT = os.environ.get('USE_HINT_TEXT', 'false').lower() == 'true'
-USE_INSTANCE_IMAGE = os.environ.get('USE_INSTANCE_IMAGE', 'true').lower() == 'true'
 RUN_WITH_BROWSING = os.environ.get('RUN_WITH_BROWSING', 'false').lower() == 'true'


@@ -121,23 +120,18 @@ def get_config(
    instance: pd.Series,
    metadata: EvalMetadata,
 ) -> AppConfig:
-    SWE_BENCH_CONTAINER_IMAGE = 'ghcr.io/opendevin/eval-swe-bench:full-v1.2.1'
-    if USE_INSTANCE_IMAGE:
-        # We use a different instance image for the each instance of swe-bench eval
-        use_official_image = bool(
-            'verified' in metadata.dataset.lower() or 'lite' in metadata.dataset.lower()
-        )
-        base_container_image = get_instance_docker_image(
-            instance['instance_id'], use_official_image
-        )
-        logger.info(
-            f'Using instance container image: {base_container_image}. '
-            f'Please make sure this image exists. '
-            f'Submit an issue on https://github.com/All-Hands-AI/OpenHands if you run into any issues.'
-        )
-    else:
-        base_container_image = SWE_BENCH_CONTAINER_IMAGE
-        logger.info(f'Using swe-bench container image: {base_container_image}')
+    # We use a different instance image for the each instance of swe-bench eval
+    use_official_image = bool(
+        'verified' in metadata.dataset.lower() or 'lite' in metadata.dataset.lower()
+    )
+    base_container_image = get_instance_docker_image(
+        instance['instance_id'], use_official_image
+    )
+    logger.info(
+        f'Using instance container image: {base_container_image}. '
+        f'Please make sure this image exists. '
+        f'Submit an issue on https://github.com/All-Hands-AI/OpenHands if you run into any issues.'
+    )

    sandbox_config = get_default_sandbox_config_for_eval()
    sandbox_config.base_container_image = base_container_image
@@ -209,75 +203,65 @@ def initialize_runtime(
    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
    assert_and_raise(obs.exit_code == 0, f'Failed to export USER: {str(obs)}')

-    if USE_INSTANCE_IMAGE:
-        # inject the init script
-        script_dir = os.path.dirname(__file__)
+    # inject the init script
+    script_dir = os.path.dirname(__file__)

-        # inject the instance info
-        action = CmdRunAction(command='mkdir -p /swe_util/eval_data/instances')
-        action.set_hard_timeout(600)
-        logger.info(action, extra={'msg_type': 'ACTION'})
-        obs = runtime.run_action(action)
-        logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-        assert_and_raise(
-            obs.exit_code == 0,
-            f'Failed to create /swe_util/eval_data/instances: {str(obs)}',
-        )
+    # inject the instance info
+    action = CmdRunAction(command='mkdir -p /swe_util/eval_data/instances')
+    action.set_hard_timeout(600)
+    logger.info(action, extra={'msg_type': 'ACTION'})
+    obs = runtime.run_action(action)
+    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
+    assert_and_raise(
+        obs.exit_code == 0,
+        f'Failed to create /swe_util/eval_data/instances: {str(obs)}',
+    )

-        swe_instance_json_name = 'swe-bench-instance.json'
-        with tempfile.TemporaryDirectory() as temp_dir:
-            # Construct the full path for the desired file name within the temporary directory
-            temp_file_path = os.path.join(temp_dir, swe_instance_json_name)
-            # Write to the file with the desired name within the temporary directory
-            with open(temp_file_path, 'w') as f:
-                if not isinstance(instance, dict):
-                    json.dump([instance.to_dict()], f)
-                else:
-                    json.dump([instance], f)
+    swe_instance_json_name = 'swe-bench-instance.json'
+    with tempfile.TemporaryDirectory() as temp_dir:
+        # Construct the full path for the desired file name within the temporary directory
+        temp_file_path = os.path.join(temp_dir, swe_instance_json_name)
+        # Write to the file with the desired name within the temporary directory
+        with open(temp_file_path, 'w') as f:
+            if not isinstance(instance, dict):
+                json.dump([instance.to_dict()], f)
+            else:
+                json.dump([instance], f)

-            # Copy the file to the desired location
-            runtime.copy_to(temp_file_path, '/swe_util/eval_data/instances/')
+        # Copy the file to the desired location
+        runtime.copy_to(temp_file_path, '/swe_util/eval_data/instances/')

        # inject the instance swe entry
        runtime.copy_to(
            str(os.path.join(script_dir, 'scripts/setup/instance_swe_entry.sh')),
            '/swe_util/',
        )
-        action = CmdRunAction(command='cat ~/.bashrc')
-        action.set_hard_timeout(600)
-        logger.info(action, extra={'msg_type': 'ACTION'})
-        obs = runtime.run_action(action)
-        logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-        assert_and_raise(obs.exit_code == 0, f'Failed to cat ~/.bashrc: {str(obs)}')

-        action = CmdRunAction(command='source ~/.bashrc')
-        action.set_hard_timeout(600)
-        logger.info(action, extra={'msg_type': 'ACTION'})
-        obs = runtime.run_action(action)
-        logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-        if isinstance(obs, ErrorObservation):
-            logger.error(f'Failed to source ~/.bashrc: {str(obs)}')
-        assert_and_raise(obs.exit_code == 0, f'Failed to source ~/.bashrc: {str(obs)}')
+    action = CmdRunAction(command='cat ~/.bashrc')
+    action.set_hard_timeout(600)
+    logger.info(action, extra={'msg_type': 'ACTION'})
+    obs = runtime.run_action(action)
+    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
+    assert_and_raise(obs.exit_code == 0, f'Failed to cat ~/.bashrc: {str(obs)}')

-        action = CmdRunAction(command='source /swe_util/instance_swe_entry.sh')
-        action.set_hard_timeout(600)
-        logger.info(action, extra={'msg_type': 'ACTION'})
-        obs = runtime.run_action(action)
-        logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-        assert_and_raise(
-            obs.exit_code == 0,
-            f'Failed to source /swe_util/instance_swe_entry.sh: {str(obs)}',
-        )
-    else:
-        action = CmdRunAction(command='source /swe_util/swe_entry.sh')
-        action.set_hard_timeout(1800)
-        logger.info(action, extra={'msg_type': 'ACTION'})
-        obs = runtime.run_action(action)
-        logger.info(obs, extra={'msg_type': 'OBSERVATION'})
-        assert_and_raise(
-            obs.exit_code == 0,
-            f'Failed to source /swe_util/swe_entry.sh: {str(obs)}',
-        )
+    action = CmdRunAction(command='source ~/.bashrc')
+    action.set_hard_timeout(600)
+    logger.info(action, extra={'msg_type': 'ACTION'})
+    obs = runtime.run_action(action)
+    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
+    if isinstance(obs, ErrorObservation):
+        logger.error(f'Failed to source ~/.bashrc: {str(obs)}')
+    assert_and_raise(obs.exit_code == 0, f'Failed to source ~/.bashrc: {str(obs)}')
+
+    action = CmdRunAction(command='source /swe_util/instance_swe_entry.sh')
+    action.set_hard_timeout(600)
+    logger.info(action, extra={'msg_type': 'ACTION'})
+    obs = runtime.run_action(action)
+    logger.info(obs, extra={'msg_type': 'OBSERVATION'})
+    assert_and_raise(
+        obs.exit_code == 0,
+        f'Failed to source /swe_util/instance_swe_entry.sh: {str(obs)}',
+    )

    action = CmdRunAction(command=f'cd /workspace/{workspace_dir_name}')
    action.set_hard_timeout(600)
@@ -549,6 +533,20 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
    return dataset


+# A list of instances that are known to be tricky to infer
+# (will cause runtime failure even with resource factor = 8)
+SWEGYM_EXCLUDE_IDS = [
+    'dask__dask-10422',
+    'pandas-dev__pandas-50548',
+    'pandas-dev__pandas-53672',
+    'pandas-dev__pandas-54174',
+    'pandas-dev__pandas-55518',
+    'pandas-dev__pandas-58383',
+    'pydata__xarray-6721',
+    'pytest-dev__pytest-10081',
+    'pytest-dev__pytest-7236',
+]
+
 if __name__ == '__main__':
    parser = get_parser()
    parser.add_argument(
@@ -572,6 +570,13 @@ if __name__ == '__main__':
    logger.info(
        f'Loaded dataset {args.dataset} with split {args.split}: {len(swe_bench_tests)} tasks'
    )
+    if 'SWE-Gym' in args.dataset:
+        swe_bench_tests = swe_bench_tests[
+            ~swe_bench_tests['instance_id'].isin(SWEGYM_EXCLUDE_IDS)
+        ]
+        logger.info(
+            f'{len(swe_bench_tests)} tasks left after excluding SWE-Gym excluded tasks'
+        )

    llm_config = None
    if args.llm_config:
@@ -615,6 +620,6 @@ if __name__ == '__main__':
        output_file,
        args.eval_num_workers,
        process_instance,
-        timeout_seconds=120 * 60,  # 2 hour PER instance should be more than enough
+        timeout_seconds=8 * 60 * 60,  # 8 hour PER instance should be more than enough
        max_retries=5,
    )
@@ -24,7 +24,7 @@ def load_completions(output_dir: str, instance_id: str):
    # create messages
    messages = result['messages']
    messages.append(result['response']['choices'][0]['message'])
-    tools = result['kwargs']['tools']
+    tools = result['kwargs'].get('tools', [])
    return {
        'messages': messages,
        'tools': tools,
@@ -75,7 +75,7 @@ def load_completions(instance_id: str):
    # create messages
    messages = result['messages']
    messages.append(result['response']['choices'][0]['message'])
-    tools = result['kwargs']['tools']
+    tools = result['kwargs'].get('tools', None)
    return {
        'messages': messages,
        'tools': tools,
@@ -0,0 +1,132 @@
+#!/bin/bash
+
+# NOTE: this script is for rolling out the SWE-Gym dataset for **TRAINING**
+# For more information, please refer to
+# 1. the Github Repo: https://github.com/SWE-Gym/SWE-Gym
+# 2. the paper: https://arxiv.org/abs/2412.21139
+
+MODEL=$1  # eg your llm config name in config.toml (eg: "llm.claude-3-5-sonnet-20241022-t05")
+EXP_NAME=$2 # "train-t05"
+N_WORKERS=${3:-64}
+N_RUNS=${4:-1}
+
+export EXP_NAME=$EXP_NAME
+# use 2x resources for rollout since some codebases are pretty resource-intensive
+export DEFAULT_RUNTIME_RESOURCE_FACTOR=2
+echo "MODEL: $MODEL"
+echo "EXP_NAME: $EXP_NAME"
+DATASET="SWE-Gym/SWE-Gym"  # change this to the "/SWE-Gym-Lite" if you want to rollout the lite subset
+SPLIT="train"
+
+if [ -z "$ALLHANDS_API_KEY" ]; then
+    echo "ALLHANDS_API_KEY is not set. Will rollout and evaluate locally using Docker. WARNING: A large value of N_WORKERS will result in a large number of Docker containers being spun up and may crash your machine."
+    export RUNTIME=docker
+else
+    echo "ALLHANDS_API_KEY is set. Continuing rollout and evaluation with remote runtime..."
+    export RUNTIME=remote
+    export SANDBOX_REMOTE_RUNTIME_API_URL="https://runtime.eval.all-hands.dev"
+    export EVAL_DOCKER_IMAGE_PREFIX="us-central1-docker.pkg.dev/evaluation-092424/swe-bench-images"
+fi
+
+EVAL_LIMIT=3000
+MAX_ITER=100
+
+
+# ===== Run inference =====
+source "evaluation/utils/version_control.sh"
+get_openhands_version
+
+echo "OPENHANDS_VERSION: $OPENHANDS_VERSION"
+echo "MODEL_CONFIG: $MODEL_CONFIG"
+echo "DATASET: $DATASET"
+echo "SPLIT: $SPLIT"
+
+# Default to NOT use Hint
+export USE_INSTANCE_IMAGE=true
+export USE_HINT_TEXT=false
+export RUN_WITH_BROWSING=false
+echo "USE_HINT_TEXT: $USE_HINT_TEXT"
+EVAL_NOTE="$OPENHANDS_VERSION-no-hint-$EXP_NAME"
+
+function run_eval() {
+  local eval_note=$1
+  COMMAND="poetry run python evaluation/benchmarks/swe_bench/run_infer.py \
+    --agent-cls CodeActAgent \
+    --llm-config $MODEL \
+    --max-iterations $MAX_ITER \
+    --eval-num-workers $N_WORKERS \
+    --eval-note $eval_note \
+    --dataset $DATASET \
+    --split $SPLIT"
+
+  if [ -n "$EVAL_LIMIT" ]; then
+    echo "EVAL_LIMIT: $EVAL_LIMIT"
+    COMMAND="$COMMAND --eval-n-limit $EVAL_LIMIT"
+  fi
+
+  # Run the command
+  eval $COMMAND
+}
+
+for run_idx in $(seq 1 $N_RUNS); do
+
+    while true; do
+        echo "### Running inference... ###"
+        unset SANDBOX_ENV_GITHUB_TOKEN # prevent the agent from using the github token to push
+        current_eval_note="$EVAL_NOTE-run_$run_idx"
+        echo "EVAL_NOTE: $current_eval_note"
+        INFER_OUTPUT=$(run_eval $current_eval_note)
+        INFER_STATUS=$?  # Capture the exit status of run_infer.sh
+        echo "INFER_STATUS: $INFER_STATUS"
+
+        echo "### Cleaning up remote runtime... ###"
+        ./evaluation/utils/scripts/cleanup_remote_runtime.sh
+
+        if [ $INFER_STATUS -eq 0 ]; then
+            echo "### Inference completed successfully. ###"
+            break
+        else
+            echo "### Inference failed with exit code $INFER_STATUS. Retrying... ###"
+        fi
+    done
+
+    # Extract the output directory using the special delimiters
+    OUTPUT_FILE=$(echo "$INFER_OUTPUT" | grep -o '### OUTPUT FILE:.* ###' | sed 's/### OUTPUT FILE: \(.*\) ###/\1/')
+    echo "Got OUTPUT_FILE: $OUTPUT_FILE"
+
+    while true; do
+        echo "### Evaluating on $OUTPUT_FILE ... ###"
+        COMMAND="poetry run python evaluation/benchmarks/swe_bench/eval_infer.py \
+        --eval-num-workers $((N_WORKERS * 2)) \
+        --input-file $OUTPUT_FILE \
+        --dataset $DATASET \
+        --split $SPLIT"
+
+        if [ -n "$EVAL_LIMIT" ]; then
+        echo "EVAL_LIMIT: $EVAL_LIMIT"
+        COMMAND="$COMMAND --eval-n-limit $EVAL_LIMIT"
+        fi
+        echo "Running command: $COMMAND"
+        # Run the command
+        eval $COMMAND
+        EVAL_STATUS=$?
+        if [ $EVAL_STATUS -eq 0 ]; then
+            echo "### Evaluation completed successfully. ###"
+            break
+        else
+            echo "### Evaluation failed with exit code $EVAL_STATUS. Retrying... ###"
+        fi
+
+        ./evaluation/utils/scripts/cleanup_remote_runtime.sh
+    done
+
+    # update the output with evaluation results
+    echo "### Updating the output with evaluation results... ###"
+    poetry run python evaluation/benchmarks/swe_bench/scripts/eval/update_output_with_eval.py $OUTPUT_FILE
+
+    echo "### Combining the final completions... ###"
+    poetry run python evaluation/benchmarks/swe_bench/scripts/eval/combine_final_completions.py $OUTPUT_FILE
+
+    echo "### DONE for run $run_idx! ###"
+    echo "You can find the final output at $(dirname $OUTPUT_FILE)/$FINAL_OUTPUT_FILE"
+done
@@ -29,11 +29,6 @@ if [ -z "$MAX_ITER" ]; then
  MAX_ITER=100
 fi

-if [ -z "$USE_INSTANCE_IMAGE" ]; then
-  echo "USE_INSTANCE_IMAGE not specified, use default true"
-  USE_INSTANCE_IMAGE=true
-fi
-
 if [ -z "$RUN_WITH_BROWSING" ]; then
  echo "RUN_WITH_BROWSING not specified, use default false"
  RUN_WITH_BROWSING=false
@@ -50,8 +45,6 @@ if [ -z "$SPLIT" ]; then
  SPLIT="test"
 fi

-export USE_INSTANCE_IMAGE=$USE_INSTANCE_IMAGE
-echo "USE_INSTANCE_IMAGE: $USE_INSTANCE_IMAGE"
 export RUN_WITH_BROWSING=$RUN_WITH_BROWSING
 echo "RUN_WITH_BROWSING: $RUN_WITH_BROWSING"

@@ -0,0 +1,287 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import pandas as pd\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "tqdm.pandas()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1. Load raw data and convert to training data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import gzip\n",
+    "import json\n",
+    "\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "FILE_PATHS = [\n",
+    "    'YOURPATH-no-hint-train-t05-run_1/output.with_completions.jsonl.gz',\n",
+    "    'YOURPATH-no-hint-train-t05-run_2/output.with_completions.jsonl.gz',\n",
+    "]\n",
+    "\n",
+    "# More memory efficient for large files\n",
+    "# Initialize lists to store the data\n",
+    "data = []\n",
+    "\n",
+    "\n",
+    "# Read file line by line\n",
+    "for FILE_PATH in FILE_PATHS:\n",
+    "    with gzip.open(FILE_PATH, 'rb') as f:  # Use 'rb' for gzipped files\n",
+    "        for i, line in tqdm(\n",
+    "            enumerate(f), desc=f\"Processing {FILE_PATH.split('/')[-1]}\"\n",
+    "        ):\n",
+    "            # Parse only the fields we need\n",
+    "            raw_data = json.loads(line)\n",
+    "            data.append(\n",
+    "                {\n",
+    "                    'resolved': raw_data['report']['resolved'],\n",
+    "                    'messages': raw_data['raw_completions']['messages']\n",
+    "                    if raw_data['raw_completions'] is not None\n",
+    "                    else None,\n",
+    "                    'git_patch': raw_data['test_result'].get('git_patch', ''),\n",
+    "                    'tools': raw_data['raw_completions']['tools']\n",
+    "                    if raw_data['raw_completions'] is not None\n",
+    "                    and 'tools' in raw_data['raw_completions']\n",
+    "                    else None,\n",
+    "                }\n",
+    "            )\n",
+    "\n",
+    "# Convert to DataFrame after collecting all data\n",
+    "df = pd.DataFrame(data)\n",
+    "print(f'#total amount of data={len(df)}')\n",
+    "df = df[~df['messages'].isna()]\n",
+    "print(f'#total amount of data after removing nan={len(df)}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Filter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def _contains_multiple_tool_calls(messages: list[dict]) -> bool:\n",
+    "    return any(\n",
+    "        message.get('tool_calls') and len(message['tool_calls']) > 1\n",
+    "        for message in messages\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "df['contains_multiple_tool_calls'] = df['messages'].apply(_contains_multiple_tool_calls)\n",
+    "display(df.groupby(['contains_multiple_tool_calls'])['resolved'].sum())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import copy\n",
+    "\n",
+    "# Convert function calling messages to non-function calling messages\n",
+    "from openhands.llm.fn_call_converter import (\n",
+    "    FunctionCallConversionError,\n",
+    "    convert_fncall_messages_to_non_fncall_messages,\n",
+    "    convert_from_multiple_tool_calls_to_single_tool_call_messages,\n",
+    ")\n",
+    "\n",
+    "total_failed = 0\n",
+    "\n",
+    "\n",
+    "def _convert_messages(messages: list[dict], tools: list[dict]) -> list[dict]:\n",
+    "    global total_failed\n",
+    "    message_copy = copy.deepcopy(messages)\n",
+    "    for message in message_copy:\n",
+    "        if message['content'] is None:\n",
+    "            message['content'] = ''\n",
+    "    try:\n",
+    "        return convert_fncall_messages_to_non_fncall_messages(\n",
+    "            message_copy, tools, add_in_context_learning_example=False\n",
+    "        )\n",
+    "    except FunctionCallConversionError:\n",
+    "        total_failed += 1\n",
+    "        # print(f'Failed to convert messages: {messages}\\nTools: {tools}')\n",
+    "        # traceback.print_exc()\n",
+    "        return None\n",
+    "\n",
+    "\n",
+    "df['converted_messages'] = df.apply(\n",
+    "    lambda row: convert_from_multiple_tool_calls_to_single_tool_call_messages(\n",
+    "        row['messages'], ignore_final_tool_result=True\n",
+    "    ),\n",
+    "    axis=1,\n",
+    ")\n",
+    "df['nonfncall_messages'] = df.apply(\n",
+    "    lambda row: _convert_messages(row['converted_messages'], row['tools']), axis=1\n",
+    ")\n",
+    "print('total nan', df['nonfncall_messages'].isna().sum())\n",
+    "df = df[~df['nonfncall_messages'].isna()]\n",
+    "print(f'Total failed: {total_failed}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tokenization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pandarallel import pandarallel\n",
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "os.environ['TOKENIZERS_PARALLELISM'] = 'false'\n",
+    "pandarallel.initialize(progress_bar=True, verbose=1, nb_workers=16)\n",
+    "tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')\n",
+    "df['n_tokens'] = df['rm_conv'].parallel_apply(\n",
+    "    lambda x: len(tokenizer.apply_chat_template(x))\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f'BEFORE: #total={len(df)}')\n",
+    "df_selected = df[df['n_tokens'] < 131072]\n",
+    "print(f'AFTER(truncated to 128k): #total={len(df_selected)}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_selected['n_tokens'].describe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ecdf of n_tokens\n",
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "\n",
+    "display(df.groupby(['resolved'])['n_tokens'].describe())\n",
+    "sns.ecdfplot(x='n_tokens', data=df, hue='resolved')\n",
+    "plt.show()\n",
+    "\n",
+    "print(f'#total={len(df)}')\n",
+    "df_selected = df[df['n_tokens'] < 131072]\n",
+    "print(f'#selected={len(df_selected)}')\n",
+    "display(df_selected.groupby(['resolved'])['n_tokens'].describe())\n",
+    "sns.ecdfplot(x='n_tokens', data=df_selected, hue='resolved')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_selected[~df_selected['resolved']]['n_tokens'].describe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_selected['resolved'].value_counts()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_selected.groupby(['resolved'])['n_tokens'].describe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Save Resolved Messages for SFT"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_selected[df_selected['resolved']][['nonfncall_messages']].rename(\n",
+    "    columns={'nonfncall_messages': 'messages'}\n",
+    ").to_json(\n",
+    "    os.path.join(\n",
+    "        'YOUR_OUTPUT_FOLDER',\n",
+    "        f'policy_traj_128k_swegym_{df_selected[\"resolved\"].value_counts()[True]}i.jsonl',\n",
+    "    ),\n",
+    "    lines=True,\n",
+    "    orient='records',\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openhands-ai-CPy6G0pU-py3.12",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -1,12 +1,12 @@
 {
  "name": "openhands-frontend",
-  "version": "0.27.0",
+  "version": "0.28.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "openhands-frontend",
-      "version": "0.27.0",
+      "version": "0.28.0",
      "dependencies": {
        "@heroui/react": "2.7.4",
        "@monaco-editor/react": "^4.7.0-rc.0",
@@ -1,6 +1,6 @@
 {
  "name": "openhands-frontend",
-  "version": "0.27.0",
+  "version": "0.28.0",
  "private": true,
  "type": "module",
  "engines": {
@@ -14,10 +14,7 @@ import { generateAgentStateChangeEvent } from "#/services/agent-state-service";
 import { FeedbackModal } from "../feedback/feedback-modal";
 import { useScrollToBottom } from "#/hooks/use-scroll-to-bottom";
 import { TypingIndicator } from "./typing-indicator";
-import {
-  useWsClient,
-  WsClientProviderStatus,
-} from "#/context/ws-client-provider";
+import { useWsClient } from "#/context/ws-client-provider";
 import { Messages } from "./messages";
 import { ChatSuggestions } from "./chat-suggestions";
 import { ActionSuggestions } from "./action-suggestions";
@@ -37,7 +34,7 @@ function getEntryPoint(
 }

 export function ChatInterface() {
-  const { send, isLoadingMessages, status, pendingMessages } = useWsClient();
+  const { send, isLoadingMessages } = useWsClient();
  const dispatch = useDispatch();
  const scrollRef = React.useRef<HTMLDivElement>(null);
  const { scrollDomToBottom, onChatBodyScroll, hitBottom } =
@@ -57,9 +54,6 @@ export function ChatInterface() {
  const params = useParams();
  const { mutate: getTrajectory } = useGetTrajectory();

-  const isClientDisconnected = status === WsClientProviderStatus.DISCONNECTED;
-  const hasPendingMessages = pendingMessages.length > 0;
-
  const handleSendMessage = async (content: string, files: File[]) => {
    if (messages.length === 0) {
      posthog.capture("initial_query_submitted", {
@@ -82,16 +76,7 @@ export function ChatInterface() {
    const timestamp = new Date().toISOString();
    const pending = true;
    dispatch(addUserMessage({ content, imageUrls, timestamp, pending }));
-
-    // Create the chat message
-    const chatMessage = createChatMessage(content, imageUrls, timestamp);
-
-    // Send or queue the message depending on connection status
-    send(chatMessage);
-
-    // Set agent state to RUNNING when a message is sent
-    send(generateAgentStateChangeEvent(AgentState.RUNNING));
-
+    send(createChatMessage(content, imageUrls, timestamp));
    setMessageToSend(null);
  };

@@ -146,20 +131,8 @@ export function ChatInterface() {
        className="flex flex-col grow overflow-y-auto overflow-x-hidden px-4 pt-4 gap-2"
      >
        {isLoadingMessages && (
-          <div className="flex flex-col items-center gap-2">
+          <div className="flex justify-center">
            <LoadingSpinner size="small" />
-            {isClientDisconnected && (
-              <div className="text-sm text-neutral-400">
-                Waiting for client to become ready...
-                {hasPendingMessages && (
-                  <div className="text-xs text-neutral-500 mt-1">
-                    {pendingMessages.length} message
-                    {pendingMessages.length !== 1 ? "s" : ""} will be sent when
-                    connected
-                  </div>
-                )}
-              </div>
-            )}
          </div>
        )}

@@ -206,7 +179,7 @@ export function ChatInterface() {
          onSubmit={handleSendMessage}
          onStop={handleStop}
          isDisabled={
-            // Allow input even when loading, but not during confirmation
+            curAgentState === AgentState.LOADING ||
            curAgentState === AgentState.AWAITING_USER_CONFIRMATION
          }
          mode={curAgentState === AgentState.RUNNING ? "stop" : "submit"}
@@ -22,11 +22,10 @@ export function AgentStatusBar() {
  const { t, i18n } = useTranslation();
  const { curAgentState } = useSelector((state: RootState) => state.agent);
  const { curStatusMessage } = useSelector((state: RootState) => state.status);
-  const { status, pendingMessages } = useWsClient();
+  const { status } = useWsClient();
  const { notify } = useNotification();

  const [statusMessage, setStatusMessage] = React.useState<string>("");
-  const hasPendingMessages = pendingMessages.length > 0;

  const updateStatusMessage = () => {
    let message = curStatusMessage.message || "";
@@ -72,13 +71,7 @@ export function AgentStatusBar() {

  React.useEffect(() => {
    if (status === WsClientProviderStatus.DISCONNECTED) {
-      if (hasPendingMessages) {
-        setStatusMessage(
-          `Connecting... (${pendingMessages.length} pending message${pendingMessages.length !== 1 ? "s" : ""})`,
-        );
-      } else {
-        setStatusMessage("Connecting...");
-      }
+      setStatusMessage("Connecting...");
    } else {
      setStatusMessage(AGENT_STATUS_MAP[curAgentState].message);
      if (notificationStates.includes(curAgentState)) {
@@ -94,7 +87,7 @@ export function AgentStatusBar() {
        }
      }
    }
-  }, [curAgentState, status, pendingMessages.length, notify, t]);
+  }, [curAgentState, notify, t]);

  return (
    <div className="flex flex-col items-center">
@@ -1,9 +1,10 @@
 import { Autocomplete, AutocompleteItem } from "@heroui/react";
+import { ReactNode } from "react";
 import { OptionalTag } from "./optional-tag";

 interface SettingsDropdownInputProps {
  testId: string;
-  label: string;
+  label: ReactNode;
  name: string;
  items: { key: React.Key; label: string }[];
  showOptionalTag?: boolean;
@@ -29,7 +30,7 @@ export function SettingsDropdownInput({
        {showOptionalTag && <OptionalTag />}
      </div>
      <Autocomplete
-        aria-label={label}
+        aria-label={typeof label === "string" ? label : name}
        data-testid={testId}
        name={name}
        defaultItems={items}
@@ -43,15 +43,18 @@ export function SettingsForm({ settings, models, onClose }: SettingsFormProps) {
  const handleFormSubmission = async (formData: FormData) => {
    const newSettings = extractSettings(formData);

-    await saveUserSettings(newSettings);
-    onClose();
-    resetOngoingSession();
+    await saveUserSettings(newSettings, {
+      onSuccess: () => {
+        onClose();
+        resetOngoingSession();

-    posthog.capture("settings_saved", {
-      LLM_MODEL: newSettings.LLM_MODEL,
-      LLM_API_KEY: newSettings.LLM_API_KEY ? "SET" : "UNSET",
-      REMOTE_RUNTIME_RESOURCE_FACTOR:
-        newSettings.REMOTE_RUNTIME_RESOURCE_FACTOR,
+        posthog.capture("settings_saved", {
+          LLM_MODEL: newSettings.LLM_MODEL,
+          LLM_API_KEY: newSettings.LLM_API_KEY ? "SET" : "UNSET",
+          REMOTE_RUNTIME_RESOURCE_FACTOR:
+            newSettings.REMOTE_RUNTIME_RESOURCE_FACTOR,
+        });
+      },
    });
  };

@@ -39,14 +39,6 @@ const isMessageAction = (
 ): event is UserMessageAction | AssistantMessageAction =>
  isUserMessage(event) || isAssistantMessage(event);

-// Check if an event is an agent state changed observation
-const isAgentStateEvent = (event: Record<string, unknown>): boolean => 
-  isOpenHandsEvent(event) && 
-  "type" in event && 
-  event.type === "observation" && 
-  "observation_id" in event && 
-  event.observation_id === "agent_state_changed";
-
 export enum WsClientProviderStatus {
  CONNECTED,
  DISCONNECTED,
@@ -57,21 +49,15 @@ interface UseWsClient {
  isLoadingMessages: boolean;
  events: Record<string, unknown>[];
  send: (event: Record<string, unknown>) => void;
-  queueMessage: (event: Record<string, unknown>) => void;
-  pendingMessages: Record<string, unknown>[];
 }

 const WsClientContext = React.createContext<UseWsClient>({
  status: WsClientProviderStatus.DISCONNECTED,
  isLoadingMessages: true,
  events: [],
-  pendingMessages: [],
  send: () => {
    throw new Error("not connected");
  },
-  queueMessage: () => {
-    throw new Error("not connected");
-  },
 });

 interface WsClientProviderProps {
@@ -123,50 +109,26 @@ export function WsClientProvider({
    WsClientProviderStatus.DISCONNECTED,
  );
  const [events, setEvents] = React.useState<Record<string, unknown>[]>([]);
-  const [pendingMessages, setPendingMessages] = React.useState<
-    Record<string, unknown>[]
-  >([]);
-  const [backendReady, setBackendReady] = React.useState(false);
  const lastEventRef = React.useRef<Record<string, unknown> | null>(null);

  const messageRateHandler = useRate({ threshold: 250 });

-  function queueMessage(event: Record<string, unknown>) {
-    setPendingMessages((prev) => [...prev, event]);
-  }
-
  function send(event: Record<string, unknown>) {
    if (!sioRef.current) {
      EventLogger.error("WebSocket is not connected.");
-      queueMessage(event);
      return;
    }
-    
-    if (!backendReady) {
-      // If backend is not ready yet, queue the message
-      EventLogger.info("Backend not ready, queueing message");
-      queueMessage(event);
-      return;
-    }
-    
    sioRef.current.emit("oh_action", event);
  }

  function handleConnect() {
    setStatus(WsClientProviderStatus.CONNECTED);
-    // Don't send queued messages yet - wait for backend ready signal
  }

  function handleMessage(event: Record<string, unknown>) {
    if (isOpenHandsEvent(event) && isMessageAction(event)) {
      messageRateHandler.record(new Date().getTime());
    }
-    
-    // Check if this is a state change event indicating backend is ready
-    if (isAgentStateEvent(event)) {
-      setBackendReady(true);
-    }
-    
    setEvents((prevEvents) => [...prevEvents, event]);
    if (!Number.isNaN(parseInt(event.id as string, 10))) {
      lastEventRef.current = event;
@@ -177,7 +139,6 @@ export function WsClientProvider({

  function handleDisconnect(data: unknown) {
    setStatus(WsClientProviderStatus.DISCONNECTED);
-    setBackendReady(false);
    const sio = sioRef.current;
    if (!sio) {
      return;
@@ -189,34 +150,11 @@ export function WsClientProvider({

  function handleError(data: unknown) {
    setStatus(WsClientProviderStatus.DISCONNECTED);
-    setBackendReady(false);
    updateStatusWhenErrorMessagePresent(data);
  }

-  // Watch for backend ready state and send queued messages when ready
-  React.useEffect(() => {
-    if (backendReady && pendingMessages.length > 0 && sioRef.current) {
-      // Backend is ready and we have pending messages
-      EventLogger.info(`Sending ${pendingMessages.length} queued messages`);
-      
-      pendingMessages.forEach((event) => {
-        sioRef.current?.emit("oh_action", event);
-      });
-
-      // Also set the agent state to RUNNING if needed
-      const agentStateEvent = {
-        action: "change_agent_state",
-        args: { agent_state: "running" },
-      };
-      sioRef.current.emit("oh_action", agentStateEvent);
-
-      setPendingMessages([]);
-    }
-  }, [backendReady, pendingMessages.length]);
-
  React.useEffect(() => {
    lastEventRef.current = null;
-    setBackendReady(false);
  }, [conversationId]);

  React.useEffect(() => {
@@ -272,11 +210,9 @@ export function WsClientProvider({
      status,
      isLoadingMessages: messageRateHandler.isUnderThreshold,
      events,
-      pendingMessages,
      send,
-      queueMessage,
    }),
-    [status, messageRateHandler.isUnderThreshold, events, pendingMessages],
+    [status, messageRateHandler.isUnderThreshold, events],
  );

  return <WsClientContext value={value}>{children}</WsClientContext>;
@@ -3,7 +3,6 @@ import React from "react";
 import posthog from "posthog-js";
 import OpenHands from "#/api/open-hands";
 import { useAuth } from "#/context/auth-context";
-import { useConfig } from "#/hooks/query/use-config";
 import { DEFAULT_SETTINGS } from "#/services/settings";

 const getSettingsQueryFn = async () => {
@@ -27,12 +26,10 @@ const getSettingsQueryFn = async () => {

 export const useSettings = () => {
  const { setGitHubTokenIsSet, githubTokenIsSet } = useAuth();
-  const { data: config } = useConfig();

  const query = useQuery({
    queryKey: ["settings", githubTokenIsSet],
    queryFn: getSettingsQueryFn,
-    enabled: config?.APP_MODE !== "saas" || githubTokenIsSet,
    // Only retry if the error is not a 404 because we
    // would want to show the modal immediately if the
    // settings are not found
@@ -278,7 +278,15 @@ function AccountSettings() {
              <SettingsDropdownInput
                testId="runtime-settings-input"
                name="runtime-settings-input"
-                label="Runtime Settings"
+                label={
+                  <>
+                    Runtime Settings (
+                    <a href="mailto:contact@all-hands.dev">
+                      get in touch for access
+                    </a>
+                    )
+                  </>
+                }
                items={REMOTE_RUNTIME_OPTIONS}
                defaultSelectedKey={settings.REMOTE_RUNTIME_RESOURCE_FACTOR?.toString()}
                isDisabled
@@ -95,6 +95,7 @@ export function handleObservationMessage(message: ObservationMessage) {
            observation,
            extras: {
              path: String(message.extras.path || ""),
+              impl_source: String(message.extras.impl_source || ""),
            },
          }),
        );
@@ -107,6 +108,7 @@ export function handleObservationMessage(message: ObservationMessage) {
            extras: {
              path: String(message.extras.path || ""),
              diff: String(message.extras.diff || ""),
+              impl_source: String(message.extras.impl_source || ""),
            },
          }),
        );
@@ -159,9 +159,16 @@ export const chatSlice = createSlice({
          .includes("error:");
      } else if (observationID === "read" || observationID === "edit") {
        // For read/edit operations, we consider it successful if there's content and no error
-        causeMessage.success =
-          observation.payload.content.length > 0 &&
-          !observation.payload.content.toLowerCase().includes("error:");
+
+        if (observation.payload.extras.impl_source === "oh_aci") {
+          causeMessage.success =
+            observation.payload.content.length > 0 &&
+            !observation.payload.content.startsWith("ERROR:\n");
+        } else {
+          causeMessage.success =
+            observation.payload.content.length > 0 &&
+            !observation.payload.content.toLowerCase().includes("error:");
+        }
      }

      if (observationID === "run" || observationID === "run_ipython") {
@@ -63,6 +63,7 @@ export interface ReadObservation extends OpenHandsObservationEvent<"read"> {
  source: "agent";
  extras: {
    path: string;
+    impl_source: string;
  };
 }

@@ -71,6 +72,7 @@ export interface EditObservation extends OpenHandsObservationEvent<"edit"> {
  extras: {
    path: string;
    diff: string;
+    impl_source: string;
  };
 }

@@ -37,16 +37,6 @@ class EventLogger {
    }
  }

-  /**
-   * Log an info message
-   * @param info The info message
-   */
-  static info(info: string) {
-    if (this.isDevMode) {
-      console.info(info);
-    }
-  }
-
  /**
   * Log an error message
   * @param error The error message
@@ -2,6 +2,7 @@
 * Generates a URL to redirect to for GitHub OAuth
 * @param clientId The GitHub OAuth client ID
 * @param requestUrl The URL of the request
+ * @param offline True for offline session, defaults to false
 * @returns The URL to redirect to for GitHub OAuth
 */
 export const generateGitHubAuthUrl = (clientId: string, requestUrl: URL) => {
@@ -10,6 +11,6 @@ export const generateGitHubAuthUrl = (clientId: string, requestUrl: URL) => {
    .replace(/(^|\.)staging\.all-hands\.dev$/, "$1auth.staging.all-hands.dev")
    .replace(/(^|\.)app\.all-hands\.dev$/, "auth.app.all-hands.dev")
    .replace(/(^|\.)localhost$/, "auth.staging.all-hands.dev");
-  const scope = "openid email profile offline_access";
+  const scope = "openid email profile";
  return `https://${authUrl}/realms/allhands/protocol/openid-connect/auth?client_id=github&response_type=code&redirect_uri=${encodeURIComponent(redirectUri)}&scope=${encodeURIComponent(scope)}`;
 };
@@ -8,8 +8,9 @@ At the user's request, repository {{ repository_info.repo_name }} has been clone
 {{ repository_instructions }}
 </REPOSITORY_INSTRUCTIONS>
 {% endif %}
-{% if runtime_info and runtime_info.available_hosts -%}
+{% if runtime_info and (runtime_info.available_hosts or runtime_info.additional_agent_instructions) -%}
 <RUNTIME_INFORMATION>
+{% if runtime_info.available_hosts %}
 The user has access to the following hosts for accessing a web application,
 each of which has a corresponding port:
 {% for host, port in runtime_info.available_hosts.items() -%}
@@ -18,5 +19,7 @@ each of which has a corresponding port:
 When starting a web server, use the corresponding ports. You should also
 set any options to allow iframes and CORS requests, and allow the server to
 be accessed from any host (e.g. 0.0.0.0).
+{% endif %}
+{{ runtime_info.additional_agent_instructions }}
 </RUNTIME_INFORMATION>
 {% endif %}
@@ -27,6 +27,17 @@ class ObservationMaskingCondenserConfig(BaseModel):
    model_config = {'extra': 'forbid'}


+class BrowserOutputCondenserConfig(BaseModel):
+    """Configuration for the BrowserOutputCondenser."""
+
+    type: Literal['browser_output_masking'] = Field('browser_output_masking')
+    attention_window: int = Field(
+        default=1,
+        description='The number of most recent browser output observations that will not be masked.',
+        ge=1,
+    )
+
+
 class RecentEventsCondenserConfig(BaseModel):
    """Configuration for RecentEventsCondenser."""

@@ -115,6 +126,7 @@ class LLMAttentionCondenserConfig(BaseModel):
 CondenserConfig = (
    NoOpCondenserConfig
    | ObservationMaskingCondenserConfig
+    | BrowserOutputCondenserConfig
    | RecentEventsCondenserConfig
    | LLMSummarizingCondenserConfig
    | AmortizedForgettingCondenserConfig
@@ -26,8 +26,10 @@ class GitHubService:
        user_id: str | None = None,
        idp_token: SecretStr | None = None,
        token: SecretStr | None = None,
+        external_token_manager: bool = False,
    ):
        self.user_id = user_id
+        self.external_token_manager = external_token_manager

        if token:
            self.token = token
@@ -418,6 +418,8 @@ class LLM(RetryMixin, DebugMixin):
                    self.model_info['max_tokens'], int
                ):
                    self.config.max_output_tokens = self.model_info['max_tokens']
+            if 'claude-3-7-sonnet' in self.config.model:
+                self.config.max_output_tokens = 64000  # litellm set max to 128k, but that requires a header to be set

        # Initialize function calling capability
        # Check if model name is in our supported list
@@ -1,6 +1,9 @@
 from openhands.memory.condenser.impl.amortized_forgetting_condenser import (
    AmortizedForgettingCondenser,
 )
+from openhands.memory.condenser.impl.browser_output_condenser import (
+    BrowserOutputCondenser,
+)
 from openhands.memory.condenser.impl.llm_attention_condenser import (
    ImportantEventSelection,
    LLMAttentionCondenser,
@@ -23,5 +26,6 @@ __all__ = [
    'LLMSummarizingCondenser',
    'NoOpCondenser',
    'ObservationMaskingCondenser',
+    'BrowserOutputCondenser',
    'RecentEventsCondenser',
 ]
@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+from openhands.core.config.condenser_config import BrowserOutputCondenserConfig
+from openhands.events.event import Event
+from openhands.events.observation import BrowserOutputObservation
+from openhands.events.observation.agent import AgentCondensationObservation
+from openhands.memory.condenser.condenser import Condenser
+
+
+class BrowserOutputCondenser(Condenser):
+    """A condenser that masks the observations from browser outputs outside of a recent attention window.
+    
+    The intent here is to mask just the browser outputs and leave everything else untouched. This is important because currently we provide screenshots and accessibility trees as input to the model for browser observations. These are really large and consume a lot of tokens without any benefits in performance. So we want to mask all such observations from all previous timesteps, and leave only the most recent one in context.
+    """
+
+    def __init__(self, attention_window: int = 1):
+        self.attention_window = attention_window
+        super().__init__()
+
+    def condense(self, events: list[Event]) -> list[Event]:
+        """Replace the content of browser observations outside of the attention window with a placeholder."""
+        results: list[Event] = []
+        cnt: int = 0
+        for event in reversed(events):
+            if (
+                isinstance(event, BrowserOutputObservation)
+                and cnt >= self.attention_window
+            ):
+                results.append(
+                    AgentCondensationObservation(
+                        f'Current URL: {event.url}\nContent Omitted'
+                    )
+                )
+            else:
+                results.append(event)
+                if isinstance(event, BrowserOutputObservation):
+                    cnt += 1
+
+        return list(reversed(results))
+
+    @classmethod
+    def from_config(
+        cls, config: BrowserOutputCondenserConfig
+    ) -> BrowserOutputCondenser:
+        return BrowserOutputCondenser(**config.model_dump(exclude=['type']))
+
+
+BrowserOutputCondenser.register_config(BrowserOutputCondenserConfig)
@@ -221,7 +221,9 @@ class Runtime(FileEditRuntimeMixin):
        try:
            if isinstance(event, CmdRunAction):
                if self.github_user_id and '$GITHUB_TOKEN' in event.command:
-                    gh_client = GithubServiceImpl(user_id=self.github_user_id)
+                    gh_client = GithubServiceImpl(
+                        user_id=self.github_user_id, external_token_manager=True
+                    )
                    token = await gh_client.get_latest_provider_token()
                    if token:
                        export_cmd = CmdRunAction(
@@ -487,3 +489,7 @@ class Runtime(FileEditRuntimeMixin):
    @property
    def web_hosts(self) -> dict[str, int]:
        return {}
+
+    @property
+    def additional_agent_instructions(self) -> str:
+        return ''
@@ -260,3 +260,7 @@ class DaytonaRuntime(ActionExecutionClient):
        )

        return self._vscode_url
+
+    @property
+    def additional_agent_instructions(self) -> str:
+        return f'When showing endpoints to access applications for any port, e.g. port 3000, instead of localhost:3000, use this format: {self._construct_api_url(3000)}.'
@@ -11,7 +11,7 @@ app = APIRouter(prefix='/api/conversations/{conversation_id}')


@app.post('/submit-feedback')
-async def submit_feedback(request: Request, conversation_id: str):
+async def submit_feedback(request: Request, conversation_id: str) -> JSONResponse:
    """Submit user feedback.

    This function stores the provided feedback data.
@@ -15,7 +15,7 @@ from openhands.server.auth import get_github_token, get_idp_token, get_user_id
 app = APIRouter(prefix='/api/github')


-@app.get('/repositories')
+@app.get('/repositories', response_model=list[GitHubRepository])
 async def get_github_repositories(
    page: int = 1,
    per_page: int = 10,
@@ -47,7 +47,7 @@ async def get_github_repositories(
        )


-@app.get('/user')
+@app.get('/user', response_model=GitHubUser)
 async def get_github_user(
    github_user_id: str | None = Depends(get_user_id),
    github_user_token: SecretStr | None = Depends(get_github_token),
@@ -73,7 +73,7 @@ async def get_github_user(
        )


-@app.get('/installations')
+@app.get('/installations', response_model=list[int])
 async def get_github_installation_ids(
    github_user_id: str | None = Depends(get_user_id),
    github_user_token: SecretStr | None = Depends(get_github_token),
@@ -99,7 +99,7 @@ async def get_github_installation_ids(
        )


-@app.get('/search/repositories')
+@app.get('/search/repositories', response_model=list[GitHubRepository])
 async def search_github_repositories(
    query: str,
    per_page: int = 5,
@@ -131,7 +131,7 @@ async def search_github_repositories(
        )


-@app.get('/suggested-tasks')
+@app.get('/suggested-tasks', response_model=list[SuggestedTask])
 async def get_suggested_tasks(
    github_user_id: str | None = Depends(get_user_id),
    github_user_token: SecretStr | None = Depends(get_github_token),
@@ -1,6 +1,8 @@
 import warnings
+from typing import Any

 import requests
+from fastapi import APIRouter

 from openhands.security.options import SecurityAnalyzers

@@ -8,10 +10,6 @@ with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    import litellm

-from fastapi import (
-    APIRouter,
-)
-
 from openhands.controller.agent import Agent
 from openhands.core.config import LLMConfig
 from openhands.core.logger import openhands_logger as logger
@@ -21,7 +19,7 @@ from openhands.server.shared import config, server_config
 app = APIRouter(prefix='/api/options')


-@app.get('/models')
+@app.get('/models', response_model=list[str])
 async def get_litellm_models() -> list[str]:
    """Get all models supported by LiteLLM.

@@ -34,7 +32,7 @@ async def get_litellm_models() -> list[str]:
    ```

    Returns:
-        list: A sorted list of unique model names.
+        list[str]: A sorted list of unique model names.
    """
    litellm_model_list = litellm.model_list + list(litellm.model_cost.keys())
    litellm_model_list_without_bedrock = bedrock.remove_error_modelId(
@@ -74,8 +72,8 @@ async def get_litellm_models() -> list[str]:
    return list(sorted(set(model_list)))


-@app.get('/agents')
-async def get_agents():
+@app.get('/agents', response_model=list[str])
+async def get_agents() -> list[str]:
    """Get all agents supported by LiteLLM.

    To get the agents:
@@ -84,14 +82,13 @@ async def get_agents():
    ```

    Returns:
-        list: A sorted list of agent names.
+        list[str]: A sorted list of agent names.
    """
-    agents = sorted(Agent.list_agents())
-    return agents
+    return sorted(Agent.list_agents())


-@app.get('/security-analyzers')
-async def get_security_analyzers():
+@app.get('/security-analyzers', response_model=list[str])
+async def get_security_analyzers() -> list[str]:
    """Get all supported security analyzers.

    To get the security analyzers:
@@ -100,15 +97,16 @@ async def get_security_analyzers():
    ```

    Returns:
-        list: A sorted list of security analyzer names.
+        list[str]: A sorted list of security analyzer names.
    """
    return sorted(SecurityAnalyzers.keys())


-@app.get('/config')
-async def get_config():
-    """
-    Get current config
-    """
+@app.get('/config', response_model=dict[str, Any])
+async def get_config() -> dict[str, Any]:
+    """Get current config.

+    Returns:
+        dict[str, Any]: The current server configuration.
+    """
    return server_config.get_config()
@@ -2,6 +2,7 @@ from fastapi import (
    APIRouter,
    HTTPException,
    Request,
+    Response,
    status,
 )

@@ -9,7 +10,7 @@ app = APIRouter(prefix='/api/conversations/{conversation_id}')


@app.route('/security/{path:path}', methods=['GET', 'POST', 'PUT', 'DELETE'])
-async def security_api(request: Request):
+async def security_api(request: Request) -> Response:
    """Catch-all route for security analyzer API requests.

    Each request is handled directly to the security analyzer.
@@ -18,7 +19,7 @@ async def security_api(request: Request):
        request (Request): The incoming FastAPI request object.

    Returns:
-        Any: The response from the security analyzer.
+        Response: The response from the security analyzer.

    Raises:
        HTTPException: If the security analyzer is not initialized.
@@ -11,8 +11,8 @@ from openhands.server.shared import SettingsStoreImpl, config
 app = APIRouter(prefix='/api')


-@app.get('/settings')
-async def load_settings(request: Request) -> GETSettingsModel | None:
+@app.get('/settings', response_model=GETSettingsModel)
+async def load_settings(request: Request) -> GETSettingsModel | JSONResponse:
    try:
        user_id = get_user_id(request)
        settings_store = await SettingsStoreImpl.get_instance(config, user_id)
@@ -40,13 +40,12 @@ async def load_settings(request: Request) -> GETSettingsModel | None:
        )


-@app.post('/settings')
+@app.post('/settings', response_model=dict[str, str])
 async def store_settings(
    request: Request,
    settings: POSTSettingsModel,
 ) -> JSONResponse:
    # Check if token is valid
-
    if settings.github_token:
        try:
            # We check if the token is valid by getting the user
@@ -9,7 +9,7 @@ app = APIRouter(prefix='/api/conversations/{conversation_id}')


@app.get('/trajectory')
-async def get_trajectory(request: Request):
+async def get_trajectory(request: Request) -> JSONResponse:
    """Get trajectory.

    This function retrieves the current trajectory and returns it.
@@ -19,6 +19,7 @@ from openhands.runtime.base import Runtime
@dataclass
 class RuntimeInfo:
    available_hosts: dict[str, int]
+    additional_agent_instructions: str


@dataclass
@@ -56,7 +57,9 @@ class PromptManager:
        self.user_template: Template = self._load_template('user_prompt')
        self.additional_info_template: Template = self._load_template('additional_info')
        self.microagent_info_template: Template = self._load_template('microagent_info')
-        self.runtime_info = RuntimeInfo(available_hosts={})
+        self.runtime_info = RuntimeInfo(
+            available_hosts={}, additional_agent_instructions=''
+        )

        self.knowledge_microagents: dict[str, KnowledgeMicroAgent] = {}
        self.repo_microagents: dict[str, RepoMicroAgent] = {}
@@ -113,6 +116,9 @@ class PromptManager:

    def set_runtime_info(self, runtime: Runtime) -> None:
        self.runtime_info.available_hosts = runtime.web_hosts
+        self.runtime_info.additional_agent_instructions = (
+            runtime.additional_agent_instructions
+        )

    def set_repository_info(
        self,
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 2.0.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 2.0.0 and should not be changed by hand.

 [[package]]
 name = "aiohappyeyeballs"
@@ -8938,7 +8938,7 @@ files = [

 [package.dependencies]
 greenlet = [
-    {version = "!=0.4.17", optional = true, markers = "python_version < \"3.14\" and (platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\") or extra == \"asyncio\""},
+    {version = "!=0.4.17", markers = "python_version < \"3.14\" and (platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\")"},
    {version = "!=0.4.17", optional = true, markers = "python_version < \"3.14\" and (platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\") or extra == \"asyncio\""},
 ]
 typing-extensions = ">=4.6.0"
@@ -9109,14 +9109,14 @@ files = [

 [[package]]
 name = "swebench"
-version = "3.0.13"
+version = "3.0.15"
 description = "The official SWE-bench package - a benchmark for evaluating LMs on software engineering"
 optional = false
 python-versions = ">=3.8"
 groups = ["evaluation"]
 files = [
-    {file = "swebench-3.0.13-py3-none-any.whl", hash = "sha256:0949e0a7269fcebb287dd951d14c049bd8189c7740fc4878354dbec756531c0f"},
-    {file = "swebench-3.0.13.tar.gz", hash = "sha256:d1cce406d0674cb1f3ca7da90089644d1ded3649c98f239a5a7ef4829d2f7c58"},
+    {file = "swebench-3.0.15-py3-none-any.whl", hash = "sha256:dd694356f9c155a55d3d2e113fe58446f7385eea0574230af5e2504426f8b85b"},
+    {file = "swebench-3.0.15.tar.gz", hash = "sha256:24e734fbcce34082665a25719075e6899382b7135103dd8c6cc09a6e23789101"},
 ]

 [package.dependencies]
@@ -9139,6 +9139,39 @@ unidiff = "*"
 inference = ["anthropic", "flash_attn", "jedi", "openai", "peft", "protobuf", "sentencepiece", "tiktoken", "torch", "transformers", "triton"]
 test = ["pytest", "pytest-cov"]

+[[package]]
+name = "swegym"
+version = "2.0.13"
+description = "Fork of SWE-bench package - a benchmark for evaluating LMs on software engineering"
+optional = false
+python-versions = ">=3.8"
+groups = ["evaluation"]
+files = []
+develop = false
+
+[package.dependencies]
+beautifulsoup4 = "*"
+chardet = "*"
+datasets = "*"
+docker = "*"
+ghapi = "*"
+GitPython = "*"
+pre-commit = "*"
+python-dotenv = "*"
+requests = "*"
+rich = "*"
+tqdm = "*"
+unidiff = "*"
+
+[package.extras]
+inference = ["anthropic", "flash_attn", "jedi", "openai", "peft", "protobuf", "sentencepiece", "tenacity", "tiktoken", "torch", "transformers", "triton"]
+
+[package.source]
+type = "git"
+url = "https://github.com/SWE-Gym/SWE-Bench-Package.git"
+reference = "HEAD"
+resolved_reference = "16dd480cce9b27bf111a362d280881c6def5d2a7"
+
 [[package]]
 name = "sympy"
 version = "1.13.1"
@@ -10855,4 +10888,4 @@ testing = ["coverage[toml]", "zope.event", "zope.testing"]
 [metadata]
 lock-version = "2.1"
 python-versions = "^3.12"
-content-hash = "83da0b681253a79417c9842862cdd102c1ab6e8770d9dd9e0c42bc7994be2cd0"
+content-hash = "c3f32c54606e5f313d9a909625f77cc3d575bf951e986633bcecd94520f36450"
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "openhands-ai"
-version = "0.27.0"
+version = "0.28.0"
 description = "OpenHands: Code Less, Make More"
 authors = ["OpenHands"]
 license = "MIT"
@@ -110,7 +110,6 @@ reportlab = "*"
 [tool.coverage.run]
 concurrency = ["gevent"]

-
 [tool.poetry.group.runtime.dependencies]
 jupyterlab = "*"
 notebook = "*"
@@ -139,13 +138,13 @@ ignore = ["D1"]
 [tool.ruff.lint.pydocstyle]
 convention = "google"

-
 [tool.poetry.group.evaluation.dependencies]
 streamlit = "*"
 whatthepatch = "*"
 retry = "*"
 evaluate = "*"
 swebench = "^3.0.8"
+swegym = { git = "https://github.com/SWE-Gym/SWE-Bench-Package.git" }
 commit0 = "*"
 func_timeout = "*"
 sympy = "*"
@@ -7,6 +7,7 @@ import pytest
 from openhands.controller.state.state import State
 from openhands.core.config.condenser_config import (
    AmortizedForgettingCondenserConfig,
+    BrowserOutputCondenserConfig,
    LLMAttentionCondenserConfig,
    LLMSummarizingCondenserConfig,
    NoOpCondenserConfig,
@@ -15,6 +16,7 @@ from openhands.core.config.condenser_config import (
 )
 from openhands.core.config.llm_config import LLMConfig
 from openhands.events.event import Event, EventSource
+from openhands.events.observation import BrowserOutputObservation
 from openhands.events.observation.agent import AgentCondensationObservation
 from openhands.events.observation.observation import Observation
 from openhands.llm import LLM
@@ -22,6 +24,7 @@ from openhands.memory.condenser import Condenser
 from openhands.memory.condenser.condenser import RollingCondenser
 from openhands.memory.condenser.impl import (
    AmortizedForgettingCondenser,
+    BrowserOutputCondenser,
    ImportantEventSelection,
    LLMAttentionCondenser,
    LLMSummarizingCondenser,
@@ -154,6 +157,46 @@ def test_observation_masking_condenser_respects_attention_window(mock_state):
            assert event == condensed_event


+def test_browser_output_condenser_from_config():
+    """Test that BrowserOutputCondenser objects can be made from config."""
+    attention_window = 5
+    config = BrowserOutputCondenserConfig(attention_window=attention_window)
+    condenser = Condenser.from_config(config)
+
+    assert isinstance(condenser, BrowserOutputCondenser)
+    assert condenser.attention_window == attention_window
+
+
+def test_browser_output_condenser_respects_attention_window(mock_state):
+    """Test that BrowserOutputCondenser only masks events outside the attention window."""
+    attention_window = 3
+    condenser = BrowserOutputCondenser(attention_window=attention_window)
+
+    events = [
+        BrowserOutputObservation('Observation 1', url='', trigger_by_action=''),
+        BrowserOutputObservation('Observation 2', url='', trigger_by_action=''),
+        create_test_event('Event 3'),
+        create_test_event('Event 4'),
+        BrowserOutputObservation('Observation 3', url='', trigger_by_action=''),
+        BrowserOutputObservation('Observation 4', url='', trigger_by_action=''),
+    ]
+
+    mock_state.history = events
+    result = condenser.condensed_history(mock_state)
+
+    assert len(result) == len(events)
+    cnt = 4
+    for event, condensed_event in zip(events, result):
+        if isinstance(event, BrowserOutputObservation):
+            if cnt > attention_window:
+                assert 'Content Omitted' in str(condensed_event)
+            else:
+                assert event == condensed_event
+            cnt -= 1
+        else:
+            assert event == condensed_event
+
+
 def test_recent_events_condenser_from_config():
    """Test that RecentEventsCondenser objects can be made from config."""
    max_events = 5
Author	SHA1	Message	Date
Xingyao Wang	3f866b4e3a	chore: temporary fix to get sonnet 3.7 working again (#7140 )	2025-03-06 15:32:27 -05:00
mamoodi	36548606d9	Release 0.28.0	2025-03-06 09:24:58 -05:00
Nan Jiang	ec087993f1	rename commit0_bench to commit0 (#7124 )	2025-03-06 02:55:39 +00:00
chuckbutkus	f12e9e94f7	Offline session login (#7086 ) Changes to allow the resolver to access keycloak and retrieve an offline token.	2025-03-05 21:58:34 +00:00
Xingyao Wang	9f720a9d69	[eval] SWE-Gym Integration (#6651 ) Co-authored-by: Robert Brennan <accounts@rbren.io> Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: Engel Nyst <enyst@users.noreply.github.com> Co-authored-by: Graham Neubig <neubig@gmail.com>	2025-03-05 20:15:02 +00:00
Xingyao Wang	bbf40c6576	docs: cleanup and update SWE-Bench documentation; and remove the support of non-instance-level image (#7118 ) Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>	2025-03-06 03:18:40 +08:00
Ivan Dagelic	1ffee80dcb	feat: prompt instructions for runtimes (#6943 ) Signed-off-by: Ivan Dagelic <dagelic.ivan@gmail.com>	2025-03-05 17:45:55 +01:00
dependabot[bot]	43bc944ac4	chore(deps-dev): bump typescript from 5.7.3 to 5.8.2 in /docs in the version-all group (#7116 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-03-05 15:09:49 +00:00
Xingyao Wang	4be33a079b	Update SWE-Bench README.md about RemoteRuntime (#7108 )	2025-03-05 23:00:54 +08:00
Aditya Bharat Soni	c76a659cde	Condenser for Browser Output Observations (#6578 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Rick van Hattem <wolph@wol.ph> Co-authored-by: Engel Nyst <enyst@users.noreply.github.com> Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk> Co-authored-by: sp.wack <83104063+amanape@users.noreply.github.com> Co-authored-by: Xingyao Wang <xingyao@all-hands.dev> Co-authored-by: tofarr <tofarr@gmail.com> Co-authored-by: Xingyao Wang <xingyao6@illinois.edu> Co-authored-by: Rohit Malhotra <rohitvinodmalhotra@gmail.com> Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akim Tsvigun <36672861+Aktsvigun@users.noreply.github.com> Co-authored-by: Akim Tsvigun <aktsvigun@nebius.com> Co-authored-by: mamoodi <mamoodiha@gmail.com> Co-authored-by: OpenHands <opendevin@all-hands.dev> Co-authored-by: Calvin Smith <email@cjsmith.io> Co-authored-by: Calvin Smith <calvin@all-hands.dev> Co-authored-by: Graham Neubig <neubig@gmail.com> Co-authored-by: Peter Dave Hello <hsu@peterdavehello.org> Co-authored-by: Ray Myers <ray.myers@gmail.com>	2025-03-04 16:28:33 -05:00
dependabot[bot]	0f68a18cbb	chore(deps): bump docker/setup-qemu-action from 3.4.0 to 3.6.0 (#7075 ) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2025-03-04 20:57:14 +00:00
Robert Brennan	c9ebabd82d	Add contact link to runtime settings label (#6880 ) Co-authored-by: openhands <openhands@all-hands.dev>	2025-03-05 00:49:53 +04:00
mamoodi	ad932e45e8	Checkout HEAD instead of Merge Commit for builds (#7085 )	2025-03-04 15:32:59 -05:00
sp.wack	3278caf3c2	Always enable GET /settings (#7101 )	2025-03-04 14:54:26 -05:00
He Du	896d7b8b96	Openhands fix issue 7091 (#7092 ) Co-authored-by: 杜贺 <duhe@duhedeMacBook-Pro-2.local>	2025-03-04 18:39:28 +01:00
Ryan H. Tran	cb61282c39	Improve error detection for `read` and `edit` observations (#7090 )	2025-03-04 15:05:15 +01:00
Graham Neubig	7a235ce6ff	Fix/mypy routes (#6900 ) Co-authored-by: openhands <openhands@all-hands.dev>	2025-03-04 03:43:09 +00:00