* [eval] increase timeout for swebench eval init/complete
* allow CmdRunAction to optionally block when .timeout is setted
* fix unit test for serialization
* fix unit tests for security analyzer
* fix integration tests
* add more timeout
* Reduce runtime tests duration by running them across CPUs
* fix hardcoded image name
* test two cpus
* Test folder change
* Up the CPU to 4 again to test
* Change to 3 CPUs
* Down to 2
* Add param to remove all openhands containers
* Add comment
* Add reruns just in case
* Fix ordering of if
* Add Handling of Cache Prompt When Formatting Messages
* Fix Value for Cache Control
* Fix Value for Cache Control
* Update openhands/core/message.py
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* Fix lint error
* Serialize Messages if Propt Caching Is Enabled
* Remove formatting message change
---------
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
Co-authored-by: tobitege <10787084+tobitege@users.noreply.github.com>
* dummy test change
* regen yml: 1st install python 3.11, then poetry
* fix caching for poetry; old entry for python was rather useless
* fix steps order (cache before poetry)
* add poetry caching to ghcr_runtime; fix fork conditions
* ghcr_runtime: more caching actions; condition fixes
* fix interim action error (order of steps)
* cache@v4 instead of v3
* fixed interim typo for 2 fork conditions
* runtime/test_env_vars: compacted multiple tests into one to reduce time
* ugh if fork condition changes again
* (feat) making prompt caching optional instead of enabled default
At present, only the Claude models support prompt caching as a experimental feature, therefore, this feature should be implemented as an optional setting rather than being enabled by default.
Signed-off-by: Yi Lin <teroincn@gmail.com>
* handle the conflict
* fix unittest mock return value
* fix lint error in whitespace
---------
Signed-off-by: Yi Lin <teroincn@gmail.com>
* feat: add SWE-bench fullset support
* fix instance image list
* update eval script and documentation
* increase timeout for remote runtime
* add push script
* handle the case when ret push is an generator
* update pbar
* set SWE-Bench default to run SWE-Bench lite
* add script to cleanup remote runtime
* fix the cases when tag is too long
* update README
* update readme for cleanup
* rename od to oh
* Update evaluation/swe_bench/README.md
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/README.md
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* gets API key and Runtime from env var
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
* CodeActAgent: fix message prep if prompt caching is not supported
* fix python version in regen tests workflow
* fix in conftest "mock_completion" method
* add disable_vision to LLMConfig; revert change in message parsing in llm.py
* format messages in several files for completion
* refactored message(s) formatting (llm.py); added vision_is_active()
* fix a unit test
* regenerate: added LOG_TO_FILE and FORCE_REGENERATE env flags
* try to fix path to logs folder in workflow
* llm: prevent index error
* try FORCE_USE_LLM in regenerate
* tweaks everywhere...
* fix 2 random unit test errors :(
* added FORCE_REGENERATE_TESTS=true to regenerate CLI
* fix test_lint_file_fail_typescript again
* double-quotes for env vars in workflow; llm logger set to debug
* fix typo in regenerate
* regenerate iterations now 20; applied iteration counter fix by Li
* regenerate: pass FORCE_REGENERATE flag into env
* fixes for int tests. several mock files updated.
* browsing_agent: fix response_parser.py adding ) to empty response
* test_browse_internet: fix skipif and revert obsolete mock files
* regenerate: fi bracketing for http server start/kill conditions
* disable test_browse_internet for CodeAct*Agents; mock files updated after merge
* missed to include more mock files earlier
* reverts after review feedback from Li
* forgot one
* browsing agent test, partial fixes and updated mock files
* test_browse_internet works in my WSL now!
* adapt unit test test_prompt_caching.py
* add DEBUG to regenerate workflow command
* convert regenerate workflow params to inputs
* more integration test mock files updated
* more files
* test_prompt_caching: restored test_prompt_caching_headers purpose
* file_ops: fix potential exception, like "cross device copy"; fixed mock files accordingly
* reverts/changes wrt feedback from xingyao
* updated docs and config template
* code cleanup wrt review feedback
* improve file editing prompts and unit test
converted most raise calls to a _output_error call in file_ops.py
* tweaks in test_agent_skill.py wrt to SEP separator
* tweaked the separator
* remove server runtime remnants and TEST_RUNTIME references
* restore use of TEST_RUNTIME args and variables
* fix integration tests
* added hint to properly escape docstrings
* revert latest prompt change
---------
Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
* change runtime to be synchronous
* fix test runtime with the new interface
* fix arg
* fix eval
* fix missing config attribute
* fix plugins
* fix on_event by revert it back to async
* update upload_file endpoint
* fix argument to upload file
* remove unncessary async for eval;
fix evaluation run in parallel
* use asyncio to run controller for eval
* revert file upload
* truncate eval test result output
* feat: refactor building logic into runtime builder
* return image name
* fix testcases
* use runtime builder for eventstream runtime
* have runtime builder return str
* add api_key to sandbox config
* draft remote runtime
* remove extra if clause
* initialize runtime based on box class
* add build logic
* use base64 for file upload
* get runtime image prefix from API
* replace ___ with _s_ to make it a valid image name
* use /build to start build and /build_status to check the build progress
* update logging
* fix exit code
* always use port
* add remote runtime
* rename runtime
* fix tests import
* make dir first if work_dir does not exists;
* update debug print to remote runtime
* fix exit close_sync
* update logging
* add retry for stop
* use all box class for test keep prompt
* fix test browsing
* add retry stop
* merge init commands to save startup time
* fix await
* remove sandbox url
* support execute through specific runtime url
* fix file ops
* simplify close
* factor out runtime retry code
* fix exception handling
* fix content type error (e.g., bad gateway when runtime is not ready)
* add retry for wait until alive;
add retry for check image exists
* Revert "add retry for wait until alive;"
This reverts commit dd013cd268.
* retry when wait until alive
* clean up msg
* directly save sdist to temp dir for _put_source_code_to_dir
* support running testcases in parallel
* tweak logging;
try to close session
* try to close session even on exception
* update poetry lock
* support remote to run integration tests
* add warning for workspace base on remote runtime
* set default runtime api
* remove server runtime
* update poetry lock
* support running swe-bench (n=1) eval on remoteruntime
* add a timeout of 30 min
* add todo for docker namespace
* update poetry loc
* linter and test_aider_linter extensions for eslint
* linter tweaks
* try enabling verbose output in linter test
* one more option for linter test
* try conftest.py for tests/unit folder
* enable verbose mode in workflow; remove conftest.py again
* debug print statements of linter results
* skip some tests if eslint is not installed at all
* more tweaks
* final test skip setups
* code quality revisions
* fix test again
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Add prompt caching
* remove anthropic-version from extra_headers
* change supports_prompt_caching method to attribute
* change caching strat and log cache statistics
* add reminder as a new message to fix caching
* fix unit test
* append reminder to the end of the last message content
* move token logs to post completion function
* fix unit test failure
* fix reminder and prompt caching
* unit tests for prompt caching
* add test
* clean up tests
* separate reminder, use latest two messages
* fix tests
---------
Co-authored-by: tobitege <10787084+tobitege@users.noreply.github.com>
Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* add eval_ids arg to run specific instance id's; fix/extend README
* fix description in parser for --eval-ids
* fix test_arg_parser.py to account for added arg
* fix typo in README to say "summarize" instead of "summarise" for script
* fix conftest.py option (#3573)
* try to fix fixture base_container_image in runtime conftest
* fix integration test mock files due to #3548
* fix test_ipython.py integration test
* Fix pypi build
The package on pypi only included opendevin/* (the poetry default). It also needs to include agenthub/*
* Bumped version so people will actually get it!
* Fix package definition
* Updated poetry lock file
* Update package name to openhands-ai
* Add py.typed to indicate that OpenHands has type annotations
* Replace package name with openhands_ai
* Fix tests to reflect new name
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
* try to fix pip unavailable
* update test case for pip
* force rebuild in CI
* remove extra symlink
* fix newline
* added semi-colon to line 31
* Dockerfile.j2: activate env at the end
* Revert "Dockerfile.j2: activate env at the end"
This reverts commit cf2f565102.
* cleanup Dockerfile
* switch default python image
* remove image agnostic (no longer used)
* fix tests
* simplify integration tests default image
* add nodejs specific runtime tests
* update tests and workflows
* switch to nikolaik/python-nodejs:python3.11-nodejs22
* update build sh to output image name correctly
* increase custom images to test
* fix test
* fix test
* fix double quote
* try fixing ci
* update ghcr workflow
* fix artifact name
* try to fix ghcr again
* fix workflow
* save built image to correct dir
* remove extra -docker-image
* make last tag to be human readable image tag
* fix hyphen to underscore
* run test runtime on all tags
* revert app build
* separate ghcr workflow
* update dockerfile for eval
* fix tag for test run
* try fix tag
* try fix tag via matrix output
* try workflow again
* update comments
* try fixing test matrix
* fix artifact name
* try fix tag again
* Revert "try fix tag again"
This reverts commit b369badd8c.
* tweak filename
* try different path
* fix filepath
* try fix tag artifact path again
* save json instead of line
* update matrix
* print all tags in workflow
* support only streaming diff logs from the runtime client
* remove strip from log line to fix indentation
* get py interpreter for jupyter
* rstrip to remove newline on the rightside for logging
* fix blocking issue for stream logs
* set python interpreter path in bash ps1
* update testcase for jupyter py interpreter path
* remove accidentally added changes
* remove accidentally added changes
* only print dockerfile when debug
* add docs
* remove extra tests that weren't supposed to be in this pr
* add back missing test
* revert
* make LogBuffer synchronous to fix hang in integration tests
* fix integration tests
* Update opendevin/runtime/client/client.py
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* fix test case
* fix integration tests
* change deque to list
* update integration tests
* rename test runtime
* fix docs
* rename opendevin to openhands in tests
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: tobitege <10787084+tobitege@users.noreply.github.com>
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* tweaks to linter.py to prep for typescript linting (not implemented yet)
* fix 2 linter unit tests
* simpler basic_lint output; updated unit test
* fix default gpt-4o model name in aider default config
* linter.py: use tsc (typescript compiler) for linting; added more tests
* make typescript linting be more forgiving
* use npx instead of npm to install typescript in Dockerfile.j2
* Fix merge mistake
* removed npx call from Dockerfile.j2
* fix run_cmd to use code parameter; replace regex in test
* fix test_lint_file_fail_typescript to ignore leading path characters
* added TODO comment to extract_error_line_from
* fixed bug in ts_lint with wrong line number parsing
* move test_runtime to ghcr_test_runtime; adapt workflows; fix runtime AttributeError
* split test_runtime.py into multiple tests in new tests/runtime folder
* moved common fixtures to tests/runtime/conftest.py
* try to fix pip unavailable
* update test case for pip
* force rebuild in CI
* remove extra symlink
* fix newline
* added semi-colon to line 31
* Dockerfile.j2: activate env at the end
* Revert "Dockerfile.j2: activate env at the end"
This reverts commit cf2f565102.
* cleanup Dockerfile
* switch default python image
* remove image agnostic (no longer used)
* fix tests
* simplify integration tests default image
* add nodejs specific runtime tests
* update tests and workflows
* switch to nikolaik/python-nodejs:python3.11-nodejs22
* update build sh to output image name correctly
* increase custom images to test
* fix test
* fix test
* fix double quote
* try fixing ci
* update ghcr workflow
* fix artifact name
* try to fix ghcr again
* fix workflow
* save built image to correct dir
* remove extra -docker-image
* make last tag to be human readable image tag
* fix hyphen to underscore
* run test runtime on all tags
* revert app build
* separate ghcr workflow
* update dockerfile for eval
* fix tag for test run
* try fix tag
* try fix tag via matrix output
* try workflow again
* update comments
* try fixing test matrix
* fix artifact name
* try fix tag again
* Revert "try fix tag again"
This reverts commit b369badd8c.
* tweak filename
* try different path
* fix filepath
* try fix tag artifact path again
* save json instead of line
* update matrix
* print all tags in workflow
* fix DOCKER_IMAGE to avoid ghcr.io/opendevin/ghcr.io/opendevin/od_runtime
* fix test matrix to only load unique test image tags
* try fix matrix again!!!!!
* add all runtime tests passed
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: tobitege <10787084+tobitege@users.noreply.github.com>
* refactor agentskills to prepare for agentless
* fix import
* fix typo
* fix imports
* fix globals
* fix import
* fix import
* disable log to file to avoid auto-created log file w/ permission issue when import od in runtime
* import agentskills from OD instead from itself directly
* add back pythonpath
* remove chown since there's no log/folder
* fix potential flake8 miss checking
* Add unit test for edit_file_by_replace function with problematic file
* Add unit test for edit_file_by_replace function with problematic file
* Add unit test for edit_file_by_replace function with problematic file
* Add unit test for edit_file_by_replace function with problematic file
* Add unit test for edit_file function with problematic file
* Add unit test for edit_file function with problematic file
* Add unit test for edit_file function with problematic file
* Add unit test for edit_file function with problematic file
* Add unit test for edit_file function with problematic file
* Update opendevin/runtime/plugins/agent_skills/agentskills.py
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
* add test intention description
* fix potential flake8 miss checking
* fix potential flake8 miss checking
* fix potential flake8 miss checking
* fix potential flake8 miss checking
* fix potential flake8 miss checking
---------
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: tobitege <tobitege@gmx.de>
