* [eval] increase timeout for swebench eval init/complete
* allow CmdRunAction to optionally block when .timeout is setted
* fix unit test for serialization
* fix unit tests for security analyzer
* fix integration tests
* add more timeout
* only check P2P when instances are non-empty;
convert P2P and F2P columns to string instead of list
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
* [eval] increase timeout for swebench eval init/complete
* allow CmdRunAction to optionally block when .timeout is setted
* fix unit test for serialization
* fix unit tests for security analyzer
* fix integration tests
* add more timeout
* feat: add SWE-bench fullset support
* fix instance image list
* update eval script and documentation
* increase timeout for remote runtime
* add push script
* handle the case when ret push is an generator
* update pbar
* set SWE-Bench default to run SWE-Bench lite
* add script to cleanup remote runtime
* fix the cases when tag is too long
* update README
* update readme for cleanup
* rename od to oh
* Update evaluation/swe_bench/README.md
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/README.md
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Update evaluation/swe_bench/scripts/cleanup_remote_runtime.sh
Co-authored-by: Graham Neubig <neubig@gmail.com>
* gets API key and Runtime from env var
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
* feat: add SWE-bench fullset support
* fix instance image list
* update eval script and documentation
* increase timeout for remote runtime
* add push script
* handle the case when ret push is an generator
* update pbar
* set SWE-Bench default to run SWE-Bench lite
* feat: add SWE-bench fullset support
* fix instance image list
* update eval script and documentation
* add push script
* handle the case when ret push is an generator
* update pbar
* change runtime to be synchronous
* fix test runtime with the new interface
* fix arg
* fix eval
* fix missing config attribute
* fix plugins
* fix on_event by revert it back to async
* update upload_file endpoint
* fix argument to upload file
* remove unncessary async for eval;
fix evaluation run in parallel
* use asyncio to run controller for eval
* revert file upload
* truncate eval test result output
* feat: refactor building logic into runtime builder
* return image name
* fix testcases
* use runtime builder for eventstream runtime
* have runtime builder return str
* add api_key to sandbox config
* draft remote runtime
* remove extra if clause
* initialize runtime based on box class
* add build logic
* use base64 for file upload
* get runtime image prefix from API
* replace ___ with _s_ to make it a valid image name
* use /build to start build and /build_status to check the build progress
* update logging
* fix exit code
* always use port
* add remote runtime
* rename runtime
* fix tests import
* make dir first if work_dir does not exists;
* update debug print to remote runtime
* fix exit close_sync
* update logging
* add retry for stop
* use all box class for test keep prompt
* fix test browsing
* add retry stop
* merge init commands to save startup time
* fix await
* remove sandbox url
* support execute through specific runtime url
* fix file ops
* simplify close
* factor out runtime retry code
* fix exception handling
* fix content type error (e.g., bad gateway when runtime is not ready)
* add retry for wait until alive;
add retry for check image exists
* Revert "add retry for wait until alive;"
This reverts commit dd013cd268.
* retry when wait until alive
* clean up msg
* directly save sdist to temp dir for _put_source_code_to_dir
* support running testcases in parallel
* tweak logging;
try to close session
* try to close session even on exception
* update poetry lock
* support remote to run integration tests
* add warning for workspace base on remote runtime
* set default runtime api
* remove server runtime
* update poetry lock
* support running swe-bench (n=1) eval on remoteruntime
* add a timeout of 30 min
* add todo for docker namespace
* update poetry loc
* aider-bench: add visualization to summarize script and readme
* added example cost and actions histogram images for readme
* moved dependencies to evaluation section
* added optional START_ID env flag to resume from that instance id
* prepare_dataset: fix comparisons by using instance id's as int
* aider bench complete_runtime: close runtime to close container
* added matrix display of instance id for logging
* fix typo in summarize_results.py saying summarise_results
* changed start_id to skip_num to skip rows from dataset (start_id wasn't supportable)
* doc changes about huggingface spaces to temporarily point back to OD
* add eval_ids arg to run specific instance id's; fix/extend README
* fix description in parser for --eval-ids
* fix test_arg_parser.py to account for added arg
* fix typo in README to say "summarize" instead of "summarise" for script
* renaming more opendevin occurences
* remove DOCKER_IMAGE variable from Makefile
* Revert rename in evaluation/swe_bench/run_infer.py
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
---------
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
* try to fix pip unavailable
* update test case for pip
* force rebuild in CI
* remove extra symlink
* fix newline
* added semi-colon to line 31
* Dockerfile.j2: activate env at the end
* Revert "Dockerfile.j2: activate env at the end"
This reverts commit cf2f565102.
* cleanup Dockerfile
* switch default python image
* remove image agnostic (no longer used)
* fix tests
* simplify integration tests default image
* add nodejs specific runtime tests
* update tests and workflows
* switch to nikolaik/python-nodejs:python3.11-nodejs22
* update build sh to output image name correctly
* increase custom images to test
* fix test
* fix test
* fix double quote
* try fixing ci
* update ghcr workflow
* fix artifact name
* try to fix ghcr again
* fix workflow
* save built image to correct dir
* remove extra -docker-image
* make last tag to be human readable image tag
* fix hyphen to underscore
* run test runtime on all tags
* revert app build
* separate ghcr workflow
* update dockerfile for eval
* fix tag for test run
* try fix tag
* try fix tag via matrix output
* try workflow again
* update comments
* try fixing test matrix
* fix artifact name
* try fix tag again
* Revert "try fix tag again"
This reverts commit b369badd8c.
* tweak filename
* try different path
* fix filepath
* try fix tag artifact path again
* save json instead of line
* update matrix
* print all tags in workflow
* fix DOCKER_IMAGE to avoid ghcr.io/opendevin/ghcr.io/opendevin/od_runtime
* fix test matrix to only load unique test image tags
* try fix matrix again!!!!!
* add all runtime tests passed
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: tobitege <10787084+tobitege@users.noreply.github.com>
* try to fix pip unavailable
* update test case for pip
* force rebuild in CI
* remove extra symlink
* fix newline
* added semi-colon to line 31
* Dockerfile.j2: activate env at the end
* Revert "Dockerfile.j2: activate env at the end"
This reverts commit cf2f565102.
* cleanup Dockerfile
* switch default python image
* remove image agnostic (no longer used)
* fix tests
* switch to nikolaik/python-nodejs:python3.11-nodejs22
* fix test
* fix test
* revert docker
* update template
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: Graham Neubig <neubig@gmail.com>
* move multi-line bash tests to test_runtime;
support multi-line bash for esruntime;
* add testcase to handle PS2 prompt
* use bashlex for bash parsing to handle multi-line commands;
add testcases for multi-line commands
* revert ghcr runtime change
* Apply stash
* fix run as other user;
make test async;
* fix test runtime for run as od
* add run-as-devin to all the runtime tests
* handle the case when username is root
* move all run-as-devin tests from sandbox;
only tests a few cases on different user to save time;
* move over multi-line echo related tests to test_runtime
* fix user-specific jupyter by fixing the pypoetry virtualenv folder
* make plugin's init async;
chdir at initialization of jupyter plugin;
move ipy simple testcase to test runtime;
* support agentskills import in
move tests for jupyter pwd tests;
overload `add_env_vars` for EventStreamRuntime to update env var also in Jupyter;
make agentskills read env var lazily, in case env var is updated;
* fix ServerRuntime agentskills issue
* move agnostic image test to test_runtime
* merge runtime tests in CI
* fix enable auto lint as env var
* update warning message
* update warning message
* test for different container images
* change parsing output as debug
* add exception handling for update_pwd_decorator
* fix unit test indentation
* add plugins as default input to Runtime class;
remove init_sandbox_plugins;
implement add_env_var (include jupyter) in the base class;
* fix server runtime auto lint
* Revert "add exception handling for update_pwd_decorator"
This reverts commit 2b668b1506.
* tries to print debugging info for agentskills
* explictly setting uid (try fix permission issue)
* Revert "tries to print debugging info for agentskills"
This reverts commit 8be4c86756.
* set sandbox user id during testing to hopefully fix the permission issue
* add browser tools for server runtime
* try to debug for old pwd
* update debug cmd
* only test agnostic runtime when TEST_RUNTIME is Server
* fix temp dir mkdir
* load TEST_RUNTIME at the beginning
* remove ipython tests
* only log to file when DEBUG
* default logging to project root
* temporarily remove log to file
* fix LLM logger dir
* fix logger
* make set pwd an optional aux action
* fix prev pwd
* fix infinity recursion
* simplify
* do not import the whole od library to avoid logger folder by jupyter
* fix browsing
* increase timeout
* attempt to fix agentskills yet again
* clean up in testcases, since CI maybe run as non-root
* add _cause attribute for event.id
* remove parent
* add a bunch of debugging statement again for CI :(
* fix temp_dir fixture
* change all temp dir to follow pytest's tmp_path_factory
* remove extra bracket
* clean up error printing a bit
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* add typing for tmp dir fixture
* clear the directory before running the test to avoid weird CI temp dir
* remove agnostic test case for server runtime
* Revert "remove agnostic test case for server runtime"
This reverts commit 30e2181c3f.
* disable agnostic tests in CI
* fix test
* make sure plugin arg is not passed when no plugin is specified;
remove redundant on_event function;
* move mock prompt
* rename runtime
* remove extra logging
* refactor run_controller's interface;
support multiple runtime for integration test;
filter out hostname for prompt
* uncomment other tests
* pass the right runtime to controller
* log runtime when start
* uncomment tests
* improve symbol filters
* add intergration test prompts that seemd ok
* add integration test workflow
* add python3 to default ubuntu image
* symlink python and fix permission to jupyter pip
* add retry for jupyter execute server
* fix jupyter pip install;
add post-process for jupyter pip install;
simplify init by add agent_skills path to PYTHONPATH;
add testcase to tests jupyter pip install;
* fix bug
* use ubuntu:22.04 for eventstream integration tests
* add todo
* update testcase
* remove redundant code
* fix unit test
* reduce dependency for runtime
* try making llama-index an optional dependency that's not installed by default
* remove pip install since it seemd not needed
* log ipython execution;
await write message since it returns a future
* update ipy testcase
* do not install llama-index in CI
* do not install llama-index in the app docker as well
* set sandbox container image in the integration test script
* log plugins & env var for runtime
* update conftest for sha256
* add git
* remove all non-alphanumeric chalracters
* add working ipy module tests!
* default to use host network
* remove is_async from browser to make thing a little more reliable;
retry loading browser when error;
* add sleep to wait a bit for http server
* kill http server before regenerate browsing tests
* fix browsing
* only set sandbox container image if undefined
* skip empty config value
* update evaluation to use the latest run_controller
* revert logger in execute_server to be compatible with server runtime
* revert logging level to fix jupyter
* set logger level
* revert the logging
* chmod for workspace to fix permission
* support getting timeout from action
* update test for server runtime
* try to fix file permission
* fix test_cmd_run_action_serialization_deserialization test (added timeout)
* poetry: pip 24.2, torch 2.2.2
* revert adding pip to pyproject.toml
* add build to dependencies in pyproject.toml
* forgot poetry lock --no-update
* fix a DelegatorAgent prompt_002.log (timeout)
* fix a DelegatorAgent prompt_003.log (timeout)
* couple more timeout attribs in prompt files
* some more prompt files
* prompts galore
* add clarification comment for timeout
* default timeout to config
* add assert
* update integraton tests for eventstream
* update integration tests
* fix timeout for action<->dict
* remove redundant on_event
* default to use instance image
* update run_controller interface
* add logging for copy
* refactor swe_bench for the new design
* fix action execution timeout
* updatelock
* remove build sandbox locally
* fix runtime
* use plain for-loop for single process
* remove extra print
* get swebench inference working
* print whole `test_result` dict
* got swebench patch post-process working
* update swe-bench evaluation readme
* refactor using shared reset_logger function
* move messy swebench prompt to a different file
* support the ability to specify whether to keep prompt
* support the ability to specify whether to keep prompt
* fix dockerfile
* fix import and remove unnecessary strip logic
* fix action serialization
* get agentbench running
* remove extra ls for agent bench
* fix agentbench metric
* factor out common documentation for eval
* update biocoder doc
* remove swe_env_box since it is no longer needed
* get biocoder working
* add func timeout for bird
* fix jupyter pwd with ~ as user name
* fix jupyter pwd with ~ as user name
* get bird working
* get browsing evaluation working
* make eda runnable
* fix id column
* fix eda run_infer
* unify eval output using a structured format;
make swebench coompatible with that format;
update client source code for every swebench run;
do not inject testcmd for swebench
* standardize existing benchs for the new eval output
* set update source code = true
* get gaia standardized
* fix gaia
* gorilla refactored but stuck at language.so to test
* refactor and make gpqa work
* refactor humanevalfix and get it working
* refactor logic reasoning and get it working
* refactor browser env so it works with eventstream runtime for eval
* add initial version of miniwob refactor
* fix browsergym environment
* get miniwob working!!
* allowing injecting additional dependency to OD runtime docker image
* allowing injecting additional dependency to OD runtime docker image
* support logic reasoning with pre-injected dependency
* get mint working
* update runtime build
* fix mint docker
* add test for keep_prompt;
add missing await close for some tests
* update integration tests for eventstream runtime
* fix integration tests for server runtime
* refactor ml bench and toolqa
* refactor webarena
* fix default factory
* Update run_infer.py
* add APIError to retry
* increase timeout for swebench
* make sure to hide api key when dump eval output
* update the behavior of put source code to put files instead of tarball
* add dishash to dependency
* sendintr when timeout
* fix dockerfile copy
* reduce timeout
* use dirhash to avoid repeat building for update source
* fix runtime_build testcase
* add dir_hash to docker build pipeline
* revert api error
* update poetry lock
* add retries for swebench run infer
* fix git patch
* update poetry lock
* adjust config order
* fix mount volumns
* enforce all eval to use "instance_id"
* remove file store from runtime
* make file_store public inside eventstream
* move the runtime logic inside `main` out
* support using async function for process_instance_fn
* refactor run_infer with the create_time
* fix file store
* Update evaluation/toolqa/utils.py
Co-authored-by: Graham Neubig <neubig@gmail.com>
* fix typo
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: super-dainiu <78588128+super-dainiu@users.noreply.github.com>
Co-authored-by: Graham Neubig <neubig@gmail.com>
* Remove global config from memory
* Remove runtime global config
* Remove from storage
* Remove global config
* Fix event stream tests
* Fix sandbox issue
* Change config
* Removed transferred tests
* Add swe env box
* Fixes on testing
* Fixed some tests
* Merge with stashed changes
* Fix typing
* Fix ipython test
* Revive function
* Make temp_dir fixture
* Remove test to avoid circular import
* fix eventstream filestore for test_runtime
* fix parse arg issue that cause integration test to fail
* support swebench pull from custom namespace
* add back simple tests for runtime
* move multi-line bash tests to test_runtime;
support multi-line bash for esruntime;
* add testcase to handle PS2 prompt
* use bashlex for bash parsing to handle multi-line commands;
add testcases for multi-line commands
* revert ghcr runtime change
* Apply stash
* fix run as other user;
make test async;
* fix test runtime for run as od
* add run-as-devin to all the runtime tests
* handle the case when username is root
* move all run-as-devin tests from sandbox;
only tests a few cases on different user to save time;
* move over multi-line echo related tests to test_runtime
* fix user-specific jupyter by fixing the pypoetry virtualenv folder
* make plugin's init async;
chdir at initialization of jupyter plugin;
move ipy simple testcase to test runtime;
* support agentskills import in
move tests for jupyter pwd tests;
overload `add_env_vars` for EventStreamRuntime to update env var also in Jupyter;
make agentskills read env var lazily, in case env var is updated;
* fix ServerRuntime agentskills issue
* move agnostic image test to test_runtime
* merge runtime tests in CI
* fix enable auto lint as env var
* update warning message
* update warning message
* test for different container images
* change parsing output as debug
* add exception handling for update_pwd_decorator
* fix unit test indentation
* add plugins as default input to Runtime class;
remove init_sandbox_plugins;
implement add_env_var (include jupyter) in the base class;
* fix server runtime auto lint
* Revert "add exception handling for update_pwd_decorator"
This reverts commit 2b668b1506.
* tries to print debugging info for agentskills
* explictly setting uid (try fix permission issue)
* Revert "tries to print debugging info for agentskills"
This reverts commit 8be4c86756.
* set sandbox user id during testing to hopefully fix the permission issue
* add browser tools for server runtime
* try to debug for old pwd
* update debug cmd
* only test agnostic runtime when TEST_RUNTIME is Server
* fix temp dir mkdir
* load TEST_RUNTIME at the beginning
* remove ipython tests
* only log to file when DEBUG
* default logging to project root
* temporarily remove log to file
* fix LLM logger dir
* fix logger
* make set pwd an optional aux action
* fix prev pwd
* fix infinity recursion
* simplify
* do not import the whole od library to avoid logger folder by jupyter
* fix browsing
* increase timeout
* attempt to fix agentskills yet again
* clean up in testcases, since CI maybe run as non-root
* add _cause attribute for event.id
* remove parent
* add a bunch of debugging statement again for CI :(
* fix temp_dir fixture
* change all temp dir to follow pytest's tmp_path_factory
* remove extra bracket
* clean up error printing a bit
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* add typing for tmp dir fixture
* clear the directory before running the test to avoid weird CI temp dir
* remove agnostic test case for server runtime
* Revert "remove agnostic test case for server runtime"
This reverts commit 30e2181c3f.
* disable agnostic tests in CI
* fix test
* make sure plugin arg is not passed when no plugin is specified;
remove redundant on_event function;
* move mock prompt
* rename runtime
* remove extra logging
* refactor run_controller's interface;
support multiple runtime for integration test;
filter out hostname for prompt
* uncomment other tests
* pass the right runtime to controller
* log runtime when start
* uncomment tests
* improve symbol filters
* add intergration test prompts that seemd ok
* add integration test workflow
* add python3 to default ubuntu image
* symlink python and fix permission to jupyter pip
* add retry for jupyter execute server
* fix jupyter pip install;
add post-process for jupyter pip install;
simplify init by add agent_skills path to PYTHONPATH;
add testcase to tests jupyter pip install;
* fix bug
* use ubuntu:22.04 for eventstream integration tests
* add todo
* update testcase
* remove redundant code
* fix unit test
* reduce dependency for runtime
* try making llama-index an optional dependency that's not installed by default
* remove pip install since it seemd not needed
* log ipython execution;
await write message since it returns a future
* update ipy testcase
* do not install llama-index in CI
* do not install llama-index in the app docker as well
* set sandbox container image in the integration test script
* log plugins & env var for runtime
* update conftest for sha256
* add git
* remove all non-alphanumeric chalracters
* add working ipy module tests!
* default to use host network
* remove is_async from browser to make thing a little more reliable;
retry loading browser when error;
* add sleep to wait a bit for http server
* kill http server before regenerate browsing tests
* fix browsing
* only set sandbox container image if undefined
* skip empty config value
* update evaluation to use the latest run_controller
* revert logger in execute_server to be compatible with server runtime
* revert logging level to fix jupyter
* set logger level
* revert the logging
* chmod for workspace to fix permission
* support getting timeout from action
* update test for server runtime
* try to fix file permission
* fix test_cmd_run_action_serialization_deserialization test (added timeout)
* poetry: pip 24.2, torch 2.2.2
* revert adding pip to pyproject.toml
* add build to dependencies in pyproject.toml
* forgot poetry lock --no-update
* fix a DelegatorAgent prompt_002.log (timeout)
* fix a DelegatorAgent prompt_003.log (timeout)
* couple more timeout attribs in prompt files
* some more prompt files
* prompts galore
* add clarification comment for timeout
* default timeout to config
* add assert
* update integraton tests for eventstream
* update integration tests
* fix timeout for action<->dict
* remove redundant on_event
* fix action execution timeout
* updatelock
---------
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: tobitege <tobitege@gmx.de>
* add copy to
* implement for ServerRuntime
* implement copyto for runtime (required by eval);
add tests for copy to
* fix exist file check
* unify copy_to_behavior and fix stuff
