* add draft dockerfile for build all
* add rsync for build
* add all-in-one docker
* update prepare scripts
* Update swe_env_box.py
* Add swe_entry.sh (buggy now)
* Parse the test command in swe_entry.sh
* Update README for instance eval in sandbox
* revert specialized config
* replace run_as_devin as an init arg
* set container & run_as_root via args
* update swe entry script
* update env
* remove mounting
* allow error after swe_entry
* update swe_env_box
* move file
* update gitignore
* get swe_env_box a working demo
* support faking user response & provide sandox ahead of time;
also return state for controller
* tweak main to support adding controller kwargs
* add module
* initialize plugin for provided sandbox
* add pip cache to plugin & fix jupyter kernel waiting
* better print Observation output
* add run infer scripts
* update readme
* add utility for getting diff patch
* use get_diff_patch in infer
* update readme
* support cost tracking for codeact
* add swe agent edit hack
* disable color in git diff
* fix git diff cmd
* fix state return
* support limit eval
* increase t imeout and export pip cache
* add eval limit config
* return state when hit turn limit
* save log to file; allow agent to give up
* run eval with max 50 turns
* add outputs to gitignore
* save swe_instance & instruction
* add uuid to swebench
* add streamlit dep
* fix save series
* fix the issue where session id might be duplicated
* allow setting temperature for llm (use 0 for eval)
* Get report from agent running log
* support evaluating task success right after inference.
* remove extra log
* comment out prompt for baseline
* add visualizer for eval
* use plaintext for instruction
* reduce timeout for all; only increase timeout for init
* reduce timeout for all; only increase timeout for init
* ignore sid for swe env
* close sandbox in each eval loop
* update visualizer instruction
* increase max chars
* add finish action to history too
* show test result in metrics
* add sidebars for visualizer
* also visualize swe_instance
* cleanup browser when agent controller finish runinng
* do not mount workspace for swe-eval to avoid accidentally overwrite files
* Revert "do not mount workspace for swe-eval to avoid accidentally overwrite files"
This reverts commit 8ef7739054.
* Revert "Revert "do not mount workspace for swe-eval to avoid accidentally overwrite files""
This reverts commit 016cfbb9f0.
* run jupyter command via copy to, instead of cp to mount
* only print mixin output when failed
* change ssh box logging
* add visualizer for pass rate
* add instance id to sandbox name
* only remove container we created
* use opendevin logger in main
* support multi-processing infer
* add back metadata, support keyboard interrupt
* remove container with startswith
* make pbar behave correctly
* update instruction w/ multi-processing
* show resolved rate by repo
* rename tmp dir name
* attempt to fix racing for copy to ssh_box
* fix script
* bump swe-bench-all version
* fix ipython with self-contained commands
* add jupyter demo to swe_env_box
* make resolved count two column
* increase height
* do not add glob to url params
* analyze obs length
* print instance id prior to removal handler
* add gold patch in visualizer
* fix interactive git by adding a git --no-pager as alias
* increase max_char to 10k to cover 98% of swe-bench obs cases
* allow parsing note
* prompt v2
* add iteration reminder
* adjust user response
* adjust order
* fix return eval
* fix typo
* add reminder before logging
* remove other resolve rate
* re adjust to new folder structure
* support adding eval note
* fix eval note path
* make sure first log of each instance is printed
* add eval note
* fix the display for visualizer
* tweak visualizer for better git patch reading
* exclude empty patch
* add retry mechanism for swe_env_box start
* fix ssh timeout issue
* add stat field for apply test patch success
* add visualization for fine-grained report
* attempt to support monologue agent by constraining it to single thread
* also log error msg when stopeed
* save error as well
* override WORKSPACE_MOUNT_PATH and WORKSPACE_BASE for monologue to work in mp
* add retry mechanism for sshbox
* remove retry for swe env box
* try to handle loop state stopped
* Add get report scripts
* Add script to convert agent output to swe-bench format
* Merge fine grained report for visualizer
* Update eval readme
* Update README.md
* Add CodeAct gpt4-1106 output and eval logs on swe-bench-lite
* Update the script to get model report
* Update get_model_report.sh
* Update get_agent_report.sh
* Update report merge script
* Add agent output conversion script
* Update swe_lite_env_setup.sh
* Add example swe-bench output files
* Update eval readme
* Remove redundant scripts
* set iteration count down to false by default
* fix: Issue where CodeAct agent was trying to log cost on local llm and throwing Undefined Model execption out of litellm (#1666)
* fix: Issue where CodeAct agent was trying to log cost on local llm and throwing Undefined Model execption out of litellm
* Review Feedback
* Missing None Check
* Review feedback and improved error handling
---------
Co-authored-by: Robert Brennan <accounts@rbren.io>
* fix prepare_swe_util scripts
* update builder images
* update setup script
* remove swe-bench build workflow
* update lock
* remove experiments since they are moved to hf
* remove visualizer (since it is moved to hf repo)
* simply jupyter execution via heredoc
* update ssh_box
* add initial docker readme
* add pkg-config as dependency
* add script for swe_bench all-in-one docker
* add rsync to builder
* rename var
* update commit
* update readme
* update lock
* support specify timeout for long running tasks
* fix path
* separate building of all deps and files
* support returning states at the end of controller
* remove return None
* support specify timeout for long running tasks
* add timeout for all existing sandbox impl
* fix swe_env_box for new codebase
* update llm config in config.py
* support pass sandbox in
* remove force set
* update eval script
* fix issue of overriding final state
* change default eval output to hf demo
* change default eval output to hf demo
* fix config
* only close it when it is NOT external sandbox
* add scripts
* tweak config
* only put in hostory when state has history attr
* fix agent controller on the case of run out interaction budget
* always assume state is always not none
* remove print of final state
* catch all exception when cannot compute completion cost
* Update README.md
* save source into json
* fix path
* update docker path
* return the final state on close
* merge AgentState with State
* fix integration test
* merge AgentState with State
* fix integration test
* add ChangeAgentStateAction to history in attempt to fix integration
* add back set agent state
* update tests
* update tests
* move scripts for setup
* update script and readme for infer
* do not reset logger when n processes == 1
* update eval_infer scripts and readme
* simplify readme
* copy over dir after eval
* copy over dir after eval
* directly return get state
* update lock
* fix output saving of infer
* replace print with logger
* update eval_infer script
* add back the missing .close
* increase timeout
* copy all swe_bench_format file
* attempt to fix output parsing
* log git commit id as metadata
* fix eval script
* update lock
* update unit tests
* fix argparser unit test
* fix lock
* the deps are now lightweight enough to be incude in make build
* add spaces for tests
* add eval outputs to gitignore
* remove git submodule
* readme
* tweak git email
* update upload instruction
* bump codeact version for eval
---------
Co-authored-by: Bowen Li <libowen.ne@gmail.com>
Co-authored-by: huybery <huybery@gmail.com>
Co-authored-by: Bart Shappee <bshappee@gmail.com>
Co-authored-by: Robert Brennan <accounts@rbren.io>
* remove extra actions
* remove message observations
* support null obs
* handle null obs
* fix frontend for changes
* fix the way messages flow to the UI
* change think to message
* add regen script
* regenerate all integration tests
* change task
* remove gh test
* fix messages
* fix tests
* help agent exit after hitting max iter
* Update opendevin/events/observation/success.py
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* Update agenthub/codeact_agent/codeact_agent.py
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
---------
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
* Add integration test framework with mock llm
* Fix MonologueAgent and PlannerAgent tests
* Remove adhoc logging
* Use existing logs
* Fix SWEAgent and PlannerAgent
* Check-in test log files
* conftest: look up under test name folder only
* Add docstring to conftest
* Finish dev doc
* Avoid non-determinism
* Remove dependency on llm embedding model
* Init embedding model only for MonologueAgent
* Add adhoc fix for sandbox discrepancy
* Test ssh and exec sandboxes
* CI: fix missing sandbox type
* conftest: Remove hack
* Reword comment for TODO
* feat: websocket connection management and sandbox bound to session.
* fix: set default value to id
* feat: add session management.
* fix for mypy
* fix for mypy
* fix the pnpm-lock.
* fix the default model is empty will throw error.
* create the pyproject file
* Fix the pyproject.toml file
* Update Makefile
* adapt makefile
* fix some execution issues
* Untrack lock files and wait for the backend to get start before frontend
* Remove LangChain dependencies
* Add github action for pytest
* add missing dependency
* rebase and fix the versions adding lock file
* add torch and pymupdfb deps
* some conflicts fixes
* Add dependencies evaluation group
* add poetry.lock
* Fix unexpected operator
---------
Co-authored-by: Robert Brennan <contact@rbren.io>
* a starting point for SWE-Bench evaluation with docker
* fix the swe-bench uid issue
* typo fixed
* fix conda missing issue
* move files based on new PR
* Update doc and gitignore using devin prediction file from #81
* fix typo
* add a sentence
* fix typo in path
* fix path
---------
Co-authored-by: Binyuan Hui <binyuan.hby@alibaba-inc.com>
* minimal docker sandbox
* make container_image as an argument (fall back to ubuntu);
increase timeout to avoid return too early for long running commands;
* add a minimal working (imperfect) example
* fix typo
* change default container name
* attempt to fix "Bad file descriptor" error
* handle ctrl+D
* add Python gitignore
* push sandbox to shared dockerhub for ease of use
* move codeact example into research folder
* add README for opendevin
* change container image name to opendevin dockerhub
* move folder; change example to a more general agent
* update Message and Role
* update docker sandbox to support mounting folder and switch to user with correct permission
* make network as host
* handle erorrs when attrs are not set yet
* convert codeact agent into a compatible agent
* add workspace to gitignore
* make sure the agent interface adjustment works for langchain_agent
