Xingyao Wang a6ba6c5277 Add SWEBench-docker eval (#2085) ...

* add initial version of swebench-docker eval

* update the branch of git repo

* add poetry run

* download dev set too and pre-load f2p and p2p

* update eval infer script

* increase timeout

* add poetry run

* install swebench from our fork

* update script

* update loc

* support single instance debug

* replace \r\n from model patch

* replace eval docker from namespace xingyaoww

* update script to auto detect swe-bench format jsonl

* support eval infer on single instance id

* change log output dir to logs

* update summarise result script

* update README

* update readme

* tweak branch

* Update evaluation/swe_bench/scripts/eval/prep_eval.sh

Co-authored-by: Graham Neubig <neubig@gmail.com>

---------

Co-authored-by: Graham Neubig <neubig@gmail.com>

2024-06-10 19:30:40 +00:00

..

docker

Add SWEBench-docker eval (#2085)

2024-06-10 19:30:40 +00:00

eval

Add SWEBench-docker eval (#2085)

2024-06-10 19:30:40 +00:00

setup

feat(SWE-Bench environment) integrate SWE-Bench sandbox (#1468)

2024-05-15 16:15:55 +00:00

eval_infer.sh

Add SWEBench-docker eval (#2085)

2024-06-10 19:30:40 +00:00

run_infer.sh

Add CodeActSWEAgent to remove browsing & github + improvements on agentskills (#2105)

2024-05-29 21:19:00 -07:00

summarise_results.py

Add SWEBench-docker eval (#2085)

2024-06-10 19:30:40 +00:00