mirror of
https://github.com/All-Hands-AI/OpenHands.git
synced 2026-04-29 03:00:45 -04:00
41ddba84bd
* add replace-based block edit & preliminary test case fix * further fix the insert behavior * make edit only work on first occurence * bump codeact version since we now use new edit agentskills * update prompt for new agentskills * update integration tests * make run_infer.sh executable * remove code block for edit_file * update integration test for prompt changes * default to not use hint for eval * fix insert emptyfile bug * throw value error when `to_replace` is empty * make `_edit_or_insert_file` return string so we can try to fix some linter errors (best attempt) * add todo * update integration test * fix sandbox test for this PR
Evaluation
This folder contains code and resources to run experiments and evaluations.
Logistics
To better organize the evaluation folder, we should follow the rules below:
- Each subfolder contains a specific benchmark or experiment. For example,
evaluation/swe_benchshould contain all the preprocessing/evaluation/analysis scripts. - Raw data and experimental records should not be stored within this repo.
- For model outputs, they should be stored at this huggingface space for visualization.
- Important data files of manageable size and analysis scripts (e.g., jupyter notebooks) can be directly uploaded to this repo.
Supported Benchmarks
- SWE-Bench:
evaluation/swe_bench - ML-Bench:
evaluation/ml_bench - HumanEvalFix:
evaluation/humanevalfix - GAIA:
evaluation/gaia - Entity deduction Arena (EDA):
evaluation/EDA - MINT:
evaluation/mint - AgentBench:
evaluation/agent_bench - BIRD:
evaluation/bird - LogicReasoning:
evaluation/logic_reasoning
Result Visualization
Check this huggingface space for visualization of existing experimental results.
Upload your results
You can start your own fork of our huggingface evaluation outputs and submit a PR of your evaluation results to our hosted huggingface repo via PR following the guide here.