[Feat] add multi-swe-bench (#8174)

Co-authored-by: ByteDance User <tiger@bytedance.localdomain>

evaluation/benchmarks/multi_swe_bench/README.md

@@ -0,0 +1,65 @@
+# Multi-swe-bench Evaluation with OpenHands
+
+## LLM Setup
+
+Please follow [here](../../README.md#setup).
+
+## Dataset Preparing
+
+Please download the [**Multi-SWE-Bench** dataset](https://huggingface.co/datasets/bytedance-research/Multi-SWE-Bench).
+And change the dataset following [script](scripts/data/data_change.py).
+
+```bash
+python evaluation/benchmarks/multi_swe_bench/scripts/data/data_change.py
+```
+
+## Docker image download
+
+Please download the multi-swe-bench dokcer images from [here](https://github.com/multi-swe-bench/multi-swe-bench?tab=readme-ov-file#run-evaluation).
+
+## Generate patch
+
+Please edit the [script](infer.sh) and run it.
+
+```bash
+bash evaluation/benchmarks/multi_swe_bench/infer.sh
+```
+
+Script variable explanation:
+
+- `models`, e.g. `llm.eval_gpt4_1106_preview`, is the config group name for your
+LLM settings, as defined in your `config.toml`.
+- `git-version`, e.g. `HEAD`, is the git commit hash of the OpenHands version you would
+like to evaluate. It could also be a release tag like `0.6.2`.
+- `agent`, e.g. `CodeActAgent`, is the name of the agent for benchmarks, defaulting to `CodeActAgent`.
+- `eval_limit`, e.g. `10`, limits the evaluation to the first `eval_limit` instances. By
+default, the script evaluates the (500 issues), which will no exceed the maximum of the dataset number.
+- `max_iter`, e.g. `20`, is the maximum number of iterations for the agent to run. By
+default, it is set to 50.
+- `num_workers`, e.g. `3`, is the number of parallel workers to run the evaluation. By
+default, it is set to 1.
+- `language`, the language of your evaluating dataset.
+- `dataset`, the absolute position of the dataset jsonl.
+
+The results will be generated in evaluation/evaluation_outputs/outputs/XXX/CodeActAgent/YYY/output.jsonl, you can refer to the [example](examples/output.jsonl).
+
+## Runing evaluation
+
+First, install [multi-swe-bench](https://github.com/multi-swe-bench/multi-swe-bench).
+
+```bash
+pip install multi-swe-bench
+```
+
+Second, convert the output.jsonl to patch.jsonl with [script](scripts/eval/convert.py), you can refer to the [example](examples/patch.jsonl).
+
+```bash
+python evaluation/benchmarks/multi_swe_bench/scripts/eval/convert.py
+```
+
+Finally, evaluate with multi-swe-bench.
+The config file config.json can be refer to the [example](examples/config.json) or [github](https://github.com/multi-swe-bench/multi-swe-bench/tree/main?tab=readme-ov-file#configuration-file-example).
+
+```bash
+python -m multi_swe_bench.harness.run_evaluation --config config.json
+```