github/OpenHands
1
1
Fork 0
mirror of https://github.com/All-Hands-AI/OpenHands.git synced 2026-01-08 22:38:05 -05:00
Code Issues Packages Projects Releases Wiki Activity

Fix issue #5735: [Bug]: Inconsistent command line arguments in evaluation directory (#5736)

Browse Source
This commit is contained in:
OpenHands
2024-12-21 15:41:39 -05:00
committed by GitHub
parent d646b2089d
commit 21948fa81b
6 changed files with 7 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
evaluation/benchmarks/toolqa/README.md
View File

@@ -11,7 +11,7 @@ Please follow instruction [here](../../README.md#setup) to setup your local deve
Make sure your Docker daemon is running, then run this bash script:
```bash
bash evaluation/benchmarks/toolqa/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [dataset] [hardness] [wolfram_alpha_appid]
bash evaluation/benchmarks/toolqa/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [dataset] [hardness] [wolfram-alpha-appid]
```
where `model_config` is mandatory, while all other arguments are optional.
@@ -32,7 +32,7 @@ By default, the script evaluates 1 instance.
`hardness`, the hardness to evaluate. You could choose from `easy` and `hard`. The default is `easy`.
`wolfram_alpha_appid` is an optional argument. When given `wolfram_alpha_appid`, the agent will be able to access Wolfram Alpha's APIs.
`wolfram-alpha-appid` is an optional argument. When given `wolfram-alpha-appid`, the agent will be able to access Wolfram Alpha's APIs.
Note: in order to use `eval_limit`, you must also set `agent`; in order to use `dataset`, you must also set `eval_limit`; in order to use `hardness`, you must also set `dataset`.

2
evaluation/benchmarks/toolqa/run_infer.py
View File

@@ -171,7 +171,7 @@ if __name__ == '__main__':
default='easy',
)
parser.add_argument(
'--wolfram_alpha_appid',
'--wolfram-alpha-appid',
type=str,
help='wolfram alpha appid to use for wolfram alpha related tests',
default='YOUR_WOLFRAMALPHA_APPID',

2
evaluation/benchmarks/toolqa/scripts/run_infer.sh
View File

@@ -53,7 +53,7 @@ COMMAND="poetry run python evaluation/benchmarks/toolqa/run_infer.py \
--max-iterations 30 \
--dataset $DATASET \
--hardness $HARDNESS \
--wolfram_alpha_appid $WOLFRAM_APPID\
--wolfram-alpha-appid $WOLFRAM_APPID\
--data-split validation \
--eval-num-workers $NUM_WORKERS \
--eval-note ${OPENHANDS_VERSION}_${LEVELS}"