Reinier van der Leer 327fb1f916 fix(benchmark): Mock mode, python evals, --attempts flag, challenge definitions ...

- Fixed `--mock` mode
   - Moved interrupt to beginning of the step iterator pipeline (from `BuiltinChallenge` to `agent_api_interface.py:run_api_agent`). This ensures that any finish-up code is properly executed after executing a single step.
   - Implemented mock mode in `WebArenaChallenge`

- Fixed `fixture 'i_attempt' not found` error when `--attempts`/`-N` is omitted

- Fixed handling of `python`/`pytest` evals in `BuiltinChallenge`

- Disabled left-over Helicone code (see 056163e)

- Fixed a couple of challenge definitions
   - WebArena task 107: fix spelling of months (Sepetember, Octorbor *lmao*)
   - synthesize/1_basic_content_gen (SynthesizeInfo): remove empty string from `should_contain` list

- Added some debug logging in agent_api_interface.py and challenges/builtin.py

2024-02-14 01:05:34 +01:00

..

code

Clean up & fix GitHub workflows (#6313)

2023-11-21 10:58:54 +01:00

data

case sensitivity, updating challenges

2023-10-20 08:26:29 -07:00

scrape

AGBenchmark codebase clean-up (#6650)

2024-01-02 22:23:09 +01:00

synthesize/1_basic_content_gen

fix(benchmark): Mock mode, python evals, --attempts flag, challenge definitions

2024-02-14 01:05:34 +01:00