Commit Graph

14 Commits

Author SHA1 Message Date
Ean Garvey
98b069488e Add tank_version.json (#1272) 2023-04-03 18:36:23 -07:00
Ean Garvey
2191fc8952 Separate pytest benchmark modes and fix model updates for SHARK downloader / pytest. (#1264)
* Only xfail windows models in CI

* downloader: make model updates more robust.

* Separate baseline and native benchmarks in pytest.

* Fix native benchmarks

* Fix torchvision model utils.
2023-04-03 08:24:21 -07:00
Ean Garvey
96b1560da5 Make batch size configurable via pytest and fix sharktank generation. (#1227)
* Fix sharktank generation and add batch_size pytest option for torch.

* Disable torch dynamo until py3.11 supported

* Compile torchmodel without dynamo if torch.compile fails

* Use release versions of TF/Keras for importer.

* Pin torchvision and remove debug prints.

* Remove duplicates from torch model list.

* Update generate_sharktank.py

* xfail a few models that fail sharktank generation/ numerics
2023-03-28 14:33:39 -05:00
Ean Garvey
6d2a485264 Add --benchmark_dispatches option to pytest. (#800)
* Add --benchmark_dispatches option to pytest.

* Update README.md and fix filepath for dispatch benchmarks
2023-02-19 12:16:18 -06:00
powderluv
d946cffabc Revert "Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#602)" (#622)
This reverts commit fe618811ee.
2022-12-13 21:49:46 -08:00
Ean Garvey
fe618811ee Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#602)
* Move most xfails to entries in tank/all_models.csv

* enable usage of pytest without specifying tank/test_models.py

* add dict_configs.py to gitignore.

* Pin versions for runtimes and torch-mlir for setup.
2022-12-13 18:11:17 -08:00
Ean Garvey
9956099516 Add pytest option for updating tank and fix save_mlir function. (#413)
* Use IREE tf tools to save .mlir modules when generating shark_tank.

* Add option to pytest for enabling auto-updates to local shark tank.

* xfail mobilenet torch on cpu, cuda and fix CI macos setup

* Update test-models.yml to disable macos vulkan CI.
2022-10-25 21:29:18 +05:30
Ean Garvey
9035a2eed3 Add --local_tank_cache flag and update requirements. (#368)
* Add --local_tank_cache flag and update requirements.

* Update requirements-importer.txt
2022-09-28 03:02:59 -05:00
Ean Garvey
cbf3f784aa Add pytest option to specify a URL for shark tank artifacts. (#363)
* Xfail updates.

* Generalize tank SHA option to bucket address and add pytest option.
2022-09-27 02:40:40 -05:00
Ean Garvey
c0f3a09a40 Include SHA in path to failure reproducers. Add --save_fails option. (#352) 2022-09-21 17:55:06 -05:00
Ean Garvey
c061a8897d Add pytest options to save reproducers. (#350)
* Add pytest options to save and/or upload reproducers.

* pass shark_module to benchmark method.
2022-09-20 20:29:46 -05:00
Daniel Garvey
5df582e7e8 creates abstract test case class (#333) 2022-09-20 07:06:38 -07:00
Ean Garvey
3514822cac Improvements to pytest benchmarks. (#267)
* Add ONNX env var flags for venv setup.

* Setup arguments for ONNX benchmarking via pytest.

* Enable ONNX benchmarking on MiniLM via pytest (experimental)

* Fix sequence lengths to 128 for TF model creation and fix issue with benchmarks.

* Disable CI CPU benchmarks on A100, change some default args.

* add xfails for roberta TF model tests on GPU.
2022-08-17 02:29:48 -05:00
Ean Garvey
71f5cfcb30 Fix package management in tank for Nightly build. (#152)
Fix formatting.
2022-06-22 15:51:57 -07:00