Commit Graph

122 Commits

Author SHA1 Message Date
Ean Garvey
c28cc4c919 Fix local_tank_cache handling in shark_downloader. (#981) 2023-02-09 14:52:03 -06:00
Ean Garvey
1eb9436836 Fix generate_sharktank args. 2023-02-07 14:06:07 +05:30
Ean Garvey
9604d9ce81 make --update_tank update only if hash mismatch 2023-02-07 14:06:07 +05:30
Ean Garvey
481d0553d8 Remove unnecessary repro_dir / shark_tmp usage 2023-02-07 14:06:07 +05:30
Daniel Garvey
157ae64f9d print to stdout for test visibility (#937)
Co-authored-by: dan <dan@nod-labs.com>
2023-02-06 01:03:27 -08:00
Ean Garvey
a90812133b Enable pytests on Windows (#901) 2023-02-01 18:36:41 -06:00
Daniel Garvey
e374074013 Windows test (#896)
* add generate_sharktank for stable_diffusion model defaults

* add windows test for sd

---------

Co-authored-by: dan <dan@nod-labs.com>
2023-02-01 12:03:54 -06:00
Daniel Garvey
6bf991edf6 adding more robust main.py testing (#889)
Co-authored-by: dan <dan@nod-labs.com>
2023-01-30 00:14:26 -08:00
Ean Garvey
fb6beae27c Adds pytest-forked dependency to fix pytest memory accumulation issues. (#876)
* Minor improvements to test-models workflow

- cleaned up pytest command line args in Validate Models job scripts.
- Removed -s flag to provide more readable logs
- Changed shark_cache location to within github workspace and removed --update_tank flag from Linux workflows.

* Use pytest-forked for managing pytest memory usage.
2023-01-26 18:20:15 -06:00
Daniel Garvey
e842c8c19b add main.py testing for sdiff (#836)
Co-authored-by: dan <dan@nod-labs.com>
Co-authored-by: powderluv <powderluv@users.noreply.github.com>
2023-01-22 01:16:17 -08:00
Ean Garvey
a14a47af12 Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#646)
-Adds date variable back to nightly.yml so shark_tank uploads are dated again
-added specification for nightly pytests to not run tests on metal (vulkan is sufficient)
-added some paths/filetypes to be ignored when triggering workflow runs. (no test-models on changes to .md files or anything in the shark/examples/ directory or its subdirectories.
-pytest only picks up tank/test_models.py, so no need to specify which file to run when running pytest from SHARK base directory.
-Cleaned up xfails so that they can be added to models as csv entries. Columns 7-9 in all_models.csv trigger xfails with cpu, cuda, vulkan, respectively, and row 10 can be populated with a reason for the xfails.
-Fixed a few defaults for shark_args and pytest args (defined in conftest.py)
-Fixes --update_tank option in shark_downloader
removes some multiprocessing in pytest / TF+CUDA support because it breaks pytest and false passes, leaving regressions at large.
-Adds xfails for and removes albert torch from gen_sharktank list (tank/torch_model_list.csv).
-Cleans up xfails for cpu, cuda, vulkan (removing old ones)
2022-12-16 12:56:32 +05:30
Phaneesh Barwaria
6aed4423b2 add vulkan lib path (#638) 2022-12-15 19:48:29 +07:00
powderluv
d946cffabc Revert "Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#602)" (#622)
This reverts commit fe618811ee.
2022-12-13 21:49:46 -08:00
Ean Garvey
fe618811ee Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#602)
* Move most xfails to entries in tank/all_models.csv

* enable usage of pytest without specifying tank/test_models.py

* add dict_configs.py to gitignore.

* Pin versions for runtimes and torch-mlir for setup.
2022-12-13 18:11:17 -08:00
Ean Garvey
616ee9b824 Don't include baseline benchmarks if setup without IMPORTER=1. (#607) 2022-12-10 14:58:29 -06:00
PhaneeshB
64d6da75c7 Resolve Mac torch-mlir torch setup dependency. Enable MacOS CI 2022-11-07 15:38:37 +05:30
Ean Garvey
9956099516 Add pytest option for updating tank and fix save_mlir function. (#413)
* Use IREE tf tools to save .mlir modules when generating shark_tank.

* Add option to pytest for enabling auto-updates to local shark tank.

* xfail mobilenet torch on cpu, cuda and fix CI macos setup

* Update test-models.yml to disable macos vulkan CI.
2022-10-25 21:29:18 +05:30
Phaneesh Barwaria
21fee8ef33 enable only one workflow job per branch (#402) 2022-10-13 12:15:30 -05:00
Phaneesh Barwaria
00a8ce75d1 Xfail vulkan tests and Enable MacOs test on CI (#383) 2022-10-13 11:14:41 -05:00
Daniel Garvey
d38e37bd99 seperate importer and benchmark deps (#393) 2022-10-08 23:31:20 -05:00
Ean Garvey
9035a2eed3 Add --local_tank_cache flag and update requirements. (#368)
* Add --local_tank_cache flag and update requirements.

* Update requirements-importer.txt
2022-09-28 03:02:59 -05:00
Ean Garvey
c0f3a09a40 Include SHA in path to failure reproducers. Add --save_fails option. (#352) 2022-09-21 17:55:06 -05:00
Ean Garvey
c061a8897d Add pytest options to save reproducers. (#350)
* Add pytest options to save and/or upload reproducers.

* pass shark_module to benchmark method.
2022-09-20 20:29:46 -05:00
Daniel Garvey
5df582e7e8 creates abstract test case class (#333) 2022-09-20 07:06:38 -07:00
Ean Garvey
6cf5564c84 Remove "gpu" device alias and migrate to using "cuda" for NVIDIA GPU. (#325)
* Replace instances of "gpu" alias for devices with "cuda"
2022-09-13 01:16:56 -05:00
Ean Garvey
7143902a90 Update test-models.yml (#323) 2022-09-12 22:47:40 -05:00
Ean Garvey
d453f2e49d Enable CPU benchmarks on test-models workflows. (#299)
* Update test-models.yml

* Update README.md
2022-09-07 01:22:58 -05:00
powderluv
4ee164c66f remove a100 cpu 2022-08-31 12:59:47 -07:00
powderluv
bf84c033bb add icelake 2022-08-31 12:58:40 -07:00
Phaneesh Barwaria
1322ec5935 Simplified Testing Interface (#289) 2022-08-24 23:54:56 -05:00
Ean Garvey
3514822cac Improvements to pytest benchmarks. (#267)
* Add ONNX env var flags for venv setup.

* Setup arguments for ONNX benchmarking via pytest.

* Enable ONNX benchmarking on MiniLM via pytest (experimental)

* Fix sequence lengths to 128 for TF model creation and fix issue with benchmarks.

* Disable CI CPU benchmarks on A100, change some default args.

* add xfails for roberta TF model tests on GPU.
2022-08-17 02:29:48 -05:00
Ean Garvey
22ff92c48b Add config.VmModule argument to from_flatbuffer call. (#266) 2022-08-14 15:11:19 -07:00
powderluv
c2aa451767 Update test-models.yml 2022-08-09 10:12:59 -07:00
Ean Garvey
6f5ceb4e61 Update test-models.yml (#244) 2022-08-04 21:56:08 -07:00
Ean Garvey
6c22139ac9 Upload benchmark results for every test-models workflow (excl. Vulkan) (#241)
* Upload benchmark results for every test-models workflow (excl. Vulkan)
2022-08-04 14:43:07 -07:00
powderluv
c6b4ad1e26 Update test-models.yml 2022-08-02 13:19:10 -07:00
Phaneesh Barwaria
198755788c Check for MacStudio Target Value (#233)
* Identify Apple M2 for Vulkan

* Check for MacOS Target
2022-08-02 13:18:36 -07:00
Ean Garvey
0ee515a7be Fix GPU benchmarks on PyTorch model tests. (#218)
* Add CUDA_BENCHMARKS option to setup.venv to enable PyTorch benchmarks on CUDA.

* Fix PyTorch GPU benchmarks for tank models.
2022-07-29 15:17:55 -07:00
Daniel Garvey
4c0deb9899 remove gpu script (#220)
No longer needed after fixes in iree
2022-07-29 15:20:03 -05:00
Ean Garvey
921ccdc40b Force TensorFlow to use CPU when generating model and golden values. (#199)
Split up gpu pytest runs temporarily
2022-07-22 20:21:16 -07:00
Ean Garvey
d047eff086 Update test-models.yml 2022-07-16 16:38:32 -05:00
monorimet
5da32601c0 Ensure CPU workflow job suite only collects CPU tests. 2022-07-16 13:50:36 -05:00
Prashant Kumar
0dcf387089 Add shark_importer for torch_models. (#183)
All the torch_models are imported to gs::shark_tank.
Scripts have been updated.
2022-07-12 20:38:19 -07:00
powderluv
c351bb50b6 Update test-models.yml 2022-07-03 13:56:45 -07:00
powderluv
ca36d40f0a Update test-models.yml 2022-07-01 12:27:49 -07:00
powderluv
cc11a71ec8 Update test-models.yml 2022-06-30 18:55:38 -07:00
powderluv
1d1a1b131f Update test-models.yml 2022-06-30 18:54:50 -07:00
powderluv
84a2969381 Update test-models.yml 2022-06-30 15:57:07 -07:00
powderluv
8199ea15e8 Update test-models.yml 2022-06-30 14:42:41 -07:00
powderluv
958e0100bf Update test-models.yml 2022-06-30 14:41:26 -07:00