- fix setup_venv.sh for benchmarks/imports etc.
- fix torch benchmarks in SharkBenchmarkRunner
- generate SD artifacts using build_tools/stable_diffusion_testing.py and --import_mlir
- decouple SD gen from tank/generate_sharktank for now
* Only xfail windows models in CI
* downloader: make model updates more robust.
* Separate baseline and native benchmarks in pytest.
* Fix native benchmarks
* Fix torchvision model utils.
* Adds a few xfails to enable macOS builder
* Convert string batch sizes to ints where needed.
* allow pytest to retry getting model artifacts
* Reduce attempts and add assert msg.
* Fix sharktank generation and add batch_size pytest option for torch.
* Disable torch dynamo until py3.11 supported
* Compile torchmodel without dynamo if torch.compile fails
* Use release versions of TF/Keras for importer.
* Pin torchvision and remove debug prints.
* Remove duplicates from torch model list.
* Update generate_sharktank.py
* xfail a few models that fail sharktank generation/ numerics
* Rollback T5 models for torch as the inputs give some issues that aren't trivial to resolve
* xfail efficientnet-b0 on torch+cuda -- see CUDA requesting shared memory size larger than allowed size openxla/iree#12771
* Minor improvements to test-models workflow
- cleaned up pytest command line args in Validate Models job scripts.
- Removed -s flag to provide more readable logs
- Changed shark_cache location to within github workspace and removed --update_tank flag from Linux workflows.
* Use pytest-forked for managing pytest memory usage.
-Adds date variable back to nightly.yml so shark_tank uploads are dated again
-added specification for nightly pytests to not run tests on metal (vulkan is sufficient)
-added some paths/filetypes to be ignored when triggering workflow runs. (no test-models on changes to .md files or anything in the shark/examples/ directory or its subdirectories.
-pytest only picks up tank/test_models.py, so no need to specify which file to run when running pytest from SHARK base directory.
-Cleaned up xfails so that they can be added to models as csv entries. Columns 7-9 in all_models.csv trigger xfails with cpu, cuda, vulkan, respectively, and row 10 can be populated with a reason for the xfails.
-Fixed a few defaults for shark_args and pytest args (defined in conftest.py)
-Fixes --update_tank option in shark_downloader
removes some multiprocessing in pytest / TF+CUDA support because it breaks pytest and false passes, leaving regressions at large.
-Adds xfails for and removes albert torch from gen_sharktank list (tank/torch_model_list.csv).
-Cleans up xfails for cpu, cuda, vulkan (removing old ones)
* Move most xfails to entries in tank/all_models.csv
* enable usage of pytest without specifying tank/test_models.py
* add dict_configs.py to gitignore.
* Pin versions for runtimes and torch-mlir for setup.
