SHARK-Studio/tank/all_models.csv at 20221226.414

mirror of https://github.com/nod-ai/SHARK-Studio.git synced 2026-05-12 03:00:11 -04:00

Files

Ean Garvey a14a47af12 Move most xfails to entries in tank/all_models.csv and temporarily remove multiprocessing and TF gpu support. (#646 )

-Adds date variable back to nightly.yml so shark_tank uploads are dated again
-added specification for nightly pytests to not run tests on metal (vulkan is sufficient)
-added some paths/filetypes to be ignored when triggering workflow runs. (no test-models on changes to .md files or anything in the shark/examples/ directory or its subdirectories.
-pytest only picks up tank/test_models.py, so no need to specify which file to run when running pytest from SHARK base directory.
-Cleaned up xfails so that they can be added to models as csv entries. Columns 7-9 in all_models.csv trigger xfails with cpu, cuda, vulkan, respectively, and row 10 can be populated with a reason for the xfails.
-Fixed a few defaults for shark_args and pytest args (defined in conftest.py)
-Fixes --update_tank option in shark_downloader
removes some multiprocessing in pytest / TF+CUDA support because it breaks pytest and false passes, leaving regressions at large.
-Adds xfails for and removes albert torch from gen_sharktank list (tank/torch_model_list.csv).
-Cleans up xfails for cpu, cuda, vulkan (removing old ones)

2022-12-16 12:56:32 +05:30

3.4 KiB

Raw Permalink Blame History

1	resnet50	mhlo	tf	1e-2	1e-3	default	nhcw-nhwc	False	False	True	Vulkan Numerical Error: mostly conv
2	albert-base-v2	mhlo	tf	1e-2	1e-2	default	None	False	False	False
3	roberta-base	mhlo	tf	1e-02	1e-3	default	nhcw-nhwc	False	False	False
4	bert-base-uncased	mhlo	tf	1e-2	1e-3	default	None	False	False	False
5	camembert-base	mhlo	tf	1e-2	1e-3	default	None	False	False	False
6	dbmdz/convbert-base-turkish-cased	mhlo	tf	1e-2	1e-3	default	nhcw-nhwc	True	True	True	https://github.com/iree-org/iree/issues/9971
7	distilbert-base-uncased	mhlo	tf	1e-2	1e-3	default	None	False	False	False
8	facebook/convnext-tiny-224	mhlo	tf	1e-2	1e-3	tf_vit	nhcw-nhwc	True	True	True	https://github.com/nod-ai/SHARK/issues/311 & https://github.com/nod-ai/SHARK/issues/342
9	funnel-transformer/small	mhlo	tf	1e-2	1e-3	default	None	True	True	True	https://github.com/nod-ai/SHARK/issues/201
10	google/electra-small-discriminator	mhlo	tf	1e-2	1e-3	default	None	False	False	False
11	google/mobilebert-uncased	mhlo	tf	1e-2	1e-3	default	None	True	False	False	Fails during iree-compile.
12	google/vit-base-patch16-224	mhlo	tf	1e-2	1e-3	tf_vit	nhcw-nhwc	False	False	True	Vulkan Numerical Error (mostly conv)
13	microsoft/MiniLM-L12-H384-uncased	mhlo	tf	1e-2	1e-3	tf_hf	None	True	False	False	Fails during iree-compile.
14	microsoft/layoutlm-base-uncased	mhlo	tf	1e-2	1e-3	default	None	False	False	False
15	microsoft/mpnet-base	mhlo	tf	1e-2	1e-2	default	None	False	False	False
16	albert-base-v2	linalg	torch	1e-2	1e-3	default	None	True	True	True	issue with aten.tanh in torch-mlir
17	alexnet	linalg	torch	1e-2	1e-3	default	None	False	False	True	Assertion Error: Zeros Output
18	bert-base-cased	linalg	torch	1e-2	1e-3	default	None	False	False	False
19	bert-base-uncased	linalg	torch	1e-2	1e-3	default	None	False	False	False
20	facebook/deit-small-distilled-patch16-224	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	True	False	Fails during iree-compile.
21	google/vit-base-patch16-224	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	True	False	https://github.com/nod-ai/SHARK/issues/311
22	microsoft/beit-base-patch16-224-pt22k-ft22k	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	True	False	https://github.com/nod-ai/SHARK/issues/390
23	microsoft/MiniLM-L12-H384-uncased	linalg	torch	1e-2	1e-3	default	None	False	False	True
24	microsoft/resnet-50	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	Vulkan Numerical Error (mostly conv)
25	google/mobilebert-uncased	linalg	torch	1e-2	1e-3	default	None	False	False	True	https://github.com/nod-ai/SHARK/issues/344
26	mobilenet_v3_small	linalg	torch	1e-1	1e-2	default	nhcw-nhwc	False	True	True	https://github.com/nod-ai/SHARK/issues/388
27	nvidia/mit-b0	linalg	torch	1e-2	1e-3	default	None	True	True	True	https://github.com/nod-ai/SHARK/issues/343
28	resnet101	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	Vulkan Numerical Error (mostly conv)
29	resnet18	linalg	torch	1e-2	1e-3	default	None	True	True	True
30	resnet50	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	Vulkan Numerical Error (mostly conv)
31	squeezenet1_0	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	https://github.com/nod-ai/SHARK/issues/388
32	wide_resnet50_2	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	Vulkan Numerical Error (mostly conv)
33	efficientnet-v2-s	mhlo	tf	1e-02	1e-3	default	nhcw-nhwc	False	False	True	https://github.com/nod-ai/SHARK/issues/575
34	mnasnet1_0	linalg	torch	1e-2	1e-3	default	nhcw-nhwc	False	False	True	https://github.com/nod-ai/SHARK/issues/388

3.4 KiB Raw Permalink Blame History

3.4 KiB

Raw Permalink Blame History