github/tinygrad
1
1
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-01-10 23:48:01 -05:00
Code Issues Packages Projects Releases Wiki Activity

add MobileNetV2 benchmark to comma CI (#10250)

Browse Source 
* add MobileNetV2 to comma CI

* symlink imagenet

* also the signature

* comment that out

* need imagenetmock

* same train and test set

* quantize on CPU=1

* verbose

* need __hexagon_divsf3

* 0x858d6c15

* quant cpu + CC=clang-19
This commit is contained in:
qazal
2025-05-19 18:22:50 +03:00
committed by GitHub
parent f9a5ad24c5
commit 90eb3c0e5d
2 changed files with 13 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.github/workflows/benchmark.yml vendored
View File

@@ -573,6 +573,14 @@ jobs:
run: PYTHONPATH="." QCOM=1 taskset -c 4-7 python3 examples/openpilot/compile3.py https://github.com/commaai/openpilot/raw/e8bea2c78ffa92685ece511e9b554122aaf1a79d/selfdrive/modeld/models/supercombo.onnx
- name: openpilot dmonitoring compile3 0.9.7
run: PYTHONPATH="." QCOM=1 taskset -c 4-7 python3 examples/openpilot/compile3.py https://github.com/commaai/openpilot/raw/v0.9.7/selfdrive/modeld/models/dmonitoring_model.onnx
- name: benchmark MobileNetV2 on DSP
run: |
# generate quantized weights
ln -s /data/home/tiny/tinygrad/extra/datasets/imagenet extra/datasets/imagenet
ln -s /data/home/tiny/tinygrad/testsig-0x858d6c15.so .
PYTHONPATH=. CC=clang-19 CPU=1 QUANT=1 CNT=0 python3 examples/test_onnx_imagenet.py https://github.com/xamcat/mobcat-samples/raw/refs/heads/master/onnx_runtime/InferencingSample/InferencingSample/mobilenetv2-7.onnx /tmp/model.quant.onnx
# benchmark on DSP with NOOPT=1, the devectorizer has issues
PYTHONPATH=. CC=clang-19 DSP=1 DONT_REALIZE_EXPAND=1 NOOPT=1 CNT=2 DEBUG=2 python3 examples/test_onnx_imagenet.py /tmp/model.quant.onnx
- name: Run process replay tests
run: cp test/external/process_replay/process_replay.py ./process_replay.py && git fetch origin master && git -c advice.detachedHead=false checkout origin/master && PYTHONPATH=. python3 process_replay.py
- uses: actions/upload-artifact@v4

9
examples/test_onnx_imagenet.py
View File

@@ -27,7 +27,7 @@ def imagenet_dataloader(cnt=0):
input_std = Tensor([0.229, 0.224, 0.225]).reshape(1, -1, 1, 1)
files = get_val_files()
random.shuffle(files)
if cnt != 0: files = files[:cnt]
files = files[:cnt]
cir = get_imagenet_categories()
for fn in files:
img = Image.open(fn)
@@ -66,7 +66,7 @@ if __name__ == "__main__":
assert t_spec.shape[1:] == (3,224,224), f"shape is {t_spec.shape}"
hit = 0
for i,(img,y) in enumerate(imagenet_dataloader(cnt=getenv("CNT", 100))):
for i,(img,y) in enumerate(imagenet_dataloader(cnt:=getenv("CNT", 100))):
GlobalCounters.reset()
p = run_onnx_jit(**{t_name:img})
assert p.shape == (1,1000)
@@ -77,5 +77,6 @@ if __name__ == "__main__":
MS_TARGET = 13.4
print(f"need {GlobalCounters.global_ops/1e9*(1000/MS_TARGET):.2f} GFLOPS for {MS_TARGET:.2f} ms")
import pickle
with open("/tmp/im.pkl", "wb") as f: pickle.dump(run_onnx_jit, f)
if cnt >= 2:
import pickle
with open("/tmp/im.pkl", "wb") as f: pickle.dump(run_onnx_jit, f)