tinygrad/test/external/mlperf_bert/preprocessing/tokenization.py at python_speed

mirror of https://github.com/tinygrad/tinygrad.git synced 2026-01-13 17:08:11 -05:00

Files

Elias Wahl 69341144ba Wikipedia preprocessing script (#4229 )

* Preprocessing script

* short seq prob

* comments + env vars

* Add preprocessing reference. Add test

* lint fix + add eval test support

* whitespaces

* point to commit

* comment

* rename

* better comments

2024-04-23 10:28:01 -04:00

12 KiB

Raw Permalink Blame History

View Raw

12 KiB Raw Permalink Blame History

12 KiB

Raw Permalink Blame History