github/tinygrad
1
1
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-01-14 09:28:04 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,106 Commits 203 Branches 14 Tags
python_speed
Commit Graph

3 Commits

Author SHA1 Message Date
George Hotz
b9eb5b5d49 clean up the LLM tokenizer (#12653) 
* clean up the LLM tokenizer

* simple tokenizer is actually simple

* ugh write good code
 2025-10-14 14:22:01 +08:00
George Hotz
6ed2dfd187 delete the arange dim mismatch restriction (#11568) 
* delete the arange dim mismatch restriction

* skip that test race
 2025-08-07 13:46:17 -07:00
leopf
4f0ee4e982 BPE tokenizer (#11415) 
* BPE works

* refactor tok

* oops

* basic tests

* fix eval

* smaller diff

* fix error

* proper vocab decoding

* use regex for splitting

* escape ucatrange

* full compat

---------

Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>
 2025-08-04 09:52:38 -07:00