leopf 4f0ee4e982 BPE tokenizer (#11415) ...

* BPE works

* refactor tok

* oops

* basic tests

* fix eval

* smaller diff

* fix error

* proper vocab decoding

* use regex for splitting

* escape ucatrange

* full compat

---------

Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>

2025-08-04 09:52:38 -07:00

..

__init__.py

more test cleanups (#2631)

2023-12-05 16:17:57 -08:00

test_allreduce.py

minimal mstack pr to fix allreduce (#10649)

2025-06-05 15:14:53 -07:00

test_attention.py

move cast to before softmax in attention (#9213)

2025-02-24 17:24:59 -05:00

test_block_reorder.py

Pass list to block_reorder in test_loads (#10881)

2025-06-19 09:49:45 -04:00

test_conv_shapetracker.py

move test_conv_shapetracker and some test_search util into unit test (#10812)

2025-06-14 13:29:32 -04:00

test_conv.py

move high level stuff to unit tests [pr] (#10708)

2025-06-08 14:05:56 -07:00

test_device.py

Fix device (#10929)

2025-06-23 15:30:19 -07:00

test_disk_cache.py

helpers.diskcache_clear (#4436)

2024-05-05 14:19:01 -04:00

test_disk_tensor.py

fix DiskDevice reuse (#11039)

2025-07-01 10:29:21 -04:00

test_dtype_spec.py

move dtype spec tests into unit test (#10808)

2025-06-13 22:21:22 -04:00

test_dtype.py

few more dtype cast convinience methods (#11480)

2025-08-02 15:47:09 -04:00

test_elf.py

CLANG -> CPU (#9189)

2025-02-20 18:03:09 -05:00

test_gguf.py

add a missing unittest.main() [pr] (#8274)

2024-12-16 14:28:10 -05:00

test_gradient.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

test_graph_rewrite.py

hotfix: infinite loop in tracking pattern matcher (#11094)

2025-07-04 19:55:26 +03:00

test_hashing.py

add kernelize to keccak for each data block (#11370)

2025-07-25 16:07:20 -04:00

test_helpers.py

fix word_wrap with newlines in input string [pr] (#11319)

2025-07-22 12:03:13 +03:00

test_indexing.py

move high level stuff to unit tests [pr] (#10708)

2025-06-08 14:05:56 -07:00

test_kernelize.py

move view left to the outer graph prereqs + testing (#10725)

2025-06-09 20:43:25 -07:00

test_linearizer_rewrite.py

tc should be in opt (#11148)

2025-07-09 14:12:21 -07:00

test_llm_tokenizer.py

BPE tokenizer (#11415)

2025-08-04 09:52:38 -07:00

test_masked_st.py

move high level tests to unit (#10760)

2025-06-10 12:55:44 -04:00

test_microbenchmarks.py

move files into uop dir (#10399)

2025-05-18 11:38:28 -07:00

test_mnist_dataset.py

move high level tests to unit (#10760)

2025-06-10 12:55:44 -04:00

test_pattern_matcher.py

if upat returns self, it's none (#10898)

2025-06-20 12:11:19 -07:00

test_qcom.py

BufferSpec and ProgramSpec [pr] (#7814)

2024-11-21 12:18:05 +08:00

test_rearrange_einops.py

move high level tests to unit (#10760)

2025-06-10 12:55:44 -04:00

test_rewrite_map.py

move symbolic and transcendental to uop [pr] (#10771)

2025-06-10 20:51:22 -07:00

test_rewrite_tracked_childen.py

rename kernelize to schedule, try 2 (#11305)

2025-07-21 11:18:36 -07:00

test_search.py

remove Kernel.membufs [pr] (#11200)

2025-07-12 14:48:47 -04:00

test_shapetracker_math.py

[bounty] Stride is flip (#8792)

2025-01-31 11:34:10 +09:00

test_shapetracker.py

remove unused ShapeTracker.consecutive [pr] (#11426)

2025-07-29 18:36:19 -04:00

test_shm_tensor.py

line reduction [pr] (#7296)

2024-10-25 17:05:09 +07:00

test_simple_schedule.py

simple schedule test [pr] (#10853)

2025-06-17 16:19:27 -07:00

test_simplify_valid_idx.py

move symbolic and transcendental to uop [pr] (#10771)

2025-06-10 20:51:22 -07:00

test_symbolic_failures.py

[bounty] Z3 symbolic fuzzer [pr] (#10514)

2025-05-28 16:28:37 -04:00

test_symbolic_shapetracker.py

move high level tests to unit (#10760)

2025-06-10 12:55:44 -04:00

test_tar.py

accept filename decorator [pr] (#8049)

2024-12-05 11:40:59 +08:00

test_tensor_io.py

tar_extract with Tensors (#7853)

2024-12-04 17:03:19 +08:00

test_tensor_uop_representation.py

update test_tensor_uop_representation comments [pr] (#10946)

2025-06-24 10:47:09 +03:00

test_tqdm.py

Change rate to int in test_tqdm (#10848)

2025-06-18 08:40:43 -04:00

test_transcendental_helpers.py

move symbolic and transcendental to uop [pr] (#10771)

2025-06-10 20:51:22 -07:00

test_uop_resolve.py

move files into uop dir (#10399)

2025-05-18 11:38:28 -07:00

test_uop_spec.py

Fix test_uop_spec (#11259)

2025-07-16 11:02:31 +03:00

test_uop_symbolic.py

1/(x*c) -> (1/c)*(1/x) (#11491)

2025-08-03 23:35:46 -04:00

test_uop_vmin_vmax.py

don't narrow range of CAST on bool/unsigned (#11156)

2025-07-09 22:20:09 -04:00

test_upat_compile.py

move files into uop dir (#10399)

2025-05-18 11:38:28 -07:00

test_view.py

Fix division by zero and mask bug in add views (#11088)

2025-07-07 10:05:47 -07:00

test_viz.py

viz: show const node when it's root (#11456)

2025-08-01 01:01:58 +03:00