leopf 4f0ee4e982 BPE tokenizer (#11415) ...

* BPE works

* refactor tok

* oops

* basic tests

* fix eval

* smaller diff

* fix error

* proper vocab decoding

* use regex for splitting

* escape ucatrange

* full compat

---------

Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>

2025-08-04 09:52:38 -07:00

..

mlperf_bert

delete SAVE_SCHEDULE=1 [pr] (#7087)

2024-10-16 07:13:20 +03:00

mlperf_resnet

PolynomialDecayWithWarmup + tests (#3649)

2024-03-07 18:53:36 -05:00

mlperf_retinanet

RetinaNet losses (#9536)

2025-03-21 15:52:54 -04:00

mlperf_unet3d

[MLPerf] UNet3D dataloader (#4343)

2024-04-28 22:34:18 -04:00

openpilot

fix failed threefry (#10646)

2025-06-05 17:17:42 -07:00

process_replay

fix double exception + add name,loc in error msg (#11487)

2025-08-04 13:41:23 +03:00

sglang_llama

refactor test/external/external_llama_eval.py (#10567)

2025-06-23 10:43:20 -07:00

external_benchmark_am.py

clean: becnhmark -> benchmark (#10620)

2025-06-03 19:28:18 -07:00

external_benchmark_bert_matmuls.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_benchmark_bert_softmax.py

update benchmark bert scripts with BS and ACC_DTYPE (#9826)

2025-04-10 02:06:02 -04:00

external_benchmark_disk_raw.py

ci: add h machines (#11416)

2025-07-29 19:21:51 +03:00

external_benchmark_hcopt.py

move kernel to opt (#10899)

2025-06-20 15:22:28 -07:00

external_benchmark_hip_compile.py

hip compile speed (#2606)

2023-12-04 13:47:40 -08:00

external_benchmark_keccak.py

Tensor.keccak("sha3_256") (#7186)

2025-06-06 15:24:05 -07:00

external_benchmark_kernel_launch.py

hotfix: GRAPH_ONE_KERNEL + fix timing

2025-02-06 17:52:20 +08:00

external_benchmark_llama_schedule.py

bump line count to 14500

2025-06-23 15:32:27 -07:00

external_benchmark_load_stable_diffusion.py

gate METAL_FAST_LOAD

2023-12-01 15:28:40 -08:00

external_benchmark_multitensor_allreduce.py

add some options to allreduce bench (#9348)

2025-03-04 23:46:36 +03:00

external_benchmark_openpilot.py

OnnxRunner file as input (#10789)

2025-07-12 14:27:46 -04:00

external_benchmark_resnet.py

delete SAVE_SCHEDULE=1 [pr] (#7087)

2024-10-16 07:13:20 +03:00

external_benchmark_schedule.py

move kernel to opt (#10899)

2025-06-20 15:22:28 -07:00

external_benchmark_sdxl_softmax.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

external_cl_half_max.py

use default dict for external_model_benchmark (#2592)

2023-12-03 15:25:43 -08:00

external_debug_metal_sd_conv.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

external_fuzz_am_interrupts.py

am: recover from any boot interrupt (#8703)

2025-01-21 22:22:23 +03:00

external_fuzz_ampt.py

Fix/hcqfuzz harnesss bug (#10923)

2025-06-23 11:22:30 +03:00

external_fuzz_hcq_signals.py

hcqfuzz: init (#10049)

2025-04-25 23:19:21 +03:00

external_fuzz_tlsf.py

Fix/hcqfuzz harnesss bug (#10923)

2025-06-23 11:22:30 +03:00

external_gpu_fail_osx.py

new openpilot compile (#6573)

2024-09-18 14:22:50 +08:00

external_hip_compiler_bug.py

CompiledASTRunner -> CompiledRunner (#4148)

2024-04-11 08:49:52 -07:00

external_jit_failure.py

fix jit realize issue (#3258)

2024-01-26 18:27:35 -08:00

external_llm_eval.py

BPE tokenizer (#11415)

2025-08-04 09:52:38 -07:00

external_metal_compile_fail.py

Use MTLCompiler directly (v2) (#7920)

2024-12-04 16:36:48 +08:00

external_model_benchmark.py

OnnxRunner file as input (#10789)

2025-07-12 14:27:46 -04:00

external_multi_gpu.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_osx_profiling.py

move dtypes to dtype.py (#2964)

2024-01-01 14:58:48 -08:00

external_test_am.py

allocate less memory in am tests (#11342)

2025-07-23 11:11:26 +03:00

external_test_amd.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_test_datasets.py

skip test TestKiTS19Dataset::test_training_set (#10936)

2025-06-23 14:27:24 -04:00

external_test_embedding.py

make embedding and GPT-2 fast (#1631)

2023-08-22 15:14:38 -07:00

external_test_example.py

refactor: ops_cloud -> ops_remote [pr] (#10166)

2025-05-05 15:59:51 -07:00

external_test_hcq_fuzz_failures.py

remove Kernel.membufs [pr] (#11200)

2025-07-12 14:48:47 -04:00

external_test_hcq.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_test_hip_compile.py

DEPRECATED_linearize -> to_program [pr] (#11198)

2025-07-12 13:46:20 -04:00

external_test_hsa_driver.py

Rename tinygrad/runtime/driver to support (#5413)

2024-07-12 11:06:42 -07:00

external_test_image.py

remove Tensor.no_grad, it's meaningless now [pr] (#10556)

2025-05-28 22:20:02 -07:00

external_test_jit_on_models.py

Pulled CLIP and UNet into Seperate Files (#5253)

2024-07-01 22:33:01 -04:00

external_test_keccak.py

Tensor.keccak("sha3_256") (#7186)

2025-06-06 15:24:05 -07:00

external_test_llama3_ff.py

work to make GEMV fast (#5824)

2024-07-30 17:41:40 -07:00

external_test_lm_head.py

isolate the 134ms kernel in train_gpt2.py (#4773)

2024-05-29 17:26:24 -04:00

external_test_losses.py

RetinaNet losses (#9536)

2025-03-21 15:52:54 -04:00

external_test_mamba.py

external that test

2024-03-29 19:35:50 -07:00

external_test_metrics.py

log_perplexity metrics (#10912)

2025-06-21 10:44:47 -04:00

external_test_mnist_data_select.py

add quick external data select test

2024-03-02 05:38:32 -08:00

external_test_nv.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

external_test_onnx_backend.py

Remove StrEnum in onnx for python 3.10 (#11345)

2025-07-23 11:52:25 -04:00

external_test_onnx_ops.py

Remove StrEnum in onnx for python 3.10 (#11345)

2025-07-23 11:52:25 -04:00

external_test_onnx_runner.py

ONNX onnx_parser and buffer_parse clean up (#11000)

2025-07-21 15:10:25 -04:00

external_test_opt.py

remove PUSH_PERMUTES from external_test_opt (#7232)

2024-10-23 00:11:34 -04:00

external_test_optim.py

CosineAnnealingLRWithWarmup (#10981)

2025-06-25 17:45:21 -04:00

external_test_simple_tokenizer.py

BPE tokenizer (#11415)

2025-08-04 09:52:38 -07:00

external_test_speed_llama.py

Make dev a property of Allocator (#10286)

2025-05-13 17:01:01 -07:00

external_test_speed_theoretical.py

test flops (and allow wide ALU in UOps) [run_process_replay] (#5749)

2024-07-26 21:07:28 -07:00

external_test_tlsf.py

factor out generic MemoryManager (#10910)

2025-06-21 16:18:33 +03:00

external_test_train_gpt2.py

move kernel to opt (#10899)

2025-06-20 15:22:28 -07:00

external_test_usb_asm24.py

usbgpu: 10 steps in benchmark to hit cache (#10273)

2025-05-13 17:06:50 +03:00

external_test_whisper_librispeech.py

names shadowing builtins (#5179)

2024-06-27 08:15:01 -04:00

external_test_yolo.py

move to new cached fetch (#2493)

2023-11-28 17:36:55 -08:00

external_test_yolov8.py

More yolo processing in tinygrad (#9928)

2025-04-24 16:21:46 -04:00

external_uop_gc.py

flip order of get_program args (#10905)

2025-06-20 17:23:23 -07:00

fuzz_fast_idiv.py

move optional out of devectorize [pr] (#11350)

2025-07-23 11:26:05 -07:00

fuzz_graph.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

fuzz_kfd.py

unify to HWQueue [pr] (#7812)

2024-11-21 10:33:08 +08:00

fuzz_linearizer.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

fuzz_shape_ops.py

move fuzz_shape_ops to run with other fuzzer (#10767)

2025-06-10 17:43:04 -04:00

fuzz_shapetracker_math.py

more typing cleanups [pr] (#8376)

2024-12-22 05:21:03 -05:00

fuzz_shapetracker_size.py

limit real_size to the size of first View of ShapeTracker (#8628)

2025-01-16 16:27:39 -05:00

fuzz_shapetracker.py

[bounty] Stride is flip (#8792)

2025-01-31 11:34:10 +09:00

fuzz_symbolic.py

[bounty] Z3 symbolic fuzzer [pr] (#10514)

2025-05-28 16:28:37 -04:00

graph_batchnorm.py

with Tensor.train() (#1935)

2023-09-28 18:02:31 -07:00

speed_beam_v_hcopt.py

move kernel to opt (#10899)

2025-06-20 15:22:28 -07:00

speed_compare_amd_am.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

speed_compare_cuda_nv.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

speed_compare_cuda_ptx.py

remove Kernel.get_program [pr] (#11203)

2025-07-12 20:50:29 -04:00

speed_v_theoretical.py

lower test_gemm_8192 on red (#10883)

2025-06-19 10:01:25 -04:00

verify_kernel.py

move kernel to opt (#10899)

2025-06-20 15:22:28 -07:00