George Hotz b9eb5b5d49 clean up the LLM tokenizer (#12653) ...

* clean up the LLM tokenizer

* simple tokenizer is actually simple

* ugh write good code

2025-10-14 14:22:01 +08:00

..

mlperf_bert

delete SAVE_SCHEDULE=1 [pr] (#7087)

2024-10-16 07:13:20 +03:00

mlperf_resnet

PolynomialDecayWithWarmup + tests (#3649)

2024-03-07 18:53:36 -05:00

mlperf_retinanet

RetinaNet losses (#9536)

2025-03-21 15:52:54 -04:00

mlperf_stable_diffusion

Training loop for Stable Diffusion mlperf (#12315)

2025-10-03 02:45:38 -04:00

mlperf_unet3d

[MLPerf] UNet3D dataloader (#4343)

2024-04-28 22:34:18 -04:00

openpilot

fix failed threefry (#10646)

2025-06-05 17:17:42 -07:00

process_replay

process_replay for get_rangeify_map (#12624)

2025-10-12 15:14:40 +03:00

sglang_llama

refactor test/external/external_llama_eval.py (#10567)

2025-06-23 10:43:20 -07:00

external_benchmark_am.py

clean: becnhmark -> benchmark (#10620)

2025-06-03 19:28:18 -07:00

external_benchmark_bert_matmuls.py

ShapeTracker.real_strides -> is_expanded [pr] (#12579)

2025-10-09 22:52:45 -04:00

external_benchmark_bert_softmax.py

update benchmark bert scripts with BS and ACC_DTYPE (#9826)

2025-04-10 02:06:02 -04:00

external_benchmark_disk_raw.py

ci: add h machines (#11416)

2025-07-29 19:21:51 +03:00

external_benchmark_hcopt.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

external_benchmark_hip_compile.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_benchmark_keccak.py

Tensor.keccak("sha3_256") (#7186)

2025-06-06 15:24:05 -07:00

external_benchmark_kernel_launch.py

cleanup tests, bump caches (#11746)

2025-08-19 21:21:07 -07:00

external_benchmark_llama_schedule.py

bump line count to 14500

2025-06-23 15:32:27 -07:00

external_benchmark_load_stable_diffusion.py

gate METAL_FAST_LOAD

2023-12-01 15:28:40 -08:00

external_benchmark_multitensor_allreduce.py

add some options to allreduce bench (#9348)

2025-03-04 23:46:36 +03:00

external_benchmark_openpilot.py

move ASSERT_MIN_STEP_TIME to compile3 (#12535)

2025-10-08 22:16:59 -04:00

external_benchmark_resnet.py

delete SAVE_SCHEDULE=1 [pr] (#7087)

2024-10-16 07:13:20 +03:00

external_benchmark_schedule.py

more test cleanups (#11544)

2025-08-06 15:05:21 -07:00

external_benchmark_sdxl_softmax.py

update Kernel API in tests + move optimize_local_size (#11907)

2025-08-28 15:12:47 -07:00

external_cl_half_max.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_debug_metal_sd_conv.py

update Kernel API in tests + move optimize_local_size (#11907)

2025-08-28 15:12:47 -07:00

external_fuzz_am_interrupts.py

am: recover from any boot interrupt (#8703)

2025-01-21 22:22:23 +03:00

external_fuzz_ampt.py

Fix/hcqfuzz harnesss bug (#10923)

2025-06-23 11:22:30 +03:00

external_fuzz_hcq_signals.py

hcqfuzz: init (#10049)

2025-04-25 23:19:21 +03:00

external_fuzz_tlsf.py

Fix/hcqfuzz harnesss bug (#10923)

2025-06-23 11:22:30 +03:00

external_gpu_fail_osx.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_hip_compiler_bug.py

CompiledASTRunner -> CompiledRunner (#4148)

2024-04-11 08:49:52 -07:00

external_jit_failure.py

fix jit realize issue (#3258)

2024-01-26 18:27:35 -08:00

external_llm_eval.py

BPE tokenizer (#11415)

2025-08-04 09:52:38 -07:00

external_metal_compile_fail.py

Use MTLCompiler directly (v2) (#7920)

2024-12-04 16:36:48 +08:00

external_metal_compile_slow.py

add Scheduler to replace Kernel with POSTOPT=2 (#11924)

2025-09-03 19:23:30 -07:00

external_model_benchmark.py

move frontend dir to nn [pr] (#12470)

2025-10-07 10:42:22 +08:00

external_multi_gpu.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_osx_profiling.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_test_am.py

allocate less memory in am tests (#11342)

2025-07-23 11:11:26 +03:00

external_test_amd.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_test_datasets.py

skip test TestKiTS19Dataset::test_training_set (#10936)

2025-06-23 14:27:24 -04:00

external_test_embedding.py

make embedding and GPT-2 fast (#1631)

2023-08-22 15:14:38 -07:00

external_test_example.py

refactor: ops_cloud -> ops_remote [pr] (#10166)

2025-05-05 15:59:51 -07:00

external_test_hcq_fuzz_failures.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_test_hcq.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

external_test_hip_compile.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

external_test_hsa_driver.py

Rename tinygrad/runtime/driver to support (#5413)

2024-07-12 11:06:42 -07:00

external_test_image.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

external_test_jit_on_models.py

Pulled CLIP and UNet into Seperate Files (#5253)

2024-07-01 22:33:01 -04:00

external_test_keccak.py

Tensor.keccak("sha3_256") (#7186)

2025-06-06 15:24:05 -07:00

external_test_llama3_ff.py

work to make GEMV fast (#5824)

2024-07-30 17:41:40 -07:00

external_test_lm_head.py

isolate the 134ms kernel in train_gpt2.py (#4773)

2024-05-29 17:26:24 -04:00

external_test_losses.py

RetinaNet losses (#9536)

2025-03-21 15:52:54 -04:00

external_test_mamba.py

external that test

2024-03-29 19:35:50 -07:00

external_test_metrics.py

log_perplexity metrics (#10912)

2025-06-21 10:44:47 -04:00

external_test_mnist_data_select.py

add quick external data select test

2024-03-02 05:38:32 -08:00

external_test_nv.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

external_test_onnx_backend.py

move frontend dir to nn [pr] (#12470)

2025-10-07 10:42:22 +08:00

external_test_onnx_ops.py

move frontend dir to nn [pr] (#12470)

2025-10-07 10:42:22 +08:00

external_test_onnx_runner.py

move frontend dir to nn [pr] (#12470)

2025-10-07 10:42:22 +08:00

external_test_opt.py

remove trivial use of RANGEIFY flag (#12550)

2025-10-09 02:29:38 -04:00

external_test_optim.py

LR scheduler for Stable Diffusion mlperf training (#12201)

2025-09-30 21:21:08 -04:00

external_test_simple_tokenizer.py

clean up the LLM tokenizer (#12653)

2025-10-14 14:22:01 +08:00

external_test_speed_llama.py

make method_cache account for compiler (#12156)

2025-09-13 17:00:11 +03:00

external_test_speed_theoretical.py

test flops (and allow wide ALU in UOps) [run_process_replay] (#5749)

2024-07-26 21:07:28 -07:00

external_test_tlsf.py

factor out generic MemoryManager (#10910)

2025-06-21 16:18:33 +03:00

external_test_train_gpt2.py

delete kernel.py (#12040)

2025-09-05 15:52:07 -07:00

external_test_usb_asm24.py

usbgpu: 10 steps in benchmark to hit cache (#10273)

2025-05-13 17:06:50 +03:00

external_test_whisper_librispeech.py

names shadowing builtins (#5179)

2024-06-27 08:15:01 -04:00

external_test_yolo.py

move to new cached fetch (#2493)

2023-11-28 17:36:55 -08:00

external_test_yolov8.py

More yolo processing in tinygrad (#9928)

2025-04-24 16:21:46 -04:00

external_uop_gc.py

cache apply_movement_op (#12609)

2025-10-11 08:53:10 +02:00

fuzz_fast_idiv.py

fix vec dtype in fast idiv (#12080)

2025-09-12 23:00:43 +02:00

fuzz_graph.py

rename lazydata to uop (#10698)

2025-06-08 08:42:22 -07:00

fuzz_kfd.py

unify to HWQueue [pr] (#7812)

2024-11-21 10:33:08 +08:00

fuzz_linearizer.py

ops_gpu -> ops_cl (#12103)

2025-09-10 15:15:48 -04:00

fuzz_shape_ops.py

move fuzz_shape_ops to run with other fuzzer (#10767)

2025-06-10 17:43:04 -04:00

fuzz_shapetracker_math.py

remove ShapeTracker.invert (#12520)

2025-10-08 18:37:34 +08:00

fuzz_shapetracker.py

[bounty] Stride is flip (#8792)

2025-01-31 11:34:10 +09:00

fuzz_symbolic.py

add ctx to z3_renderer arg (#11867)

2025-08-27 03:38:15 +02:00

graph_batchnorm.py

with Tensor.train() (#1935)

2023-09-28 18:02:31 -07:00

speed_compare_amd_am.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

speed_compare_cuda_nv.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

speed_compare_cuda_ptx.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00

speed_v_theoretical.py

lower test_gemm_8192 on red (#10883)

2025-06-19 10:01:25 -04:00

verify_kernel.py

move opt under codegen (#11569)

2025-08-07 14:19:17 -07:00