qazal
ee245b48a9
refactor reduceop swizzling (prep for UOps.SWIZZLE) [compare_schedule] ( #6269 )
2024-08-24 18:17:19 +03:00
gswangg
3cf507ae7f
remove extra.ops and LazyOp support from Kernel ( #6267 )
...
* remove extra.ops and BufferOps
* remove extra.ops and LazyOp support in Kernel
2024-08-24 16:44:38 +03:00
qazal
ccb05d8baa
fixup neg tests [run_process_replay] ( #6268 )
2024-08-24 16:35:43 +03:00
gswangg
ea76b93814
migrate test_linearizer_dumb.py to UOp AST ( #6241 )
...
* add imports and update test_unmerged_ifs to UOp AST
* test_max_simplify_and_cancel
* test_expander_new_srcs
* test_llama_embedding
* test_unaligns_idxs
* test_unrolled_float4_align
* test_upcasted_stores_out_of_order
* remove LazyOp
* remove extra/ops and replace ReduceOps.SUM with BinaryOps.ADD
2024-08-24 16:27:29 +03:00
gswangg
e44653e25a
migrate test_linearizer_failures.py to UOp AST ( #6240 )
...
* add imports and update test_failure_1 to UOp AST
* update test_failure_2 with UOp AST
* update test_failure_3
* test_failure_5
* test_failure_6
* test_failure_7
* test_failure_8
* test_failure_9
* test_failure_10
* test_failure_11
* test_failure_12
* test_failure_12_multireduce
* uncomment skip and migrate test_failure_13
* test_failure_14
* test_failure_15
* test_failure_16
* test_failure_17
* test_failure_18
* test_failure_19
* test_failure_20
* test_failure_21
* test_failure_22
* test_failure_23
* test_failure_24
* test_failure_25
* test_failure_26
* test_failure_27
* test_failure_28
* test_failure_29
* test_failure_30
* test_failure_31
* test_failure_32
* test_failure_33
* test_failure_34
* test_failure_36
* test_failure_37
* test_failure_38
* test_update_39
* test_failure_40
* test_failure_41
* test_failure_42
* test_failure_43
* test_failure_44
* test_failure_45
* test_failure_46
* test_failure_47
* test_failure_48
* test_failure_49
* test_failure_50
* remove LazyOp
* reskip test_failure_22
* remove extra/ops
* replace ReduceOps with BinaryOps
* fixup that import
---------
Co-authored-by: qazal <qazal.software@gmail.com >
2024-08-24 16:26:58 +03:00
qazal
1b4ad982e5
share REDUCE_ALU in multi and schedule [run_process_replay] ( #6266 )
2024-08-24 16:16:38 +03:00
gswangg
1dc6040877
migrate test_search.py to UOp AST ( #6245 )
...
* add imports and update test_kernel_count with UOp AST
* test_filter_global_buffer
* remove LazyOp
* remove extra.ops and ReduceOps
---------
Co-authored-by: qazal <77887910+Qazalin@users.noreply.github.com >
2024-08-24 16:13:53 +03:00
qazal
ae23540d6e
refresh process replay schedule ref in reset.py ( #6265 )
2024-08-24 16:12:51 +03:00
gswangg
7be5eede71
migrate test_linearizer_overflows.py to UOp AST ( #6244 )
...
* add imports, remove ConstBuffer, and update test_overflow_1 with UOp AST
* test_overflow_2
* test_overflow_3
* test_overflow_4
* test_overflow_5
* test_overflow_6
* test_overflow_7
* TestLinearizerOverflowAlt::test_overflow_1
* TestLinearizerOverflowAlt::test_overflow_2
* remove LazyOp
* remove extra.ops
* remove ReduceOps
2024-08-24 16:10:29 +03:00
chenyu
943ab97d24
fix Tensor.prod for multitensor ( #6264 )
2024-08-24 08:52:24 -04:00
qazal
bcb2f1caa3
init REDUCE_AXIS with BinaryOps ( #6256 )
...
* REDUCE_AXIS arg with BinaryOps
* more work in kernel.py
fixup sops.gz
* fix TestGraphRewriteEfficiency
2024-08-24 11:28:41 +03:00
chenyu
da5cf11859
fix acc init value for MUL ( #6263 )
2024-08-23 23:19:44 -04:00
wozeparrot
a7bf20c7cd
feat: updated tinybox docs ( #6261 )
...
* feat: updated tinybox docs
* fix: grammar
2024-08-23 18:27:46 -07:00
George Hotz
26498b322e
add BEAM to external_benchmark_schedule.py
2024-08-23 18:10:46 -07:00
George Hotz
53a73038e3
hotfix: TestGraphRewriteEfficiency.test_create_many_uops
2024-08-23 15:51:57 -07:00
George Hotz
7c3ba3fa8a
improve match stats + custom early reject [run_process_replay] ( #6260 )
...
* improve match stats [run_process_replay]
* custom_early_reject
2024-08-23 15:28:57 -07:00
George Hotz
0b0a8829fb
allowed_len early stop [run_process_replay] ( #6257 )
...
* vectorize single rule [run_process_replay]
* allowed_len gate
* i mean, i guess i like the rule
* cleaner way to write that, and faster
2024-08-23 13:31:07 -07:00
George Hotz
a18744188f
more early reject [run_process_replay] ( #6254 )
...
* simple matcher in alu [run_process_replay]
* never mind, i don't like simple matcher
* allowed_len == 0 is okay sometimes
* more generic matcher
2024-08-23 12:16:44 -07:00
qazal
0d4887e9df
use UOps.WMMA everywhere ( #6255 )
...
* add UOps.WMMA_AXIS
* delete ReduceOps.WMMA from ops
2024-08-23 15:03:26 -04:00
chenyu
66d0b14a20
simpler CMPLT UOp _min_max [run_process_replay] ( #6251 )
2024-08-23 10:36:16 -04:00
chenyu
590c0922b6
Tensor.prod ( #6250 )
...
* Tensor.prod
a new reduce op!
* onnx ReduceProd
2024-08-23 10:06:32 -04:00
qazal
78d6bd8b41
start graph rewrite in the scheduler ( #6248 )
...
* start graph rewrite in the scheduler
* test: enable it
* test timings
* only fails in multi reduce
* more isolated tests
2024-08-23 13:15:55 +03:00
chenyu
75700edf73
minor bitcast touchup ( #6246 )
...
`not A == B` -> `A != B`
2024-08-22 20:25:28 -04:00
chenyu
4d40de867b
remove redundant c1-(x+c2) rule [run_process_replay] ( #6243 )
2024-08-22 16:45:49 -04:00
George Hotz
238896ca02
loooking into graph rewrite speed ( #6239 )
...
* loooking into graph rewrite speed
* track, replace is slow
* if all same, no permutations [run_process_replay]
* types so compile works
* no implied comprehension
* TRACK_MATCH_STATS=2
2024-08-22 13:17:55 -07:00
chenyu
f62c4b3b5f
remove redundant -(x*c) pattern [run_process_replay] ( #6242 )
...
covered by `x*c0*c1`
2024-08-22 16:11:02 -04:00
chenyu
e745e16441
remove UnaryOps.NEG ( #6238 )
...
* Remove UnaryOps.NEG
generated new dataset with
```
time JIT=2 PYTHONPATH=. ./extra/optimization/generate_dataset.sh
gzip /tmp/sops
mv /tmp/sops.gz extra/datasets/
```
* fix that
2024-08-22 14:21:39 -04:00
nimlgen
6c4ddd6260
hcq skip tests when no multidev ( #6235 )
...
* hcq skip tests when no multidev
* linter
* a bit higher tinout
2024-08-22 18:27:16 +03:00
chenyu
08539f08b0
fix UOp repr with Variable in arg ( #6236 )
2024-08-22 11:06:33 -04:00
chenyu
3fc8203475
remove NEG from handwritten ast in tests ( #6234 )
...
* remove NEG from handwritten ast in tests
* test_linearizer_failures
2024-08-22 09:06:59 -04:00
chenyu
1c5ef5b793
format test_linearizer_failure ( #6231 )
...
made it easier to remove NEG
2024-08-21 21:10:56 -04:00
George Hotz
5cdec79469
simpler expand without dont_expand_args [run_process_replay] ( #6230 )
...
* simpler expand without dont_expand_args [run_process_replay]
* Revert "simpler expand without dont_expand_args [run_process_replay]"
This reverts commit 81693024c097c31e601f1a199a631e9eda0d9638.
* exclude_args
* why does that fix it
* correct fix
* _swizzle_args should be fast
* add comment
* zip is tuples
2024-08-21 17:48:45 -07:00
nimlgen
78c94abe9c
raise time limit for ci in test_profile_multidev_transfer ( #6227 )
2024-08-21 22:42:03 +03:00
gswangg
c74b318458
migrate test_linearizer.py to UOp AST, pt. 2 ( #6228 )
2024-08-21 22:16:11 +03:00
George Hotz
c3168952f0
wip: tracking pattern matcher [run_process_replay] ( #6225 )
...
* wip: tracking pattern matcher
* better
* proper dedup
* timing
* early reject
* mergable match stats
* TrackedPattenMatcher
* fix TrackedPattenMatcher
* cleanups
* clean that too
* remove early_reject
* Revert "remove early_reject"
This reverts commit dc2aef14b8f5da58f5ec9566daf252513cac394c.
* total
* sort by time
* match_stats cleanup
2024-08-21 11:57:26 -07:00
chenyu
a666450e4d
UOp pattern x + x -> x * 2 ( #6224 )
...
* UOp pattern x + x -> x * 2
now there's no NEG, with this it covers all kinds of a*x+b*x
* can remove x-x
2024-08-21 12:06:19 -04:00
chenyu
c9a9631818
no UnaryOps.NEG in generated UOp patterns ( #6209 )
...
* no UnaryOps.NEG in generated UOp patterns
removed pattern `x * (-1) -> -x` and `x != True`
* those are fine because NEG became CMPNE and True
* fix sd validation L2 norm
2024-08-21 11:08:22 -04:00
qazal
3b8cc5a3e0
more multireduce tests prep for neg removal [run_process_replay] ( #6220 )
2024-08-21 12:45:24 +03:00
qazal
86c036f0d3
reorder uops.py [run_process_replay] ( #6219 )
...
* reorder uops.py [run_process_replay]
* nop spacing
2024-08-21 11:39:55 +03:00
qazal
f03e5a4b3b
test_multireduce const has a shape ( #6218 )
2024-08-21 11:02:45 +03:00
George Hotz
911bf7216c
remove unused match rules [run_process_replay] ( #6217 )
2024-08-21 00:16:04 -07:00
George Hotz
2c42e9c2c6
faster rewrite, no folder in expand/reduce [run_process_replay] ( #6216 )
...
* faster rewrite, no folder in expand/reduce [run_process_replay]
* is removing the expander there okay
* parens
* don't reconstruct exact match uop
* fast do_reduce
* expand pyint
* most of the parents gains with less lines
2024-08-20 23:36:58 -07:00
George Hotz
16f420f7a7
split full_graph_rewrite and linearize_uop [run_process_replay] ( #6215 )
...
* split full_graph_rewrite and linearize_uop
* fix tests
* graph rewrite in test uops
* add types
2024-08-20 20:12:33 -07:00
George Hotz
9faf205601
CIFAR trainer + various bugfixes / improvements ( #6146 )
...
* move cifar into datasets
* support for pathlib Tensors, tar_extract, and fetch gunzip
* too early for Device.DEFAULT
* simpler hlb_cifar + .to(None) is default
* new compiler failure, start beautiful_cifar
* beautiful cifar runs but is broken
* jit train step
* cleaner
* std_mean, not mean_std
* more correct
* fast indexing
* don't print that
* torch load broken
* add eval
* nicer bar
* decoraters are the way to do this
* bounds check the load
* a few ops
* batchnorm bugfix, if track_running_stats is False, use online estimate
* full timing
* fix fusion
* unneeded realize
* master tensor
2024-08-20 16:58:46 -07:00
George Hotz
296368f0dd
Revert "delete arg from cast [run_process_replay] ( #6202 )" ( #6214 )
...
This reverts commit ec52a09393
2024-08-20 16:45:30 -07:00
nimlgen
89c4cffd86
nv fix size in SET_SEMAPHORE_A ( #6213 )
2024-08-21 01:47:10 +03:00
qazal
ec52a09393
delete arg from cast [run_process_replay] ( #6202 )
2024-08-20 14:06:16 -07:00
Francis Lam
7376b67e36
extra/gemm/triton_nv_matmul: fix Program arguments ( #6212 )
...
remove op_estimate
2024-08-20 14:05:38 -07:00
madt2709
4bb98d8882
Fix track_running_stats in batchnorm ( #6200 )
...
* Fix track_running_stats in batchnorm
* Fix linter
* Update test_fold_conv_batchnorm_notrain to keep allowed at 1
* Add test_fold_conv_batchnorm_notrain_no_running_stats
* Save 1 line
2024-08-20 14:01:22 -07:00
George Hotz
d9c62a33c3
add cifar to datasets.py ( #6210 )
2024-08-20 11:42:49 -07:00
