github/tinygrad
1
1
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-02-04 03:35:16 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,007 Commits 213 Branches 14 Tags
289fd2e940145056bb98bf0643dd4627f0ddd705
Commit Graph

56 Commits

Author SHA1 Message Date
Elias Wahl
73bddc44f6 Fix fake dataloader (#5326) 2024-07-08 09:07:44 -04:00
Elias Wahl
e267f3161d Add MLLogger (#5125) 
* add MLPerf logger

* eval steps

* start with step 1

* compliance for 3.1.0 and 4.0.0

* more compliance

* assert, comment and contiguous
 2024-06-26 12:23:56 -04:00
Elias Wahl
f31ef11537 Better default hparams for large BS (#5030) 
* better default hparams for large BS

* bf16 too

* use tuple
 2024-06-18 11:13:06 -04:00
Elias Wahl
7bfa9101c0 Float in scaled dot product attention (#4985) 
* Monkeypatch scaled-dot-product-attention

* Use dot instead of matmul

* new api

* imports

* least_upper_dtype
 2024-06-18 08:16:41 -04:00
Elias Wahl
d2e3c391e8 Residual in MLM loss + Change default steps (#4935) 
* Residual in mlm loss

* Reduce default steps to 160K * 24

* oops

* comment
 2024-06-12 16:09:18 -04:00
Elias Wahl
e576aca044 Disable dropout (#4837) 2024-06-04 18:57:26 -04:00
Elias Wahl
bb248a0dd1 Optional half matmul (#4835) 
* half linear

* move weight cast back

* oops

* matmul dtype var

* todo comment
 2024-06-04 17:53:41 -04:00
Elias Wahl
04e237328b Refactor to class style (#4804) 2024-06-04 14:08:31 -07:00
Elias Wahl
c4b0acf095 Global norm + small changes (#4749) 
* norm

* no empty

* default loss scaler in float
 2024-05-27 18:35:27 -04:00
Elias Wahl
acc0039cfc Resume fix + scheduler for non weight decay params (#4679) 
* move ckpt dir

* fix resume. Add scheduler group
 2024-05-21 19:38:13 -04:00
Elias Wahl
993091adfa loss scaler + nan fixes (#4661) 2024-05-20 17:08:35 -04:00
chenyu
bed70b130c mlperf bert getenv-able EVAL_STEP_FREQ (#4534) 2024-05-11 14:36:56 -04:00
chenyu
04a4980a51 touchup bert script (#4531) 
small adjustments, remove duplicated training setting and stop the script once target is hit
 2024-05-11 13:02:02 -04:00
chenyu
b00b6b16f0 fix TRAIN_BEAM and Tensor.training for mlperf bert (#4525) 
also hard coded bert model config instead of looking up a file
 2024-05-11 00:18:36 -04:00
chenyu
b399d98e41 fix resnet eval (#4507) 2024-05-10 00:49:00 -04:00
chenyu
0e8aa0e288 use fake data in beam searching resnet (#4504) 2024-05-09 23:43:50 -04:00
chenyu
047c7f3e5b polish resnet mlperf logging (#4490) 
don't include save final check point time in run time, and some cosmetic order changes
 2024-05-09 13:04:24 -04:00
chenyu
d78e159aa3 resnet logging move RUN_START to start of the script (#4488) 2024-05-09 12:32:32 -04:00
chenyu
1f6bf9d2f7 real diskcache_clear in model_train resnet (#4445) 
clear cache if INITMLPERF is set, or running run_and_time. dev_beam and dev_run do not clear cache
 2024-05-08 19:06:09 -04:00
chenyu
1b4645bea6 hotfix resnet move init_start to start of the script (#4481) 2024-05-08 19:03:52 -04:00
chenyu
db7e15c46f hotfix resnet only log epoch start with RUNMLPERF (#4477) 2024-05-08 15:14:41 -04:00
chenyu
062c6dd65d mlperf logging, truncate dir in logs and log seed (#4475) 2024-05-08 12:54:02 -04:00
chenyu
b62a65b617 redo faster sparse_categorical_crossentropy (#4461) 
update LR and DECAY in resnet default that help convergence too
 2024-05-08 11:21:43 -04:00
wozeparrot
603d3a351b feat: allow keeping multiple cookies (#4440) 2024-05-05 19:26:48 -07:00
David Hou
b767d59684 resnet trainer: keep old cookie around until next step has been queued (#4401) 
* keep old cookie around until next step has been queued (-10ms 6gpu)

* also for eval

* drop cookie before data_get?

* Revert "drop cookie before data_get?"

This reverts commit b01e6aa2b2.

* Revert "Revert "drop cookie before data_get?""

This reverts commit 23464e73d4.
 2024-05-03 12:15:21 -04:00
chenyu
2c3b7f8e70 pad resnet training data with training data mean (#4369) 
update model_train resnet to pad training
 2024-05-02 20:26:15 -04:00
chenyu
ab01a9433d resnet eval 4n+3 if epoch < 33 (#4391) 
the rule is as thoroughly as 4n+k and we can stop the clock as soon as eval hits target. this can save 24 evals or 12 minutes
 2024-05-02 16:52:07 -04:00
chenyu
bf31837e6d resnet correct steps_in_val_epoch in logging (#4389) 
also added random seed from system in scripts
 2024-05-02 10:51:36 -04:00
chenyu
22376e53b7 resnet mlperf logging (#4361) 
* resnet mlperf logging

* cropping too much?
 2024-05-02 00:00:04 -04:00
chenyu
6628e13a5f pad resnet eval data in model_train (#4374) 
asserted if eval sample count is different from total eval file count.
 2024-05-01 14:33:42 -04:00
chenyu
826cccd54d fix mean underflow for half tensor (#4377) 
* fix mean underflow for half tensor

divide only the reduce factor. added unit test and non-nan assertion in resnet training. also added a failed test cast for symbolic shape var

* skip for python backend
 2024-05-01 13:38:57 -04:00
Elias Wahl
babe87a8ae BERT: Checkpoint loading tests (#4359) 
* Move checkpoint init to helpers. Add test

* linters

* Move the steps outside of the main train loop

* Move data_get

* data_get belongs to helpers
 2024-04-30 14:43:41 -04:00
Elias Wahl
71ff68b445 dropout after eval step (#4351) 2024-04-29 15:47:21 -04:00
Elias Wahl
27613dd881 MLPerf BERT: Main training loop (#4288) 
* BERT language modeling head + trunc normal initializers

* add train loop + helpers

* shuffle in dataloaders + slight changes in main loop

* beam change

* Minor changes

* random.shuffle

* HParam update

* Use deque for dataloader

* wandb bert project name

* half fixes

* BENCHMARK + remove epoch

* cast + print()

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2024-04-29 14:35:27 -04:00
chenyu
ec65aea32f resnet stop the script once hit target (#4303) 
* resnet stop the script once hit target

* comment
 2024-04-25 23:54:56 -04:00
chenyu
f9a7badace use LR=7 for resnet with BS=1536 (#4299) 
had 3 runs after lr float32, seems quite stable and converges at epoch 34 and 35
 2024-04-25 15:23:10 -04:00
chenyu
c1fbacb182 resnet benchmarks use DEFAULT_FLOAT=HALF (#4285) 
also update LR default to scaled based on 1536 (the BS we are submitting)
 2024-04-24 12:10:57 -04:00
chenyu
8401de9922 resnet benchmark return early in eval (#4278) 
only do few eval steps to compile, and skip second epoch when doing beam + benchmark. save 2 minutes
 2024-04-24 00:55:01 -04:00
chenyu
6637ecc5fe use IGNORE_JIT_FIRST_BEAM to not BEAM in jit cnt=0 (#4269) 
we want to have different BEAM values for resnet train and eval. global JITBEAM cannot do this. added the flag to change beam behavior at cnt=0 (so it default behaves the same with or without TinyJit), and for cnt=1 it uses existing BEAM.value.

Also updated the context var BEAM in resnet to be outside of TinyJit. saves about 3 minutes compile time
 2024-04-23 18:59:43 -04:00
chenyu
37f8be6450 resnet print epoch ops and mem in benchmark (#4244) 
* resnet print epoch ops and mem in benchmark

also added a flag to optionally disable reset jitted steps

* real per epoch stats
 2024-04-21 18:32:31 -04:00
chenyu
f7416916df update resnet hparams based on BS=1632 RCP (#4210) 
https://github.com/mlcommons/logging/blob/master/mlperf_logging/rcp_checker/training_4.0.0/rcps_resnet.json
 2024-04-18 12:01:46 -04:00
chenyu
d5b67c1ca3 log resnet TRAIN_BEAM / EVAL_BEAM (#4181) 
also run eval in benchmark mode if either one is positive
 2024-04-15 19:29:08 -04:00
chenyu
6a2168e698 TRAIN_BEAM and EVAL_BEAM for resnet (#4177) 
working on measuring compile time
 2024-04-15 14:57:21 -04:00
chenyu
e20d6f9221 correct resnet estimate time (#4169) 
7.99 hours was rendered as 7h0m.
 2024-04-14 02:21:46 -04:00
George Hotz
97c402d69e use imagenet spawn (#4096) 2024-04-06 08:34:10 -07:00
George Hotz
fffd9b05f5 mock mnist data for imagenet trainer (#4095) 
* mock mnist data for imagenet

* move print and test

* needed to reshape
 2024-04-06 08:08:40 -07:00
George Hotz
93824e59eb support MOCKDATA=1 for resnet (#4090) 
* mockdata for resnet

* fix eval, revert hsa
 2024-04-05 17:19:18 -07:00
chenyu
ecf38f498e beam search resnet eval too in BENCHMARK (#4000) 2024-03-29 21:07:23 -04:00
David Hou
4b95350c41 fp16 resnet (without expand backwards sum in float, doesn't work) (#3816) 
* fp16 resnet

* cast running mean and var back to default float

* extra cast

* check symbolic no overflow

* add linearizer failure

* loss scaler after grad contig

* oops

* i think this works

* don't loss scale fp32

* remove overflow test case

* remove symbolic bounds check

* loss scaler should be float

* temporarily disable padto cuz bug

shruggie

* make running stats in batchnorm float32?

* calculate lars stuff in fp32?

* oops

* remove most changes

* move loss scaler out of optimizer

* no more FP16 var

* oops

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2024-03-28 01:25:37 -04:00
wozeparrot
9a9cac58f9 add lars to nn (#3750) 
* feat: add lars

* feat: don't remove this comment

* clean: smaller diff

* clean: shorter line

* feat: remove mlperf lars, switch resnet

* fix: fully remove mlperf lars

* clean: comment

* feat: contiguous

* feat: no weight decay on skip params

* feat: optimizergroup

* feat: classic momentum

* fix: pylint

* clean: move comment

* fix: correct algo

* feat: lrschedulergroup

* feat: skip list tests

* feat: :| forgot that params are a thing

* feat: remove skip_list params from main params

* feat: set moment

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2024-03-24 11:43:12 -04:00