Commit Graph

72 Commits

Author SHA1 Message Date
George Hotz
b78addf2f8 Whisper (#919)
* no whispering yet

* whispering

* live whisper

* small support
2023-06-03 18:55:14 -07:00
George Hotz
ed1963b899 Fast DiskTensor to other Tensor (#916)
* make disktensors fast

* loading

* loader for sd and llama
2023-06-03 12:25:41 -07:00
George Hotz
791530045d Refactor LoadOps (#910)
* test

* work

* upd test

* loadops

* cleanups

* real ones

* remove LazyNumpyArray

* fix assign test

* remove range

* np.require

* llama uses arange kernels

* no caching consts

* fix enet

* torch load support

* tests cleanup

* fix shufflenet

* fix image

* fix torch_load test
2023-06-03 09:40:43 -07:00
George Hotz
8a928ed2f3 nn init matches torch (#901) 2023-06-01 21:24:11 -07:00
wozeparrot
bfea5215e9 Add weight decay to SGD (#883)
* feat: add weight decay to sgd

* fix: fix tests
2023-06-01 13:13:18 -07:00
Joqsan
ef129bcb85 Zero dim Tensor support (#777)
* add and reorganize test_slice_* tests

* refactor Tensor.__getitem__()

* preliminary tests for 1) 0D tensors and 2) varargs for Tensor.zeros and Tensor.ones

* always compare shapes of the numpy arrays obtained from tinygrad and torch tensors

* add more tests for 0D support

* remove test_tensor.test_slicing(). All slicing tests at test/test_ops.py

* add zero-dim support

* make test_end2end.py consistent with 0dim support

* add test for tensor with zero in shape

* don't simplify ones if shape is ()

* skip tests that need zero-size tensor support.

- zero-size tensor support not related to 0dim tensors.

* add tests for __getitem__() supporting strides >= 1

* refactor __getitem__: support for strides >= 1

* minor refactors and add comments to __getitem__

* add tests for slices with negative steps

* add support for slices with negative strides
2023-06-01 11:32:02 -07:00
kposborne2
ae83e9844c add output_padding to transposed conv (#875) 2023-06-01 00:03:22 -07:00
Rabia Eda Yılmaz
3075988468 Added kaiming_uniform initialization for Conv2d and Linear layers (#756)
* added kaiming_uniform init for conv2d and linear layers

* fix: set getattr

* up

* fix: set getattr

* fix comments

* better does not mean it is good

* more nonlinearities

* added test

checks the distribution of default relu option

* prettier

* fix kernel size

* edit distribution of returned tensor

* complete tests and fix fan_mode

* added higher dim test

* prettier test

* fix silly blank

* just leaky_relu mode

* default fan in and leaky relu

* update params

* fix test

* shorter

* generalize Tensor.uniform and adjust kaiming init

- added low and high parameters to Tensor.uniform function, so it can have a specific range (default is 0 to 1)
- adjusted return line of kaiming_uniform

* range from -1 to 1

* delete comment

* adjusted test_uniform

* fixed

* delete comment
2023-05-29 15:09:55 -07:00
wozeparrot
8c6085a715 Rewrite Adam/W as functions of LAMB (#839)
* feat: rewrite adam/w as functions of lamb

* feat: use adam style adam update + comment

* fix: nvm need to use the lamb adam update
2023-05-29 09:21:35 -07:00
Jacky Lee
5d212864b5 Add MLPerf UNet3D model (#775)
* Add ResNet inference test and cannon

* Test with ResNet50

* test_car works with resnet fix

* Add KiTS19 dataset

* KiTS19: Implement iterate

* No batch load for this dataset

* Save results on iterate

* Implement dice score

* Add data prep and eval functions

* Resolve shape issue

* Conversion works but wrong values

* Segfaults when load_from_pretrained is called

* Fix segfault and assign properly

* Final result generated, though very slow

* Store and load final result to save time

* Fix typo in finalize

* Score computes

* More bug fixes, dice score is very low

* Working broken code

* Assign output values to result

* Getting a much higher score now

* Fix dataset preprocessing

* Mean DICE score of 88.5

* Ugh, typo

* Attempt to reimplement model

* Rename layers

* Tiny model works, kinda

* Accuracy? gone

* Implement InstanceNorm and match torch

* Test instance norm 2d and 3d

* Combined input block with downsample block

* Tiny model works, support strided convtranspose

* Commands to download dataset

* Clean up a bit

* unet3d_v2 -> unet3d

* Remove duplicated code

* Oops, put tests back
2023-05-28 20:38:19 -07:00
wozeparrot
7460bd9b02 Add LAMB optimizer (#821)
* feat: initial lamb optimizer

* feat: corrently match tf impl and add test
2023-05-28 15:09:05 -07:00
kposborne2
2163a1b049 Add shrink step to fix strided conv_transpose2d, and add to nn (#823)
* implement conv transpose 2d

* don't inherit, remove old assert

---------

Co-authored-by: Kyle <kposborne@gmail.com>
2023-05-28 07:52:45 -07:00
symlon
04284414db Batchnorm2d fixed running_var (#807)
* BatchNorm2d match pytorch

* removed comment

* Batchnorm2d test multiple sizes
2023-05-26 12:32:49 -07:00
wozeparrot
0dc333cfab Promote Embedding to nn (#798)
* feat: promote Embedding to nn

* fix: fix failing test

* feat: add test with jit

* feat: rewrite embedding to no longer need stacked for loops

* clean+fix: don't know how that happened
2023-05-25 18:39:45 -07:00
George Hotz
810f03dafa conv3d + unet3d (#772)
* conv3d, needs test

* test passes, padding wrong on unet

* unet3d

* no conv3d on images
2023-05-12 13:54:07 -07:00
George Hotz
03b38864db fix batchnorm at training (#753)
* e2e testing

* min failure

* no affine on bn, still fails

* why did i think i could detach that?

* allow more kernels for bn

* some test issue i don't understand
2023-04-19 08:01:04 -07:00
George Hotz
9fb3f9ace3 Revert "move t.grad realize on SGD"
This reverts commit ccdc0290d6.
2023-04-18 17:50:08 -07:00
George Hotz
e93e04ed6e Revert "huh...this is faster"
This reverts commit aedd4685fa.
2023-04-18 17:50:07 -07:00
George Hotz
aedd4685fa huh...this is faster 2023-04-18 17:36:31 -07:00
George Hotz
ccdc0290d6 move t.grad realize on SGD 2023-04-18 16:47:51 -07:00
George Hotz
b12b60af20 fix binop, other tests failure (#723)
* fix binop, other tests failure

* that was a bad idea

* better layernorm

* inference kernel count tests

* new style reshape pushing

* fixup replacement

* 199 kernels is okay. fix flops

* push reshape through unaryops only

* GRAPH=2 draws the phantom ops

* found resnet issue

* non working test

* mul is cheaper than div

* OPT inflation

* SHUFFLE_PAD_OPS in OPT=2
2023-03-22 18:15:07 -07:00
George Hotz
d6f4219952 LayerNorm2d for 2 lines 2023-03-20 16:58:43 -07:00
George Hotz
30b795874a remove RMSprop, nobody uses it anymore 2023-03-20 12:31:34 -07:00
George Hotz
5495c7d64e linearizer! (#714)
* linearizer outputs something

* working ish

* cstyle codegen

* clang mostly works

* fix load valid

* fix numberless loop

* fancy gen

* working

* fix enet compiler

* cleanups

* float4 upcasting

* less lines

* supports_float4

* constant folding

* mulacc

* internet tests flaky in CI

* 90% image support

* fix image generic

* bugs exposed with shapetracker and single view

* new llvm

* use vload, remove OLD

* that's really poorly done

* ending up being more lines
2023-03-19 23:43:49 -07:00
Cyril Roumégous
b629fd4cd8 add AdamW optimizer (#716)
* add AdamW optimizer

* one liner Adam optimizer
2023-03-19 12:51:06 -07:00
George Hotz
f5467cfedc Devicebufferless (#708)
* runs one metal kernel

* conv2d works

* ops tests are passing

* const folding

* all ops work

* pre commit always passes

* torch works

* working still

* fix graph test

* tests passing

* image almost works

* image conv works

* most images

* fix custom

* fix assignment

* fix compile enet

* clean up comments

* fix realize return value

* include shapetracker in LB repr

* copy should make a copy

* reenable method cache

* fix lna

* dtypes in graph

* forward only for IMAGE=2

* simple realize

* getting close

* fixup new api, it's good except the kernel count

* back to 197 kernels

* tests should pass

* go to a real float

* no type_on_cpu

* fix the docs

* put shapetracker back in it's proper place
2023-03-18 14:40:23 -07:00
George Hotz
b512edc9ff no decorators for image methods. move out RawMallocBuffer. -7 lines 2023-03-12 16:28:45 -07:00
George Hotz
ed9ab6ff03 move image to nn/image.py 2023-03-12 16:21:42 -07:00
George Hotz
58d3824cbe better get_state_dict 2023-03-12 00:10:48 -08:00
George Hotz
046b3952c3 get_state_dict 2023-03-11 23:46:53 -08:00
George Hotz
305b9f2d21 multistep optim tests passing 2023-03-11 17:49:53 -08:00
Cyril Roumégous
3f08613a2a apply flake8 E203 rule (#684) 2023-03-11 11:35:16 -08:00
George Hotz
b14d31d6db ConvNeXt + extras (#657)
* simple convnext implementation

* shorter function names

* need to realize the random functions now

* creating an optimizer realizes all params

* assign contiguous

* fix lazy lazy

* why was i doing that...add convnext to tests

* LazyNumpyArray

* enable assert + comment

* no two tiny
2023-03-06 22:10:56 -08:00
Cyril Roumégous
c10131ddf5 reduce number of lines (#645) 2023-03-05 15:42:32 -08:00
George Hotz
3c8da6bd03 add typing 2023-02-28 10:54:46 -08:00
George Hotz
643e8b0388 fix tests, test bn evaluate too 2023-02-27 10:39:47 -08:00
George Hotz
2f17d151b3 fix batchnorm not realizing 2023-02-27 10:19:54 -08:00
George Hotz
e74779f19d typing fixup 2023-02-27 09:52:04 -08:00
George Hotz
edc8fbfff2 woah, why isn't OPT=2 2023-02-27 08:03:31 -08:00
George Hotz
a52913b242 test conv shapetracker has one view 2023-02-27 07:54:47 -08:00
Jacky Lee
e172f0087a BatchNorm2D -> BatchNorm2d (#558)
* BatchNorm2D -> BatchNorm2d

* Fix typo
2023-02-16 12:31:49 -08:00
George Hotz
191c76cfd7 hlb_cifar10 torch version 2023-02-11 18:04:40 -08:00
George Hotz
9152bb5b4a momentum support in SGD 2023-02-11 10:22:37 -08:00
George Hotz
51037815b9 add comment so we don't remove self.t tensor again 2023-02-10 23:07:07 -06:00
George Hotz
c0ea538ba0 Revert "revert t as tensor, constant folding should be done better"
This reverts commit 1d800a94ad.
2023-02-10 23:06:00 -06:00
George Hotz
1d800a94ad revert t as tensor, constant folding should be done better 2023-02-10 22:58:39 -06:00
George Hotz
a5a55ac19e GlobalCounters cache + assign in optim 2023-02-08 17:10:55 -06:00
George Hotz
2e1bdc889a write out all the functions, no auto binding (#543)
* write out all the functions, no auto binding

* cleanups, more types

* Slice is for internal calls only

* improve typing

* ugh, put slice back
2023-02-08 12:41:39 -06:00
George Hotz
d854337f0d nn/optim.py compiles now 2023-02-08 11:25:18 -06:00
Andrey
4977d6f225 using tuples in isinstance (#534) 2023-02-06 14:40:26 -06:00