Commit Graph

11106 Commits

Author SHA1 Message Date
George Hotz
2e89e75dcb layernorm fixes transformer instability 2020-12-28 12:58:15 -05:00
George Hotz
628d21f899 doc touchup 2020-12-28 10:45:26 -05:00
George Hotz
fafece9db7 avgpool2d is a second class op 2020-12-28 10:41:59 -05:00
George Hotz
593233b668 log and exp are first class ops 2020-12-28 10:00:30 -05:00
Marcel Bischoff
ffff98db78 Evaluation in Transformers (#218)
* 2serious

* load/save

* fixing GPU

* added DEBUG

* needs BatchNorm or doesn't learn anything

* old file not needed

* added conv biases

* added extra/training.py and checkpoint

* assert in test only

* save

* padding

* num_classes

* checkpoint

* checkpoints for padding

* training was broken

* merge

* rotation augmentation

* more aug

* needs testing

* streamline augment, augment is fast thus bicubic

* tidying up

* transformer eval
2020-12-28 09:24:51 -05:00
George Hotz
65b07d2f4f fix onehot embed 2020-12-27 18:50:38 -05:00
George Hotz
d864e1c71a transformer is training 2020-12-27 18:46:32 -05:00
George Hotz
a361ef6861 fixup training loop 2020-12-27 18:35:56 -05:00
George Hotz
f15bec6dbc make multidot work on CPU 2020-12-27 17:25:37 -05:00
George Hotz
131e04c90c cpu only decorator 2020-12-27 17:18:55 -05:00
George Hotz
2f1b2c0a3b add transpose, start on transformer 2020-12-27 16:59:12 -05:00
gamwe6
d379502c04 Cleaning (#211)
* Cleaned

* Brought the lines into line

Co-authored-by: gamwe6 <gamwe6@users.noreply.github.com>
2020-12-27 09:58:51 -05:00
George Hotz
8a335f03ad clock speed 32x32 2020-12-22 18:18:52 -05:00
George Hotz
aae2e35208 benchmarking 512x512 GEMM 2020-12-22 18:01:36 -05:00
George Hotz
bd18e03138 conv from weights works 2020-12-22 17:42:17 -05:00
George Hotz
b3cf53e39b more docs 2020-12-22 17:14:38 -05:00
George Hotz
4065eae0fb docs for tensor stride 2020-12-22 17:06:36 -05:00
George Hotz
6fb127d5c7 l2 cache note 2020-12-22 16:48:19 -05:00
George Hotz
78a06a1285 more readme 2020-12-22 16:23:08 -05:00
George Hotz
0ab951f21c better readme 2020-12-22 15:57:33 -05:00
George Hotz
6ca449afd2 sum works 2020-12-22 12:53:20 -05:00
George Hotz
ebc7f8305c 3x3 gemm in conv 2020-12-22 12:00:44 -05:00
George Hotz
ff52f832dc conv hwx 2020-12-22 11:49:54 -05:00
George Hotz
5b93d9c03a aneregs 2020-12-22 10:41:12 -05:00
George Hotz
ccfd03e401 parse ane struct 2020-12-22 10:17:06 -05:00
Nicklas Boman
06f359baa3 issue-193 - Move torch loader out of efficientnet code (#213) 2020-12-22 00:19:16 -05:00
George Hotz
3abda29648 add debug to ane lib 2020-12-21 18:49:33 -05:00
George Hotz
c48917fdf0 ane reg notes 2020-12-21 15:26:52 -05:00
George Hotz
451e944648 aneregs from debug 2020-12-21 13:37:02 -05:00
George Hotz
16f0cb7d80 fix sign python 2020-12-20 15:44:10 -05:00
George Hotz
e03759f01d channels concat 2020-12-19 09:43:21 -08:00
George Hotz
73ae6d7821 working on gemm 2020-12-19 09:30:29 -08:00
Liam
075cf4bc02 Update examples to new API (#205) 2020-12-19 09:04:12 -08:00
George Hotz
3c9150a9f4 lil note 2020-12-18 09:54:20 -08:00
George Hotz
be41e73ad9 ane docs continue 2020-12-18 09:49:13 -08:00
George Hotz
e5f2055776 ane refactor, fix examples 2020-12-18 09:19:06 -08:00
George Hotz
fbcd1912cf ane struct 2020-12-18 09:06:25 -08:00
iainwo
56d44637f3 fixed pylint, formatted python files iwth cblack on localhost (#204)
* fixed pylint, formatted python files iwth cblack on localhost

* Revert "fixed pylint, formatted python files iwth cblack on localhost"

This reverts commit 07e2b88466.

* dedented 4-spaces added linter

Co-authored-by: Iain Wong <iainwong@outlook.com>
2020-12-17 14:37:31 -08:00
iainwo
799ad5ba17 refactor/softmax (#201)
* generalized logsoftmax and sigmoid with softmax

* reverted sigmoid impl

Co-authored-by: Iain Wong <iainwong@outlook.com>
2020-12-17 09:53:28 -08:00
gamwe6
dad061dafb Added Python 3 style super() without arguments (#200)
Co-authored-by: gamwe6 <gamwe6@users.noreply.github.com>
2020-12-16 20:50:16 -08:00
Liam
bcf1518309 All devices are equal! (#196)
* Update all devices to be tested

ANE, CPU and OCL all now support all tests.

However tests are not currently passing on GPU and I cannot test on CPU.

Failing GPU test are not an issue caused by this update. Tests have not
been passing due to a missing "six" required installation.

OpenCL Tests have not been run since commit: 1a1c63a08b

devices have 3 types and are handle by a new DeviceTypes enum. (The goal
is to revert to Tensor.<type>, but this current setup allows for keyword
argument defaults: `device=DeviceType.CPU`)

All references to Tensor.GPU/CPU/ANE as been converted to the
corresponding `DeviceTypes` enum.

Refactor of the conversion code to allow for any device to any device
conversion.

* Add six dependency in requirements.txt

* Resolve failure to run tests

Move six into gpu required installs. Remove six from standard
installation.

* Remove repeated data conversion

* Refactor method names

Also reduce code with .to and .to_

* Dynamic device handlers

* Refactor DeviceTypes -> Device

* Add mem copy profiling back

* test_backward_pass_diamond_model passing

* Resolve Sum issue on GPU

* Revert batchnorm2d tests

* Update README with upadated API

* ANE testing with

* Last minute line gains
2020-12-15 23:44:08 -08:00
James Roberts
78210b5e40 less lines (#197) 2020-12-14 13:53:00 -08:00
George Hotz
b86bbd2e72 readmes 2020-12-13 21:32:20 -08:00
Marcel Bischoff
da72a0eed4 Big MNIST model with PIL augmentation and load/save (#160)
* 2serious

* load/save

* fixing GPU

* added DEBUG

* needs BatchNorm or doesn't learn anything

* old file not needed

* added conv biases

* added extra/training.py and checkpoint

* assert in test only

* save

* padding

* num_classes

* checkpoint

* checkpoints for padding

* training was broken

* merge

* rotation augmentation

* more aug

* needs testing

* streamline augment, augment is fast thus bicubic

* tidying up
2020-12-13 20:45:55 -08:00
George Hotz
f50dcc12ac 1k lines 2020-12-13 20:37:58 -08:00
George Hotz
4d8235d5f7 readme update 2020-12-13 20:24:33 -08:00
NeuralLink
1a1c63a08b Gan is real...Look what tiny just generated! (#192)
* mode collapse solved

* info add

* delete unnecessary imports

* readme
2020-12-13 20:23:12 -08:00
Marcel Bischoff
6785614239 tinygrad.utils to extra.utils fix in mnist_gan (#190) 2020-12-12 20:52:36 -08:00
NeuralLink
d901ef6b23 🎉 effort to generate mnist data using GAN with tinygrad. [WIP] (#166)
* 🎉 effort to generate mnist data with tinygrad.

* dropout added

* working gan

* minor bug fixes

* more bug fixes

* todo reg l2

* detach

* logsoftmax twice
2020-12-12 17:58:04 -08:00
Mufeed VH
e6a5c6c93e Added indentation linter (#187)
* Added indentation linter

* pylint package latest
2020-12-12 17:15:09 -08:00