Commit Graph

17 Commits

Author SHA1 Message Date
George Hotz
c0c2c0b041 support larger ViT models 2021-12-12 10:45:10 -08:00
George Hotz
4b7c31b5b7 break vit into it's own file 2021-11-30 11:19:22 -05:00
George Hotz
835869974c clean up vit code 2021-11-30 10:58:03 -05:00
George Hotz
535f02cc64 use sequential 2021-11-30 00:25:39 -05:00
George Hotz
de938c2d9d vit is now tested 2021-11-30 00:23:06 -05:00
George Hotz
aff810e722 unify transformer block 2021-11-29 18:58:15 -05:00
George Hotz
58ed46963e fix broadcastdot 2021-11-29 18:54:57 -05:00
George Hotz
125e74293f promote layernorm to tensor op 2021-11-29 18:08:21 -05:00
George Hotz
dca076dbf1 remove dumb nn ops 2021-11-29 18:05:31 -05:00
George Hotz
70544e7e9f sum hook override 2021-11-29 17:14:24 -05:00
George Hotz
8097b8f7d6 vit works 2021-11-29 16:28:14 -05:00
George Hotz
7c07c5efdd plz fix vit 2021-11-29 15:45:19 -05:00
George Hotz
ca160504e1 affine is always the last dim 2021-11-29 15:22:49 -05:00
George Hotz
e86f7a4aa3 deterministic 2021-11-29 15:10:15 -05:00
George Hotz
f909ab194f gelu with broken test 2021-11-29 15:00:50 -05:00
George Hotz
9ce881f88c fix bug in getitem, drop int axis 2021-11-29 14:01:24 -05:00
George Hotz
99b6051467 add ff_dim to transformer 2021-11-29 12:40:52 -05:00