github/tinygrad
1
1
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-01-07 22:23:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add a quick start guide (#900)

Browse Source 
* feat: initial quick start guide

* fix: fix link

* feat: add note about jit

* feat: add note about load/store ops

* feat: add link to discord

* feat: add note about saving and loading models

* fix: correct code for saving and loading

* feat: overhaul docs

* fix: fix link

* feat: wording

* feat: add link to discord

* feat: contributing guidelines

* feat: make contributing section more doc focused

* feat: add link to env_vars from readme

* fix: wording

* feat: move community to bottom

* feat: showcase

* feat: linebreak

* feat: redesigned header

* feat: tweaks

* feat: tweaks

* feat: badge for lines of code

* feat: move installation instructions to repo readme

* feat: readme overhaul number 2

* feat: move visualization to quick start guide

* feat: readme 2 electric boogaloo

* fix: grammar

* fix: formatting

* feat: no ugly line

* feat: add line back

* feat: new load method

* feat: split adding accelerator docs out

* feat: showcase whisper

* feat: smaller tweaks

* feat: bring back oneliner
This commit is contained in:
wozeparrot
2023-06-04 11:51:20 -04:00
committed by GitHub
parent d429553730
commit e9c1ae3825
10 changed files with 693 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files

244
README.md
View File

@@ -1,91 +1,57 @@
<p align="center">
<img src="https://raw.githubusercontent.com/geohot/tinygrad/master/docs/logo.png">
</p>
<div align="center">
--------------------------------------------------------------------
[![logo](https://raw.githubusercontent.com/geohot/tinygrad/master/docs/logo.png)](https://tinygrad.org)
![Unit Tests](https://github.com/geohot/tinygrad/workflows/Unit%20Tests/badge.svg)
tinygrad: For something between [PyTorch](https://github.com/pytorch/pytorch) and [karpathy/micrograd](https://github.com/karpathy/micrograd). Maintained by [tiny corp](https://tinygrad.org).
[![tinygrad discord](https://discordapp.com/api/guilds/1068976834382925865/widget.png?style=banner2)](https://discord.gg/ZjZadyC7PK)
<h3>
For something in between a [pytorch](https://github.com/pytorch/pytorch) and a [karpathy/micrograd](https://github.com/karpathy/micrograd)
[Homepage](https://github.com/geohot/tinygrad) | [Documentation](/docs) | [Examples](/examples) | [Showcase](/docs/showcase.md) | [Discord](https://discord.gg/ZjZadyC7PK)
</h3>
[![GitHub Repo stars](https://img.shields.io/github/stars/geohot/tinygrad)](https://github.com/geohot/tinygrad/stargazers)
[![Unit Tests](https://github.com/geohot/tinygrad/actions/workflows/test.yml/badge.svg)](https://github.com/geohot/tinygrad/actions/workflows/test.yml)
[![Discord](https://img.shields.io/discord/1068976834382925865)](https://discord.gg/ZjZadyC7PK)
[![Lines of code](https://img.shields.io/tokei/lines/github/geohot/tinygrad)](https://github.com/geohot/tinygrad)
</div>
---
This may not be the best deep learning framework, but it is a deep learning framework.
The sub 1000 line core of it is in `tinygrad/`
Due to its extreme simplicity, it aims to be the easiest framework to add new accelerators to, with support for both inference and training.
Due to its extreme simplicity, it aims to be the easiest framework to add new accelerators to, with support for both inference and training. Support the simple basic ops, and you get SOTA [vision](https://arxiv.org/abs/1905.11946) `models/efficientnet.py` and [language](https://arxiv.org/abs/1706.03762) `models/transformer.py` models.
Eventually, we will have a [tinygrad accelerator](https://geohot.github.io/blog/jekyll/update/2021/06/13/a-breakdown-of-ai-chip-companies.html), then tinygrad will be ***fast***. But, for now, it is slow.
We are working on support for the Apple Neural Engine and the Google TPU in the `accel/` folder. Eventually, [we will build custom hardware](https://geohot.github.io/blog/jekyll/update/2021/06/13/a-breakdown-of-ai-chip-companies.html) for tinygrad, and it will be blindingly fast. Now, it is slow.
## Features
This project is maintained by [tiny corp](https://tinygrad.org/).
### LLaMA and Stable Diffusion
### Installation
tinygrad can run [LLaMA](/docs/showcase.md#llama) and [Stable Diffusion](/docs/showcase.md#stable-diffusion)!
```bash
git clone https://github.com/geohot/tinygrad.git
cd tinygrad
python3 -m pip install -e .
```
### Contributing
There's a lot of interest in tinygrad lately. Here's some guidelines for contributing:
* Bugfixes are the best and always welcome! Like [this one](https://github.com/geohot/tinygrad/pull/421/files).
* If you don't understand the code you are changing, don't change it!
* All code golf PRs will be closed, but [conceptual cleanups](https://github.com/geohot/tinygrad/pull/372/files) are great.
* Features are welcome. Though if you are adding a feature, you need to include tests.
* Improving test coverage is great, with reliable non brittle tests.
### Example
```python
from tinygrad.tensor import Tensor
x = Tensor.eye(3, requires_grad=True)
y = Tensor([[2.0,0,-2.0]], requires_grad=True)
z = y.matmul(x).sum()
z.backward()
print(x.grad.numpy()) # dz/dx
print(y.grad.numpy()) # dz/dy
```
### Same example in torch
```python
import torch
x = torch.eye(3, requires_grad=True)
y = torch.tensor([[2.0,0,-2.0]], requires_grad=True)
z = y.matmul(x).sum()
z.backward()
print(x.grad) # dz/dx
print(y.grad) # dz/dy
```
## Is tinygrad fast?
### Laziness
Try a matmul. See how, despite the style, it is fused into one kernel with the power of laziness.
```python
```sh
DEBUG=3 OPTLOCAL=1 python3 -c "from tinygrad.tensor import Tensor;
N = 1024; a, b = Tensor.randn(N, N), Tensor.randn(N, N);
c = (a.reshape(N, 1, N) * b.permute(1,0).reshape(1, N, N)).sum(axis=2);
print((c.numpy() - (a.numpy() @ b.numpy())).mean())"
```
Change to `DEBUG=4` to see the generated code.
And we can change `DEBUG` to `4` to see the generated code.
## Neural networks?
### Neural networks
It turns out, a decent autograd tensor library is 90% of what you need for neural networks. Add an optimizer (SGD, Adam, AdamW implemented) from tinygrad.nn.optim, write some boilerplate minibatching code, and you have all you need.
As it turns out, 90% of what you need for neural networks are a decent autograd/tensor library.
Throw in an optimizer, a data loader, and some compute, and you have all you need.
### Neural network example (from test/models/test_mnist.py)
#### Neural network example (from test/models/test_mnist.py)
```python
```py
from tinygrad.tensor import Tensor
import tinygrad.nn.optim as optim
@@ -100,7 +66,7 @@ class TinyBobNet:
model = TinyBobNet()
optim = optim.SGD([model.l1, model.l2], lr=0.001)
# ... and complete like pytorch, with (x,y) data
# ... complete data loader here
out = model.forward(x)
loss = out.mul(y).mean()
@@ -109,114 +75,86 @@ loss.backward()
optim.step()
```
## GPU and Accelerator Support
## Accelerators
tinygrad supports GPUs through PyOpenCL.
tinygrad already supports numerous accelerators, including:
```python
- [x] CPU
- [x] GPU (OpenCL)
- [x] C Code (Clang)
- [x] LLVM
- [x] METAL
- [x] CUDA
- [x] Triton
- [x] PyTorch
And it is easy to add more! Your accelerator of choice only needs to support a total of 20 (optionally 21) low level ops.
More information can be found in the [documentation for adding new accelerators](/docs/adding_new_accelerators.md).
## Installation
The current recommended way to install tinygrad is from source.
### From source
```sh
git clone https://github.com/geohot/tinygrad.git
cd tinygrad
python3 -m pip install -e . # or `py3 -m pip install -e .` if you are on windows
```
Don't forget the `.` at the end!
## Documentation
Documentation along with a quick start guide can be found in the [docs/](/docs) directory.
### Quick example comparing to PyTorch
```py
from tinygrad.tensor import Tensor
(Tensor.ones(4,4).gpu() + Tensor.ones(4,4).gpu()).cpu()
x = Tensor.eye(3, requires_grad=True)
y = Tensor([[2.0,0,-2.0]], requires_grad=True)
z = y.matmul(x).sum()
z.backward()
print(x.grad.numpy()) # dz/dx
print(y.grad.numpy()) # dz/dy
```
### hlops (in tensor.py)
The same thing but in PyTorch:
```py
import torch
hlops are syntactic sugar around mlops. They support most things torch does.
x = torch.eye(3, requires_grad=True)
y = torch.tensor([[2.0,0,-2.0]], requires_grad=True)
z = y.matmul(x).sum()
z.backward()
### mlops
mlops are mid level ops. They understand derivatives. They are very simple.
```
Relu, Log, Exp, Sin # unary ops
Sum, Max # reduce ops (with axis argument)
Maximum, Add, Sub, Mul, Pow, Div, Equal # binary ops (no broadcasting, use expand)
Expand, Reshape, Permute, Pad, Shrink, Flip # movement ops
print(x.grad.numpy()) # dz/dx
print(y.grad.numpy()) # dz/dy
```
You no longer need to write mlops for a new accelerator
## Contributing
### Adding an accelerator (llops)
There has been a lot of interest in tinygrad lately. Here are some basic guidelines for contributing:
The autodiff stuff is all in mlops now so you can focus on the raw operations
- Bug fixes are the best and always welcome! Like [this one](https://github.com/geohot/tinygrad/pull/421/files).
- If you don't understand the code you are changing, don't change it!
- All code golf PRs will be closed, but [conceptual cleanups](https://github.com/geohot/tinygrad/pull/372/files) are great.
- Features are welcome. Though if you are adding a feature, you need to include tests.
- Improving test coverage is great, with reliable non-brittle tests.
```
Buffer # class of memory on this device
unary_op (NOOP, EXP, LOG, CAST, SIN) # A -> A
reduce_op (SUM, MAX) # A -> B (smaller size, B has 1 in shape)
binary_op (ADD, SUB, MUL, DIV, POW, CMPEQ, MAX) # A + A -> A (all the same size)
movement_op (EXPAND, RESHAPE, PERMUTE, PAD, SHRINK, STRIDE) # A -> B (different size)
fused_op [[optional]] (MULACC) # A * A -> B
```
## ImageNet inference
Despite being tiny, tinygrad supports the full EfficientNet. Pass in a picture to discover what it is.
```bash
python3 examples/efficientnet.py https://media.istockphoto.com/photos/hen-picture-id831791190
```
Or, if you have a webcam and cv2 installed
```bash
python3 examples/efficientnet.py webcam
```
PROTIP: Set "DEBUG=2" environment variable if you want to see why it's slow.
### tinygrad supports Stable Diffusion!
You might need to download the [weight](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt) of Stable Diffusion and put it into weights/
Run `python3 examples/stable_diffusion.py`
<p align="center">
<img src="https://raw.githubusercontent.com/geohot/tinygrad/master/docs/stable_diffusion_by_tinygrad.jpg">
</p>
<p align="center">
"a horse sized cat eating a bagel"
</p>
### tinygrad supports LLaMA
After putting the weights in weights/LLaMA, you can have a chat with Stacy. She lives inside tinygrad.
```bash
python3 examples/llama.py
```
### tinygrad supports GANs
See `examples/mnist_gan.py`
<p align="center">
<img src="https://raw.githubusercontent.com/geohot/tinygrad/master/docs/mnist_by_tinygrad.jpg">
</p>
### tinygrad supports yolo
See `examples/yolov3.py`
<p align="center">
<img src="https://raw.githubusercontent.com/geohot/tinygrad/master/docs/yolo_by_tinygrad.jpg">
</p>
### Drawing Execution Graph
```bash
GRAPH=1 python3 test/models/test_mnist.py TestMNIST.test_sgd_onestep
# requires dot, outputs /tmp/net.svg
```
Additional guidelines can be found in [CONTRIBUTING.md](/CONTRIBUTING.md).
### Running tests
For more examples on how to run the full test suite please refer to the [CI workflow](.github/workflows/test.yml).
```bash
Some examples:
```sh
python3 -m pip install -e '.[testing]'
python3 -m pytest
python3 -m pytest -v -k TestTrain
python3 ./test/models/test_train.py TestTrain.test_efficientnet
```