github/tinygrad
1
1
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-01-30 17:28:24 -05:00
Code Issues Packages Projects Releases Wiki Activity

layernorm backward is indepedent of its mean (#4806)

Browse Source
This commit is contained in:
chenyu
2024-06-03 09:49:59 -04:00
committed by GitHub
parent 2dae657415
commit 3cc6ae0d85
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tinygrad/tensor.py
View File

@@ -2590,7 +2590,7 @@ class Tensor:
print(t.mean().item(), t.std().item())
```
"""
y = (self - self.mean(axis, keepdim=True))
y = (self - self.detach().mean(axis, keepdim=True))
return y.mul((y*y).mean(axis, keepdim=True).add(eps).rsqrt())
def batchnorm(self, weight:Optional[Tensor], bias:Optional[Tensor], mean:Tensor, invstd:Tensor, axis:Union[int,Tuple[int,...]]=1) -> Tensor: