chenyu 5ae252ae83 use at least float32 for optim.lr (#4297) ...

* use at least float32 for optim.lr

when doing mixed precision training (float32 weight, default_float=half), still use float32 to store lr.
it would have been upcasted later in actual weight update, but would have lost precision.
this improved resnet convergence significantly

* undo type annotation

2024-04-25 14:42:28 -04:00

5.4 KiB

Raw Blame History

View Raw