home-llm/docs/experiment-notes-tinyllama.md

TinyLlama/TinyLlama-1.1B-Chat-v1.0

tinyhome-rev1

• full fine tune
• epochs: 1
• 2048 train ctx
• batch size 32
• learning rate 1e-5
• weight decay 0.1
• gradient clipping 1.0
• dataset size: medium

• evaluation results:
  • 100: 0.5814777327935222
  • 200: 0.7120445344129555
  • 300: 0.7616396761133604
  • 400: 0.7899797570850202
  • 500: 0.8643724696356275
  • 600: 0.8674089068825911
  • 700: 0.8780364372469636
  • 800: 0.8755060728744939
  • 900: 0.8709514170040485
  • 1000: 0.8775303643724697
  • Final: 0.8724696356275303

tinyhome-rev2

• dataset size: medium
• learning rate 2e-5

• evaluation results:
  • 100:
  • 200:
  • 300:
  • 400:
  • 500:
  • 600:
  • 700:
  • 800:
  • 900:
  • 1000:
  • Final: 0.9817813765182186