rev1 - original test
- 1 epoch
- train ctx 1900
- I think the learning rate was way too high (2e-4)
- the service invocation syntax is ambiguous causing repeat code blocks
- it doesn't get the device name right like ever
- eval dataset was disabled

rev2 - it kinda works
- eval dataset at 10%
- 2 epochs
- train ctx 1200
- batch size 1
- learning rate linear 5e-5
- gradient accumulation steps 1
+ fixed invocation syntax
+ still repeatedly spits out code blocks but at least it closes them correctly
+ names are MUCH more accurate. will still hallucinate names that don't exist

rev3 - ok it definitely works
- 4 epochs
- batch size 2
- learning rate cosine 5e-5
+ doesn't seem like there's much difference but the loss is lower
+ still halluncinates device names. (need to figure this one out)
+ need more examples for: garage_door, media_player, 

rev4 - got to way lower loss. it tries really hard to stop generating text
- 4 epochs
- train ctx 512
- batch size 2
- learning rate cosine 1e-4
- added system prompt and moved services block before states block

rev 4.1 - really doesn't work as well. loss dropped REALLY fast and then never got as low as rev4
- 4 epochs
- train ctx 512
- batch size 3
- learning rate cosine 1e-4
- proper pad token

rev 4.2 - yeah nah it's the pad token
- batch size 2

rev 5 - new dataset
- 3 epochs (4th epoch was overfit)
- train cx 512
- batch size 2
- learning rate cosine 1e-5
+ actually stops generating text. not at the right... place but still!
+ messing with temperature makes it generate some interesting output.

rev 5.1 - gradient accumulation test
- 3 epochs
- train cx 512
- batch size 8
- learning rate cosine 1e-5
+ very meh

rev 5.2 - learning rate test
- 3 epochs
- train cx 512
- batch size 8
- learning rate cosine 1e-4
+ higher learning rate really helped with the higher batch size
+ is able to more reliably generate the correct device name again
+ still need more examples for multi-device actions (really need room/group support in dataset)
+ need to have more variance in request format. need more informal + more formal versions

rev 5.3 - learning rate test 2
- 4 epochs
- train cx 512
- batch size 8
- learning rate cosine 6e-5
+ lower learning rate seemed to not be as effective even though it ran for longer

rev 6 - dataset revamp again
- 3 epochs
- train cx 512
- batch size 8
- learning rate cosine 1e-4
- all questions + responses are lowercase now
- ensured there are no duplicate entries in the states block
+ definitely a bit overfit
+ maybe not so overfit. able to 0 shot asking to do stuff to a christmas tree

rev 6.1 - lower train rate
- 3 epochs
- train cx 512
- batch size 8
- learning rate cosine 6e-5
+ also definitely a bit overfit. can't generate names it hasn't seen before

rev 6.2 - fewer epochs
- 2 epochs
- train cx 512
- batch size 8
- learning rate cosine 6e-5

rev 6.3 - higher batch
- 2 epochs
- train cx 512
- batch size 12
- learning rate cosine 1e-4

rev 7 - tweak dataset again
- 2 epochs
- train ctx 512
- batch size 8
- learning rate 1e-4
+ when generating results, don't end with a space. it works WAY better

rev 7.1 - failed padding attempt

rev 7.2 - try to overfit less + no newline at end
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-4
+ it definitly works with only one epoch

rev 7.3 - try adding fake end of sentence token
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-4

rev 8 - dataset tweaks. add status requests
+ service requests still mostly work but status requests are pretty broken

rev 8.1 - tweak example counts + ratios
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-4
+ seems to have worked better with lower example counts

rev 8.2 - try to fit learning rate so loss doesn't bottom out till the end of training
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 8e-5 (didn't change loss at all)
- learning rate 5e-5 (same)
- learning rate 1e-5 (wayyyy better)
+ pretty sure i've been overcranking most of these and destroying most of the model
+ oh yuuhhhhh it's overcranked. nails both request types (plus even ending generation)
+ needs ambiguous device name examples because I totally just asked it an ambiguous question and it answered the one I wasn't expecting

rev 8.3 - further reduced training rate
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 8e-6
+ certainly not overfit like < rev7
+ has some creativity with how it repsonds
+ will often get the device name wrong on the first try

rev 8.4 - re-ordered prompt
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 8e-6
- put actions before response and also made actions it's own "block"
+ it *works* but is incredibly open ended
+ basically never stops generating text

rev 8.5 - tweaked prompt format again
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 8e-6
- re-orderd response before actions again but made actions less like a "block" so it might stop generation
+ that worked rather badly

rev 8.6 - make prompt look more like other examples it has seen before
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 8e-6
- change ```done to just ``` and add 3 newlines at the end (idk it keeps doing that for other prompts before stopping)
+ it wants to generate the other prompt types much more with this config
+ only get the correct response about 50% of the time
+ it totally stops correctly when it DOES work

rev 8.7 - try to fit a bit more. the last iteration jumps around on which format it chooses
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-5
+ similar issues as last model
+ altering the format (with newlines) makes it pick our format more often
+ comparing to 8.6 with modified format shows this one is better at getting device names right

rev 8.8 - train with newlines instead of spaces in requets/response
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-5
+ definitely worse than the previous one
+ for some reason both 8.7 and 8.8 are horrible when using their actual template but if you deviate slightly it works a lot better on inference

rev 8.9 - actually fix pad token
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-5
+ properly generates a response (+ terminates) when using the actual template

rev 9 - reduced dataset size
- 1 epoch
- train ctx 512
- batch size 8
- learning rate 1e-5
+ didn't work as well
+ would often not generate a service call
+ went back to 8.9

------

home-1b-rev1
- 1 epoch
- 2048 train ctx
- batch size 8
- learning rate 1e-5
- weight decay 0.1
- gradient clipping 1.0
- dataset changes:
  - updated chatml format
  - json function calling
  - included the alpaca split
+ it works OK with low temperatures
+ seems to handle the alpaca dataset not so well

home-1b-rev2