44 Commits

Author	SHA1	Message	Date
Alex O'Connell	7b01251f5d	fixes for training zephyr base	2024-02-04 11:40:03 -05:00
Alex O'Connell	cecf9bc53e	move to jsonl, finish sharegpt dataset format, and add flag to add chatml prompt template	2024-01-31 23:00:32 -05:00
Alex O'Connell	d901eaffdf	start working on other base models	2024-01-30 22:12:46 -05:00
Alex O'Connell	8b1cc5a587	add random sampler with largest first to optimize pytorch memory usage	2024-01-30 22:12:12 -05:00
Alex O'Connell	2bdfb79f89	clear cuda cache after evaluation to clear out the irregularly sized blocks from the allocation cache	2024-01-30 22:12:12 -05:00
Alex O'Connell	da7b7b4d95	try to checkpoint saved lora modules	2024-01-30 22:12:12 -05:00
Alex O'Connell	30361809ae	count tokens faster	2024-01-27 14:54:46 -05:00
Alex O'Connell	9723a98139	more dataset + model experiments using the evaluation script	2024-01-27 14:54:14 -05:00
Alex O'Connell	e6fae06133	wizardlm merge + fix eval	2024-01-25 20:46:59 -05:00
Alex O'Connell	57634519ca	move to eval script instead of during training	2024-01-25 20:46:59 -05:00
Alex O'Connell	c3cb5c5354	try writing an accuracy metric	2024-01-25 20:46:59 -05:00
Alex O'Connell	0b9beb85ca	finish trianing new models	2024-01-17 18:24:25 -05:00
Alex O'Connell	2a0dbd8806	fix training script for Phi-2 lora	2024-01-16 21:47:15 -05:00
Alex O'Connell	8f99025507	scale dataset halfway back to where it was	2024-01-13 23:36:56 -05:00
Alex O'Connell	e479b5ffac	more fixes	2024-01-13 15:29:37 -05:00
Alex O'Connell	23dae4faa9	fix lora train error with new phi model	2024-01-12 23:24:16 -05:00
Alex O'Connell	74bd516148	resize dataset + add state attributes to status requests	2024-01-12 12:44:26 -05:00
Alex O'Connell	35e609108c	update x86 wheel	2024-01-06 19:08:19 -05:00
Alex O'Connell	ce0ec6b48f	fix chatml again + gbnf + phi1.5	2023-12-31 16:00:24 -05:00
Alex O'Connell	b1575f2512	chatml prompt tweaks, support multi-turn examples in the SFT masking step, and pre-tokenize dataset to avoid CPU bottleneck	2023-12-26 23:52:27 -05:00
Alex O'Connell	03b3bf0b20	add shebangs	2023-12-23 09:13:52 -05:00
Alex O'Connell	206463a2bd	cleanup + license + readme updates	2023-12-22 23:55:14 -05:00
Alex O'Connell	8614152751	more cleanup	2023-12-20 23:18:39 -05:00
Alex O'Connell	d429e37378	add quantization scripts	2023-12-19 21:49:09 -05:00
Alex O'Connell	b921a3234b	lots of changes from phi-2 fine-tuning	2023-12-18 00:57:07 -05:00
Alex O'Connell	5f59b0d84c	support saving more frequently	2023-12-14 00:44:26 -05:00
Alex O'Connell	2384b75583	add merge with alpaca dataset	2023-12-13 21:53:48 -05:00
Alex O'Connell	46b6ee4f65	updates with new Phi model release	2023-12-13 18:29:47 -05:00
Alex O'Connell	cb7cc1ed53	more plugin work. it passes info around properly	2023-10-29 23:22:59 -04:00
Alex O'Connell	2246586a08	turn on resize so it doesn't break	2023-10-28 00:18:18 -04:00
Alex O'Connell	ad16cfa41d	more cmd line flags + gptq support	2023-10-28 00:15:03 -04:00
Alex O'Connell	82b4ea4c65	add load in 8bit + enable lora	2023-10-27 00:23:13 -04:00
Alex O'Connell	110b988527	fix generate	2023-10-26 23:28:19 -04:00
Alex O'Connell	373fc5fe9e	smaller dataset + random sampling evaluation	2023-10-26 23:10:40 -04:00
Alex O'Connell	abd2818a2b	add lora + argument parsing to trainer	2023-10-26 21:27:54 -04:00
Alex O'Connell	7f8e2fe112	save rev 8.9	2023-10-23 14:43:19 -04:00
Alex O'Connell	575efeac41	save state for rev 8.3	2023-10-23 01:09:41 -04:00
Alex O'Connell	b60c1e8274	udpates for rev 7.3	2023-10-22 22:27:40 -04:00
Alex O'Connell	aefcdb2637	prep rev6.1	2023-10-17 00:19:14 -04:00
Alex O'Connell	40735164a7	tweaks for rev 6	2023-10-16 23:22:39 -04:00
Alex O'Connell	f265c77612	dataset tweak to remove duplicates in states block + add actual test split	2023-10-16 23:13:38 -04:00
Alex O'Connell	82b7930624	set up for rev 5.1	2023-10-16 00:01:38 -04:00
Alex O'Connell	01cef3dcd9	save settings for rev5	2023-10-15 22:58:44 -04:00
Alex O'Connell	1d9049c919	add initial setup	2023-10-14 23:30:15 -04:00