Elias Wahl 69341144ba Wikipedia preprocessing script (#4229) ...

* Preprocessing script

* short seq prob

* comments + env vars

* Add preprocessing reference. Add test

* lint fix + add eval test support

* whitespaces

* point to commit

* comment

* rename

* better comments

2024-04-23 10:28:01 -04:00

..

create_pretraining_data.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00

external_test_preprocessing_part.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00

pick_eval_samples.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00

tokenization.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00