github/autogen
1
1
Fork 0
mirror of https://github.com/microsoft/autogen.git synced 2026-04-20 03:02:16 -04:00
Code Issues Packages Projects Releases Wiki Activity

fixing roberta add_prefix_space bug (#546)

Browse Source 
* fixing roberta add_prefix_space bug
This commit is contained in:
Xueqing Liu
2022-05-12 10:57:25 -04:00
committed by GitHub
parent 2a8decdc50
commit 2ca9e41e4b
1 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
flaml/model.py
View File

@@ -542,7 +542,13 @@ class TransformersEstimator(BaseEstimator):
)
else:
return AutoTokenizer.from_pretrained(
self._training_args.model_path, use_fast=True
self._training_args.model_path,
use_fast=True,
add_prefix_space=True
if "roberta" in self._training_args.model_path
else False, # If roberta model, must set add_prefix_space to True to avoid the assertion error at
# https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/tokenization_roberta_fast.py#L249
)
@property