Huggingface 分词器中的填充如何工作？

Question

我尝试了以下标记化示例：

tokenizer = BertTokenizer.from_pretrained(MODEL_TYPE, do_lower_case=True)
sent = "I hate this. Not that.",        
_tokenized = tokenizer(sent, padding=True, max_length=20, truncation=True)
print(_tknzr.decode(_tokenized['input_ids'][0]))
print(len(_tokenized['input_ids'][0]))

输出是：

[CLS] i hate this. not that. [SEP]
9

tokenizer注意:的参数max_length=20。如何让 Bert 分词器将 11 个[PAD]分词附加到这句话中以使其总计20？

Answer 1

应该设置padding="max_length"：

_tokenized = tokenizer(sent, padding="max_length", max_length=20, truncation=True)