要求截断为 max_length 但未提供最大长度，并且模型没有预定义的最大长度。默认不截断

Question

我正在按照 HuggingFace 的序列分类教程学习 NLP https://huggingface.co/transformers/custom_datasets.html#sequence-classification-with-imdb-reviews 原始代码运行没有问题。但是，当我尝试加载不同的标记生成器（例如来自 的标记生成器）时google/bert_uncased_L-4_H-256_A-4，会出现以下警告：

Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

from transformers import AutoTokenizer
from pathlib import Path

def read_imdb_split(split_dir):
    split_dir = Path(split_dir)
    texts = []
    labels = []
    for label_dir in ["pos", "neg"]:
        for text_file in (split_dir/label_dir).iterdir():
            texts.append(text_file.read_text())
            labels.append(0 if label_dir is "neg" else 1)

    return texts[:50], labels[:50]

if __name__ == '__main__':
    test_texts, test_labels = read_imdb_split('aclImdb/test')
    tokenizer = AutoTokenizer.from_pretrained('google/bert_uncased_L-4_H-256_A-4')
    test_encodings = tokenizer(test_texts, truncation=True, padding=True)
    for input_id in test_encodings["input_ids"]:
        print(len(input_id))

输出显示所有input_id的len = 1288。看起来它们都被填充到1288。但是我如何指定截断目标长度，例如512？

Answer 1

指定model_max_length加载分词器的时间。

tokenizer = AutoTokenizer.from_pretrained('google/bert_uncased_L-4_H-256_A-4', model_max_length=512)