flan-t5-xxl：ValueError：需要包含卸载权重的“state_dict”或“save_folder”

Question

我尝试在我的 Mac M1 和 Google Colab 中运行 Hugging Face 中的 flan-t5-xxx 模型，两者都有相同的错误：

ValueError: Need either a state_dict or a save_folder containing offloaded weights.

模型卡中的代码：

from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")
input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

Answer 1

对于需要的人：

创建一个文件夹（例如名称 save_folder）。然后更新：

model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")

到

model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", offload_folder="save_folder")