该模型没有从输入中返回损失 - LabSE 错误

Question

我想使用小队数据集微调 LabSE 以进行问答。我收到这个错误： ValueError: The model did not return a loss from the inputs, only the following keys: last_hidden_state,pooler_output. For reference, the inputs it received are input_ids,token_type_ids,attention_mask.

我正在尝试使用 pytorch 微调模型。我尝试使用较小的批量大小，但只使用了 10% 的训练数据集，因为我遇到了内存分配问题。如果内存分配问题消失，则会发生此错误。老实说，我坚持了下来。你有什么提示吗？

我正在尝试使用 Huggingface 教程，但我想使用其他评估（我想自己做），所以我跳过了使用数据集的评估部分。

from datasets import load_dataset
raw_datasets = load_dataset("squad", split='train')


from transformers import BertTokenizerFast, BertModel
from transformers import AutoTokenizer


model_checkpoint = "setu4993/LaBSE"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = BertModel.from_pretrained(model_checkpoint)



max_length = 384
stride = 128


def preprocess_training_examples(examples):
    questions = [q.strip() for q in examples["question"]]
    inputs = tokenizer(
        questions,
        examples["context"],
        max_length=max_length,
        truncation="only_second",
        stride=stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

    offset_mapping = inputs.pop("offset_mapping")
    sample_map = inputs.pop("overflow_to_sample_mapping")
    answers = examples["answers"]
    start_positions = []
    end_positions = []

    for i, offset in enumerate(offset_mapping):
        sample_idx = sample_map[i]
        answer = answers[sample_idx]
        start_char = answer["answer_start"][0]
        end_char = answer["answer_start"][0] + len(answer["text"][0])
        sequence_ids = inputs.sequence_ids(i)

        # Find the start and end of the context
        idx = 0
        while sequence_ids[idx] != 1:
            idx += 1
        context_start = idx
        while sequence_ids[idx] == 1:
            idx += 1
        context_end = idx - 1

        # If the answer is not fully inside the context, label is (0, 0)
        if offset[context_start][0] > start_char or offset[context_end][1] < end_char:
            start_positions.append(0)
            end_positions.append(0)
        else:
            # Otherwise it's the start and end token positions
            idx = context_start
            while idx <= context_end and offset[idx][0] <= start_char:
                idx += 1
            start_positions.append(idx - 1)

            idx = context_end
            while idx >= context_start and offset[idx][1] >= end_char:
                idx -= 1
            end_positions.append(idx + 1)

    inputs["start_positions"] = start_positions
    inputs["end_positions"] = end_positions
    return inputs


train_dataset = raw_datasets.map(
    preprocess_training_examples,
    batched=True,
    remove_columns=raw_datasets.column_names,
)
len(raw_datasets), len(train_dataset)

from transformers import TrainingArguments

args = TrainingArguments(
    "bert-finetuned-squad",
    save_strategy="epoch",
    learning_rate=2e-5,
    num_train_epochs=3,
    weight_decay=0.01,
)

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
trainer.train()

Answer 1

你好，

请确保您熟悉以下内容：

• 您可能需要将TrainingArguments中的label_names参数与您提供的标签列或键一起传递，否则您需要知道您选择的模型接受的默认前向参数是什么

例如：使用BertForQuestionAnswering模型，在Huggingface github上 我们可以看到我们需要start_positions和end_positions作为 key/column_name，这是模型在前向传递期间接受的内容。

• 此外，从同一个链接，您需要验证Trainer中的标签/目标所需的形状（这可能与 logits 形状不同），并根据链接提供形状。

如果您或某人能够通过上述修复解决该错误，请告诉我！

谢谢！