Huggingface Transformer 问题答案置信度得分

Question

我们如何从huggingface转换器问题答案的示例代码中获取答案置信度得分？我看到管道确实返回了分数，但是下面的核心也可以返回置信度分数吗？

\n

from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering\nimport tensorflow as tf\n\ntokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")\nmodel = TFAutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")\n\ntext = r"""\n Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose\narchitectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet\xe2\x80\xa6) for Natural Language Understanding (NLU) and Natural\nLanguage Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between\nTensorFlow 2.0 and PyTorch.\n"""\n\nquestions = [\n    "How many pretrained models are available in Transformers?",\n    "What does Transformers provide?",\n    "Transformers provides interoperability between which frameworks?",\n]\n\nfor question in questions:\n    inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="tf")\n    input_ids = inputs["input_ids"].numpy()[0]\n\n    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n    answer_start_scores, answer_end_scores = model(inputs)\n\n    answer_start = tf.argmax(\n        answer_start_scores, axis=1\n    ).numpy()[0]  # Get the most likely beginning of answer with the argmax of the score\n    answer_end = (\n        tf.argmax(answer_end_scores, axis=1) + 1\n    ).numpy()[0]  # Get the most likely end of answer with the argmax of the score\n    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))\n\n    print(f"Question: {question}")\n    print(f"Answer: {answer}\\n")\n

\n

代码摘自\n https://huggingface.co/transformers/usage.html

\n

Answer 1

该分数只是应用 softmax 函数后答案开始标记答案结束标记的 logits 的乘积。请看下面的示例：\n管道输出：

\n

import torch\nfrom transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline\n\ntokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")\nmodel = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")\n\ntext = r"""\n Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose\narchitectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet\xe2\x80\xa6) for Natural Language Understanding (NLU) and Natural\nLanguage Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between\nTensorFlow 2.0 and PyTorch.\n"""\n\nquestion = "How many pretrained models are available in Transformers?"\n\nquestion_answerer = pipeline("question-answering", model = model, tokenizer= tokenizer)\n\nprint(question_answerer(question=question, context = text))\n

\n

输出：

\n

{\'score\': 0.5254509449005127, \'start\': 256, \'end\': 264, \'answer\': \'over 32+\'}\n

\n

不带管道：

\n

inputs = tokenizer(question, text, add_special_tokens=True, return_tensors="pt")\noutputs = model(**inputs)\n

\n

首先，我们创建一个掩码，每个上下文标记为 1，否则为 0（问题标记和特殊标记）。为此，我们使用batchencoding.sequence_ids方法：

\n

non_answer_tokens = [x if x in [0,1] else 0 for x in inputs.sequence_ids()]\nnon_answer_tokens = torch.tensor(non_answer_tokens, dtype=torch.bool)\nnon_answer_tokens\n

\n

输出：

\n

tensor([False, False, False, False, False, False, False, False, False, False,\n        False, False, False,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True, False])\n

\n

我们使用这个掩码将特殊标记和问题标记的 logits 设置为负无穷大，然后应用 softmax（负无穷大可以防止这些标记影响 softmax 结果）：

\n

tensor([False, False, False, False, False, False, False, False, False, False,\n        False, False, False,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n         True,  True,  True, False])\n

\n

输出：

\n

tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n         0.0000e+00, 1.0567e-04, 9.7031e-05, 1.9445e-06, 1.5849e-06, 1.2075e-07,\n         3.1704e-08, 4.7796e-06, 1.8712e-07, 6.2977e-08, 1.5481e-07, 8.0004e-08,\n         3.7896e-07, 1.6438e-07, 9.7762e-08, 1.0898e-05, 1.6518e-07, 5.6349e-08,\n         2.4848e-07, 2.1459e-07, 1.3785e-06, 1.0386e-07, 1.8803e-07, 8.1887e-08,\n         4.1088e-07, 1.5618e-07, 2.5624e-06, 1.8526e-06, 2.6710e-06, 6.8466e-08,\n         1.7953e-07, 3.6242e-07, 2.2788e-07, 2.3384e-06, 1.2147e-05, 1.6065e-07,\n         3.3257e-07, 2.6021e-07, 2.8140e-06, 1.3698e-07, 1.1066e-07, 2.8436e-06,\n         1.2171e-07, 9.9341e-07, 1.1684e-07, 6.8935e-08, 5.6335e-08, 1.3314e-07,\n         1.3038e-07, 7.9560e-07, 1.0671e-07, 9.1864e-08, 5.6394e-07, 3.0210e-08,\n         7.2176e-08, 5.4452e-08, 1.2873e-07, 9.2636e-08, 9.6012e-07, 7.8008e-08,\n         1.3124e-07, 1.3680e-06, 8.8716e-07, 8.6627e-07, 6.4750e-06, 2.5951e-07,\n         6.1648e-07, 8.7724e-07, 1.0796e-05, 2.6633e-07, 5.4644e-07, 1.7553e-07,\n         1.6015e-05, 5.0054e-07, 8.2263e-07, 2.6336e-06, 2.0743e-05, 4.0008e-07,\n         1.9330e-06, 2.0312e-04, 6.0256e-01, 3.9638e-01, 3.1568e-04, 2.2009e-05,\n         1.2485e-06, 2.4744e-06, 1.0092e-05, 3.1047e-06, 1.3597e-04, 1.5105e-06,\n         1.4960e-06, 8.1164e-08, 1.6534e-06, 4.6181e-07, 8.7354e-08, 2.2356e-07,\n         9.1145e-07, 8.8194e-06, 4.4202e-07, 1.9238e-07, 2.8077e-07, 1.4117e-05,\n         2.0613e-07, 1.2676e-06, 8.1317e-08, 2.2337e-06, 1.2399e-07, 6.1745e-08,\n         3.4725e-08, 2.7878e-07, 4.1457e-07, 0.0000e+00]],\n       grad_fn=<SoftmaxBackward>)\n

\n

现在可以使用这些概率来提取答案的开始和结束标记并计算答案分数：

\n

from torch.nn.functional import softmax\n\npotential_start = torch.where(non_answer_tokens, outputs.start_logits, torch.tensor(float(\'-inf\'),dtype=torch.float))\npotential_end = torch.where(non_answer_tokens, outputs.end_logits, torch.tensor(float(\'-inf\'),dtype=torch.float))\n\npotential_start = softmax(potential_start, dim = 1)\npotential_end = softmax(potential_end, dim = 1)\npotential_start\n

\n

输出：

\n

tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,\n         0.0000e+00, 1.0567e-04, 9.7031e-05, 1.9445e-06, 1.5849e-06, 1.2075e-07,\n         3.1704e-08, 4.7796e-06, 1.8712e-07, 6.2977e-08, 1.5481e-07, 8.0004e-08,\n         3.7896e-07, 1.6438e-07, 9.7762e-08, 1.0898e-05, 1.6518e-07, 5.6349e-08,\n         2.4848e-07, 2.1459e-07, 1.3785e-06, 1.0386e-07, 1.8803e-07, 8.1887e-08,\n         4.1088e-07, 1.5618e-07, 2.5624e-06, 1.8526e-06, 2.6710e-06, 6.8466e-08,\n         1.7953e-07, 3.6242e-07, 2.2788e-07, 2.3384e-06, 1.2147e-05, 1.6065e-07,\n         3.3257e-07, 2.6021e-07, 2.8140e-06, 1.3698e-07, 1.1066e-07, 2.8436e-06,\n         1.2171e-07, 9.9341e-07, 1.1684e-07, 6.8935e-08, 5.6335e-08, 1.3314e-07,\n         1.3038e-07, 7.9560e-07, 1.0671e-07, 9.1864e-08, 5.6394e-07, 3.0210e-08,\n         7.2176e-08, 5.4452e-08, 1.2873e-07, 9.2636e-08, 9.6012e-07, 7.8008e-08,\n         1.3124e-07, 1.3680e-06, 8.8716e-07, 8.6627e-07, 6.4750e-06, 2.5951e-07,\n         6.1648e-07, 8.7724e-07, 1.0796e-05, 2.6633e-07, 5.4644e-07, 1.7553e-07,\n         1.6015e-05, 5.0054e-07, 8.2263e-07, 2.6336e-06, 2.0743e-05, 4.0008e-07,\n         1.9330e-06, 2.0312e-04, 6.0256e-01, 3.9638e-01, 3.1568e-04, 2.2009e-05,\n         1.2485e-06, 2.4744e-06, 1.0092e-05, 3.1047e-06, 1.3597e-04, 1.5105e-06,\n         1.4960e-06, 8.1164e-08, 1.6534e-06, 4.6181e-07, 8.7354e-08, 2.2356e-07,\n         9.1145e-07, 8.8194e-06, 4.4202e-07, 1.9238e-07, 2.8077e-07, 1.4117e-05,\n         2.0613e-07, 1.2676e-06, 8.1317e-08, 2.2337e-06, 1.2399e-07, 6.1745e-08,\n         3.4725e-08, 2.7878e-07, 4.1457e-07, 0.0000e+00]],\n       grad_fn=<SoftmaxBackward>)\n

\n

PS：请记住，此答案不涵盖任何特殊情况（结束令牌在开始令牌之前）。

\n