使用随机权重初始化 HuggingFace Bert

Question

如何用随机权重初始化 BERT？我想在掩码语言建模任务中比较多语言、单语言和随机初始化 BERT 的性能。而在前一种情况下，它非常简单：

from transformers import BertTokenizer, BertForMaskedLM

tokenizer_multi = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
model_multi = BertForMaskedLM.from_pretrained('bert-base-multilingual-cased')
model_multi.eval()

tokenizer_mono = BertTokenizer.from_pretrained('bert-base-cased')
model_mono = BertForMaskedLM.from_pretrained('bert-base-cased')
model_mono.eval()

我不知道如何加载随机权重。

提前致谢！

Answer 1

您可以使用 Hugginface 功能初始化随机 BERT 模型（来自文档https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/bert#transformers.BertConfig）

from transformers import BertConfig, BertModel

# Initializing a BERT bert-base-uncased style configuration
configuration = BertConfig()

# Initializing a model (with random weights) from the bert-base-uncased style configuration
model = BertModel(configuration)

# Accessing the model configuration
configuration = model.config