如何使用 Huggingface 变压器加载基于 llama 的微调 pef/lora 模型？

Question

我遵循本教程（colab笔记本）来微调我的模型。

尝试加载我本地保存的模型

model = AutoModelForCausalLM.from_pretrained("finetuned_model")

产量Killed.

---

尝试从集线器加载模型：

产量

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "lucas0/empath-llama-7b"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(cwd+"/tokenizer.model")

# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)

产量

AttributeError: /home/ubuntu/empath/lora/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

完整的堆栈跟踪

模型创建：

我使用 PEFT 和 LoRa 微调了模型：

model = AutoModelForCausalLM.from_pretrained(
"decapoda-research/llama-7b-hf",
torch_dtype=torch.float16,
device_map='auto',
)

我必须下载并手动指定 llama 标记器。

tokenizer = LlamaTokenizer(cwd+"/tokenizer.model")
tokenizer.pad_token = tokenizer.eos_token

参加培训：

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)

data = pd.read_csv("my_csv.csv")
dataset = Dataset.from_pandas(data)
tokenized_dataset = dataset.map(lambda samples: tokenizer(samples["text"]))

trainer = transformers.Trainer(
    model=model,
    train_dataset=tokenized_dataset,
    args=transformers.TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=100,
        max_steps=100,
        learning_rate=1e-3,
        fp16=True,
        logging_steps=1,
        output_dir='outputs',
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = True  # silence the warnings. Please re-enable for inference!
trainer.train()

并将其保存在本地：

trainer.save_model(cwd+"/finetuned_model")
print("saved trainer locally")

以及到枢纽：

model.push_to_hub("lucas0/empath-llama-7b", create_pr=1)

如何加载我的微调模型？

Answer 1

要加载微调的 peft/lora 模型，请查看 guanco 示例，/sf/answers/5346067331/

import torch
from peft import PeftModel    
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer

model_name = "decapoda-research/llama-7b-hf"
adapters_name = "lucas0/empath-llama-7b"

print(f"Starting to load the model {model_name} into memory")

m = AutoModelForCausalLM.from_pretrained(
    model_name,
    #load_in_4bit=True,
    torch_dtype=torch.bfloat16,
    device_map={"": 0}
)
m = PeftModel.from_pretrained(m, adapters_name)
m = m.merge_and_unload()
tok = LlamaTokenizer.from_pretrained(model_name)
tok.bos_token_id = 1

stop_token_ids = [0]

print(f"Successfully loaded the model {model_name} into memory")

您至少需要 A10g GPU 运行时才能正确加载模型。

欲了解更多详情，请参阅

• https://github.com/artidoro/qlora#tutorials-and-demonstrations
• 推理笔记本： https://colab.research.google.com/drive/1ge2F1QSK8Q7h0hn3YKuBCOAS0bK8E0wf ?usp=sharing
• 培训笔记本： https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k ?usp=sharing