如何使用 Pytorch 中的预训练权重修改具有 4 个通道作为输入的 resnet 50？

Question

我想更改 resnet50，以便可以切换到 4 通道输入，对 RGB 通道使用相同的权重，并使用均值 0 和方差 0.01 的法线初始化最后一个通道。

这是我的代码：

import torch.nn as nn
import torch
from torchvision import models

from misc.layer import Conv2d, FC

import torch.nn.functional as F
from misc.utils import *

import pdb

class Res50(nn.Module):
    def __init__(self,  pretrained=True):
        super(Res50, self).__init__()

        self.de_pred = nn.Sequential(Conv2d(1024, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))
        
        self._initialize_weights()

        res = models.resnet50(pretrained=pretrained)
        pretrained_weights = res.conv1.weight

        res.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3,bias=False)

        res.conv1.weight[:,:3,:,:] = pretrained_weights
        res.conv1.weight[:,3,:,:].data.normal_(0.0, std=0.01)
        
        self.frontend = nn.Sequential(
            res.conv1, res.bn1, res.relu, res.maxpool, res.layer1, res.layer2
        )
        
        self.own_reslayer_3 = make_res_layer(Bottleneck, 256, 6, stride=1)        
        self.own_reslayer_3.load_state_dict(res.layer3.state_dict())

        
    def forward(self,x):
        x = self.frontend(x)
        x = self.own_reslayer_3(x)
        x = self.de_pred(x)
        x = F.upsample(x,scale_factor=8)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                m.weight.data.normal_(0.0, std=0.01)
                if m.bias is not None:
                    m.bias.data.fill_(0)
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.fill_(1)
                m.bias.data.fill_(0)

但它会产生以下错误，有人有任何建议吗？

/usr/local/lib/python3.6/dist-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
Traceback (most recent call last):
  File "train.py", line 62, in <module>
    cc_trainer = Trainer(loading_data,cfg_data,pwd)
  File "/content/drive/My Drive/Folder/Code/trainer.py", line 28, in __init__
    self.optimizer = optim.Adam(self.net.CCN.parameters(), lr=cfg.LR, weight_decay=1e-4) #remenber was 1e-4
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py", line 44, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 206, in add_param_group
    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor

Answer 1

理想情况下，ResNet 接受 3 通道输入。为了使其适用于 4 通道输入，您必须添加一个额外的层（2D conv），将 4 通道输入传递到该层，以使该层的输出适合 ResNet 架构。

脚步

1. 复制模型重量

   weight = model.conv1.weight.clone()
   

2. 为 4 通道输入添加额外的 2d 转换

   model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False) #here 4 indicates 4-channel input
   

3. 您可以在额外的 con2d 之上添加 Relu 和 BatchNorm。在这个例子中，我没有使用。

4. 将额外的cov2d与ResNet模型连接起来（之前复制的权重）

   with torch.no_grad():
       model.conv1.weight[:, :3] = weight
       model.conv1.weight[:, 3] = model.conv1.weight[:, 0]
   

5. 完毕

抱歉，我没有修改你的代码。您可以调整代码中的更改。

Answer 2

.data也尝试设置第一个通道：

res.conv1.weight[:,:3,:,:].data[...] = pretrained_weights