在Apple提供的MetalImageRecognition示例代码中,如何对权重和偏差参数执行批量规范化？

Question

README提供以下注释:

/* 
The weights for this particular network were batch normalized but for inference we may use : 

w =  gamma  / ?(s + 0.001), b = ß - ( A * m ) 

s: variance 
m: mean 
gamma : gamma 
ß: beta 

w: weights of a feature channel 
b: bias of a feature channel 

for every feature channel separately to get the corresponding weights and  bias 
*/  

我已经能够使用TensorFlow将所有训练过的参数从重新训练的Inception模型导出到二进制.例如,对于第一个卷积节点,这些是可用的二进制文件:

conv0/BatchNorm/beta conv0/BatchNorm/beta/ExponentialMovingAverage conv0/BatchNorm/beta/RMSProp conv0/BatchNorm/beta/RMSProp_1 conv0/BatchNorm/moving_mean conv0/BatchNorm/moving_mean/ExponentialMovingAverage conv0/BatchNorm/moving_variance conv0/BatchNorm/moving_variance/ExponentialMovingAverage conv0/weights conv0/weights/ExponentialMovingAverage conv0/weights/Regularizer/L2Loss/value/avg conv0/weights/RMSProp conv0/weights/RMSProp_1

这些文件是否被转换或重新计算以获得相应的conv.dat文件,或者TensorFlow中是否有一个函数可以通过批量规范化导出每个节点？

任何其他方向都非常有用,因为这里连接点的资源很少.

谢谢.

Answer 1

我创建了一个repo,它将Inception v3批量标准化权重转换为MPSCNNConvolution所需的非标准化权重.

https://github.com/kakugawa/MetalCNNWeights

在Sergey Ioffe和Christian Szegedy(https://arxiv.org/pdf/1502.03167v3.pdf)的论文"批量标准化:通过减少内部协变量加速深度网络训练"中,我们可以使用算法2,输出,步骤11推导:

[Weight = \frac{\gamma}{\sqrt{Var[x]+0.001}} * Weight_{BN}](http://mathurl.com/z7snq3z.png)
[Bias = \beta - (\frac{\gamma}{\sqrt{Var[x]+0.001}}) * E[x]](http://mathurl.com/zo4shhf.png)

注意:在发布图片或2个链接之前,我需要10个声望点.