pytorch中的层标准化？

Question

层标准化不应该是x = torch.tensor([[1.5,0,0,0,0]])吗[[1.5,-0.5,-0.5,-0.5]]？根据本文和pytorch doc中的方程。但torch.nn.LayerNorm给予[[ 1.7320, -0.5773, -0.5773, -0.5773]]

这是示例代码：

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)

y1 = layerNorm(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)

在哪里：

y1 == tensor([[ 1.7320, -0.5773, -0.5773, -0.5773]])
y2 == tensor([[ 1.5000, -0.5000, -0.5000, -0.5000]])

Answer 1

代替

var = x.var(-1, keepdim = True)

你应该使用

var = x.var(-1, keepdim = True, unbiased=False)

这将产生与 pytorch 相同的结果，完整代码：

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)
y1 = layerNorm(x)
mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True, unbiased=False)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)

Answer 2

显然，代码应该是这样的：

...
var = x.mean((x-mean)**2, -1, keepdim = True)
...

希望这对任何遇到同样错误的人都有帮助。