tensorflow:在tf.map_fn的fn中创建变量返回值错误

Question

我对map_fn中的变量初始化有疑问.

我试图在张量中的每个单独元素上分别应用一些公路图层,所以我认为map_fn可能是最好的方法.

segment_list = tf.reshape(raw_segment_embedding,[batch_size*seqlen,embed_dim])
segment_embedding = tf.map_fn(lambda x: stack_highways(x, hparams), segment_list)

现在问题是我的fn,即stack_highways,创建变量,并且由于某种原因,tensorflow无法初始化这些变量并给出此错误.

W = tf.Variable(tf.truncated_normal(W_shape, stddev=0.1), name='weight')

ValueError: Initializer for variable body/model/parallel_0/body/map/while/highway_layer0/weight/ is from inside a control-flow construct, such as a loop or conditional. When creating a variable inside a loop or conditional, use a lambda as the initializer. 

我现在很无能,根据错误我认为它不是关于范围但我不知道如何使用lambda作为初始化器(我甚至不知道究竟是什么意思).下面是stack_highways的实现,任何建议都会非常感激..

def weight_bias(W_shape, b_shape, bias_init=0.1):
  """Fully connected highway layer adopted from 
     https://github.com/fomorians/highway-fcn/blob/master/main.py
  """
  W = tf.Variable(tf.truncated_normal(W_shape, stddev=0.1), name='weight')
  b = tf.Variable(tf.constant(bias_init, shape=b_shape), name='bias')
  return W, b




def highway_layer(x, size, activation, carry_bias=-1.0):
  """Fully connected highway layer adopted from 
     https://github.com/fomorians/highway-fcn/blob/master/main.py
  """
  W, b = weight_bias([size, size], [size])
  with tf.name_scope('transform_gate'):
    W_T, b_T = weight_bias([size, size], bias_init=carry_bias)


    H = activation(tf.matmul(x, W) + b, name='activation')
    T = tf.sigmoid(tf.matmul(x, W_T) + b_T, name='transform_gate')
    C = tf.sub(1.0, T, name="carry_gate")


    y = tf.add(tf.mul(H, T), tf.mul(x, C), name='y') # y = (H * T) + (x * C)
    return y




def stack_highways(x, hparams):
  """Create highway networks, this would not create
  a padding layer in the bottom and the top, it would 
  just be layers of highways.


  Args:
    x: a raw_segment_embedding
    hparams: run hyperparameters


  Returns:
    y: a segment_embedding
  """
  highway_size = hparams.highway_size
  activation = hparams.highway_activation #tf.nn.relu
  carry_bias_init = hparams.highway_carry_bias
  prev_y = None
  y = None
  for i in range(highway_size):
    with tf.name_scope("highway_layer{}".format(i)) as scope:
      if i == 0: # first, input layer
        prev_y = highway_layer(x, highway_size, activation, carry_bias=carry_bias_init)
      elif i == highways - 1: # last, output layer
        y = highway_layer(prev_y, highway_size, activation, carry_bias=carry_bias_init)
      else: # hidden layers
        prev_y = highway_layer(prev_y, highway_size, activation, carry_bias=carry_bias_init)
  return y

最温暖的问候,科尔曼

Answer 1

TensorFlow提供了两种初始化变量的主要方法:

1. "lambda"初始化器:返回初始化值的callable.TF提供了很多很好的包装者.
2. 张量值初始化:这是您当前使用的.

错误消息表明在使用a while_loop(map_fn内部调用)中的变量时需要使用第一种类型的初始化程序.(一般来说,lambda初始化器对我来说似乎更健壮.)

此外,在过去,当从控制流中使用时,tf.get_variable似乎优于tf.Variable.

所以,我怀疑你可以通过修改你的weight_bias功能来解决你的问题:

def weight_bias(W_shape, b_shape, bias_init=0.1):
  """Fully connected highway layer adopted from 
     https://github.com/fomorians/highway-fcn/blob/master/main.py
  """
  W = tf.get_variable("weight", shape=W_shape,
          initializer=tf.truncated_normal_initializer(stddev=0.1))
  b = tf.get_variable("bias", shape=b_shape,
          initializer=tf.constant_inititializer(bias_init))
  return W, b

希望有所帮助!