如何在TensorFlow中定义各个层的重量衰减？

Question

在CUDA ConvNet中,我们可以为每个层编写类似这样的内容(源代码):

[conv32]
epsW=0.001
epsB=0.002
momW=0.9
momB=0.9
wc=0

其中wc=0指的是L2重量衰减.

如何在TensorFlow中实现同样的目标？

Answer 1

您可以将要添加重量衰减的所有变量添加到集合名称"变量"中,然后计算整个集合的L2范数权重衰减.

  # Create your variables
  weights = tf.get_variable('weights', collections=['variables'])

  with tf.variable_scope('weights_norm') as scope:
    weights_norm = tf.reduce_sum(
      input_tensor = WEIGHT_DECAY_FACTOR*tf.pack(
          [tf.nn.l2_loss(i) for i in tf.get_collection('weights')]
      ),
      name='weights_norm'
  )

  # Add the weight decay loss to another collection called losses
  tf.add_to_collection('losses', weights_norm)

  # Add the other loss components to the collection losses     
  # ...

  # To calculate your total loss
  tf.add_n(tf.get_collection('losses'), name='total_loss')

Answer 2

get_variable(
name,
shape=None,
dtype=None,
initializer=None,
regularizer=None,
trainable=True,
collections=None,
caching_device=None,
partitioner=None,
validate_shape=True,
use_resource=None,
custom_getter=None)

这是tensorflow函数的用法get_variable.您可以轻松指定正规化器来进行重量衰减.

以下是一个例子:

weight_decay = tf.constant(0.0005, dtype=tf.float32) # your weight decay rate, must be a scalar tensor.
W = tf.get_variable(name='weight', shape=[4, 4, 256, 512], regularizer=tf.contrib.layers.l2_regularizer(weight_decay))