如何在TensorFlow中添加正则化？

Question

我在使用TensorFlow实现的许多可用神经网络代码中发现,正则化术语通常通过手动向损失值添加附加项来实现.

我的问题是:

1. 有没有比手动更优雅或推荐的正规化方法？

2. 我也发现get_variable有一个论点regularizer.该如何使用？根据我的观察,如果我们将正则化器传递给它(例如tf.contrib.layers.l2_regularizer,表示正则化术语的张量将被计算并添加到名为的图形集合中tf.GraphKeys.REGULARIZATOIN_LOSSES.TensorFlow是否会自动使用该集合(例如,在训练时由优化器使用)？或者是否应该自己使用该系列？

Answer 1

正如您在第二点所说,使用regularizer参数是推荐的方法.您可以在其中使用它get_variable,或在您的设置中将其设置一次,variable_scope并使所有变量正规化.

损失收集在图表中,您需要手动将它们添加到您的成本函数中.

  reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
  reg_constant = 0.01  # Choose an appropriate one.
  loss = my_normal_loss + reg_constant * sum(reg_losses)

希望有所帮助!

Answer 2

现有答案的几个方面对我来说并不是很清楚,所以这里有一个循序渐进的指南:

1. 定义正则化器.这是可以设置正则化常数的地方,例如:

   regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
   

2. 通过以下方式创建

       weights = tf.get_variable(
           name="weights",
           regularizer=regularizer,
           ...
       )
   

   同样,变量可以通过常规weights = tf.Variable(...)构造函数创建,然后是tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, weights).

3. 定义一些loss术语并添加正则化术语:

   reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
   reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
   loss += reg_term
   

   注意:它看起来像是tf.contrib.layers.apply_regularization一个AddN,所以或多或少相当于sum(reg_variables).

Answer 3

我会提供一个简单正确的答案,因为我找不到一个.你需要两个简单的步骤,其余的是由tensorflow魔术完成的:

1. 在创建变量或图层时添加正则化程序:

   tf.layers.dense(x, kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001))
   # or
   tf.get_variable('a', regularizer=tf.contrib.layers.l2_regularizer(0.001))
   

2. 定义损失时添加正则化项:

   loss = ordinary_loss + tf.losses.get_regularization_loss()
   

Answer 4

contrib.learn基于Tensorflow网站上的Deep MNIST教程,使用该库执行此操作的另一个选项如下.首先,假设您已导入相关库(例如import tensorflow.contrib.layers as layers),您可以使用单独的方法定义网络:

def easier_network(x, reg):
    """ A network based on tf.contrib.learn, with input `x`. """
    with tf.variable_scope('EasyNet'):
        out = layers.flatten(x)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=200,
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = tf.nn.tanh)
        out = layers.fully_connected(out, 
                num_outputs=10, # Because there are ten digits!
                weights_initializer = layers.xavier_initializer(uniform=True),
                weights_regularizer = layers.l2_regularizer(scale=reg),
                activation_fn = None)
        return out 

然后,在main方法中,您可以使用以下代码段:

def main(_):
    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)
    x = tf.placeholder(tf.float32, [None, 784])
    y_ = tf.placeholder(tf.float32, [None, 10])

    # Make a network with regularization
    y_conv = easier_network(x, FLAGS.regu)
    weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, 'EasyNet') 
    print("")
    for w in weights:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")
    reg_ws = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES, 'EasyNet')
    for w in reg_ws:
        shp = w.get_shape().as_list()
        print("- {} shape:{} size:{}".format(w.name, shp, np.prod(shp)))
    print("")

    # Make the loss function `loss_fn` with regularization.
    cross_entropy = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    loss_fn = cross_entropy + tf.reduce_sum(reg_ws)
    train_step = tf.train.AdamOptimizer(1e-4).minimize(loss_fn)

为了实现这一点,您需要遵循我之前链接的MNIST教程并导入相关库,但是学习TensorFlow是一个很好的练习,很容易看出正则化如何影响输出.如果将正则化应用为参数,则可以看到以下内容:

- EasyNet/fully_connected/weights:0 shape:[784, 200] size:156800
- EasyNet/fully_connected/biases:0 shape:[200] size:200
- EasyNet/fully_connected_1/weights:0 shape:[200, 200] size:40000
- EasyNet/fully_connected_1/biases:0 shape:[200] size:200
- EasyNet/fully_connected_2/weights:0 shape:[200, 10] size:2000
- EasyNet/fully_connected_2/biases:0 shape:[10] size:10

- EasyNet/fully_connected/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_1/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0
- EasyNet/fully_connected_2/kernel/Regularizer/l2_regularizer:0 shape:[] size:1.0

请注意,正则化部分根据可用项目为您提供三个项目.

通过0,0.0001,0.01和1.0的正则化,我得到的测试精度值分别为0.9468,0.9476,0.9183和0.1135,显示了高正则化项的危险性.

Answer 5

If anyone's still looking, I'd just like to add on that in tf.keras you may add weight regularization by passing them as arguments in your layers. An example of adding L2 regularization taken wholesale from the Tensorflow Keras Tutorials site:

model = keras.models.Sequential([
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu, input_shape=(NUM_WORDS,)),
    keras.layers.Dense(16, kernel_regularizer=keras.regularizers.l2(0.001),
                       activation=tf.nn.relu),
    keras.layers.Dense(1, activation=tf.nn.sigmoid)
])

There's no need to manually add in the regularization losses with this method as far as I know.

Reference: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit#add_weight_regularization