TensorFlow:向LSTM添加正则化

Question

在Tensorflow LSTM正规化之后我试图在训练LSTM单元的参数时将正则化项添加到成本函数中.

抛开一些常数,我有:

def RegularizationCost(trainable_variables):
    cost = 0
    for v in trainable_variables:
        cost += r(tf.reduce_sum(tf.pow(r(v.name),2)))
    return cost

...

regularization_cost = tf.placeholder(tf.float32, shape = ())
cost = tf.reduce_sum(tf.pow(pred - y, 2)) + regularization_cost
optimizer = tf.train.AdamOptimizer(learning_rate = 0.01).minimize(cost)

...

tv = tf.trainable_variables()
s = tf.Session()
r = s.run

...

while (...):
    ...

    reg_cost = RegularizationCost(tv)
    r(optimizer, feed_dict = {x: x_b, y: y_b, regularization_cost: reg_cost})

我reg_cost遇到的问题是,添加正则化项会极大地减慢学习过程,实际上正则化项随着每次迭代的增加而明显增加,当时相关的术语pred - y几乎停滞reg_cost不动,即似乎没有考虑到.

我怀疑我是以完全错误的方式添加这个术语.我不知道如何在成本函数本身中添加这个术语,所以我使用了标量tf.placeholder的解决方法,并"手动"计算了正则化成本.怎么做得好？

Answer 1

仅计算L2损失一次:

tv = tf.trainable_variables()
regularization_cost = tf.reduce_sum([ tf.nn.l2_loss(v) for v in tv ])
cost = tf.reduce_sum(tf.pow(pred - y, 2)) + regularization_cost
optimizer = tf.train.AdamOptimizer(learning_rate = 0.01).minimize(cost)

您可能希望删除bias那些不应该正则化的变量.