相关疑难解决方法(0)

我正在使用TensorFlow构建一个简单的多层感知器,我还需要获得神经网络输入处的损失的梯度(或误差信号).

这是我的代码,它有效:

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(self.network, self.y))
optimizer = tf.train.AdagradOptimizer(learning_rate=nn_learning_rate).minimize(cost)
...
for i in range(epochs):
    ....
    for batch in batches:
        ...
        sess.run(optimizer, feed_dict=feed_dict)
        grads_wrt_input = sess.run(tf.gradients(cost, self.x), feed_dict=feed_dict)[0]

(编辑包括训练循环)

没有最后一行(grads_wrt_input...),这在CUDA机器上运行得非常快.但是,tf.gradients()性能会大幅降低十倍或更多.

我记得节点处的错误信号是作为反向传播算法中的中间值计算的,我已经使用Java库DeepLearning4j成功地完成了这个.我的印象是,这将是对已经构建的计算图的轻微修改optimizer.

如何更快地制作,或者是否有其他方法来计算输入损失的梯度？

我正在训练一个标准的,简单的多层感知器神经网络,在TensorFlow中有三个隐藏层.我添加了一个文本进度条,以便我可以观察迭代时代的进度.我发现的是每个迭代的处理时间在前几个时期之后增加.这是一个示例截图,显示每次迭代的增加:

在这种情况下,前几次迭代大约需要1.05s/it,而100%则需要4.01s/it.

相关代码列于此处:

# ------------------------- Build the TensorFlow Graph -------------------------

with tf.Graph().as_default():

    (a bunch of statements for specifying the graph)

# --------------------------------- Training ----------------------------------

    sess = tf.InteractiveSession()
    sess.run(tf.initialize_all_variables())

    print "Start Training"

    pbar = tqdm(total = training_epochs)
    for epoch in range(training_epochs):
        avg_cost = 0.0
    batch_iter = 0

    while batch_iter < batch_size:
        train_features = []
        train_labels = []
        batch_segments = random.sample(train_segments, 20)
        for segment in batch_segments:
            train_features.append(segment[0])
            train_labels.append(segment[1])
        sess.run(optimizer, feed_dict={x: train_features, y_: train_labels})
        line_out = "," + str(batch_iter) + "\n"
        train_outfile.write(line_out)
        line_out …