TensorFlow:训练for循环中的每次迭代都较慢

Question

我正在训练一个标准的,简单的多层感知器神经网络,在TensorFlow中有三个隐藏层.我添加了一个文本进度条,以便我可以观察迭代时代的进度.我发现的是每个迭代的处理时间在前几个时期之后增加.这是一个示例截图,显示每次迭代的增加:

在这种情况下,前几次迭代大约需要1.05s/it,而100%则需要4.01s/it.

相关代码列于此处:

# ------------------------- Build the TensorFlow Graph -------------------------

with tf.Graph().as_default():

    (a bunch of statements for specifying the graph)

# --------------------------------- Training ----------------------------------

    sess = tf.InteractiveSession()
    sess.run(tf.initialize_all_variables())

    print "Start Training"

    pbar = tqdm(total = training_epochs)
    for epoch in range(training_epochs):
        avg_cost = 0.0
    batch_iter = 0

    while batch_iter < batch_size:
        train_features = []
        train_labels = []
        batch_segments = random.sample(train_segments, 20)
        for segment in batch_segments:
            train_features.append(segment[0])
            train_labels.append(segment[1])
        sess.run(optimizer, feed_dict={x: train_features, y_: train_labels})
        line_out = "," + str(batch_iter) + "\n"
        train_outfile.write(line_out)
        line_out = ",," + str(sess.run(tf.reduce_mean(weights['h1']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h2']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h3']), feed_dict={x: train_features, y_: train_labels})) + "\n"
        train_outfile.write(line_out)
        avg_cost += sess.run(cost, feed_dict={x: train_features, y_: train_labels})/batch_size

        batch_iter += 1

    pbar.update(1)  # Increment the progress bar by one

train_outfile.close()
print "Completed training"

在搜索stackoverflow时,我发现每次迭代后处理时间越来越长,其他人也遇到问题,每次迭代都比上一次更长.但是,我相信我的可能会有所不同,因为他们使用如下语句清楚地将操作添加到图表中:

distorted_image = tf.image.random_flip_left_right(image_tensor)

虽然我是TensorFlow的新手,但我不相信我犯了同样的错误,因为我的循环中唯一的东西是sess.run()调用.

任何帮助深表感谢.

Answer 1

你有三个地方:

sess.run(tf.reduce_mean(weights['h1']), ...)

每次在while循环的迭代中向tf.reduce_mean()图中追加一个新节点,这会增加开销.尝试在while循环之外创建它们:

with tf.Graph().as_default():
  ...
  m1 = tf.reduce_mean(weights['h1'])

while batch_iter < batch_size:
  ...
  line_out = ",," + str(sess.run(m1, feed_dict={x: train_features, y_: train_labels}))