函数逼近Tensorflow

Question

我试图在Tensorflow中创建一个近似正弦函数的神经网络.我已经找到了一些通用函数逼近器的例子,但是我并没有完全理解代码,因为我对Tensorflow很新,我想自己编写代码来理解每一步.

这是我的代码:

import tensorflow as tf
import numpy as np
import math, random
import matplotlib.pyplot as plt


# Create the arrays x and y that contains the inputs and the outputs of the function to approximate
x = np.arange(0, 2*np.pi, 2*np.pi/1000).reshape((1000,1))
y = np.sin(x)
# plt.plot(x,y)
# plt.show()

# Define the number of nodes
n_nodes_hl1 = 100
n_nodes_hl2 = 100

# Define the number of outputs and the learn rate
n_classes = 1
learn_rate = 0.1

# Define input / output placeholders
x_ph = tf.placeholder('float', [None, 1])
y_ph = tf.placeholder('float')


# Routine to compute the neural network (2 hidden layers)
def neural_network_model(data):
    hidden_1_layer = {'weights': tf.Variable(tf.random_normal([1, n_nodes_hl1])),
                      'biases': tf.Variable(tf.random_normal([n_nodes_hl1]))}

    hidden_2_layer = {'weights': tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                      'biases': tf.Variable(tf.random_normal([n_nodes_hl2]))}

    output_layer = {'weights': tf.Variable(tf.random_normal([n_nodes_hl2, n_classes])),
                      'biases': tf.Variable(tf.random_normal([n_classes]))}


    # (input_data * weights) + biases
    l1 = tf.add(tf.matmul(data, hidden_1_layer['weights']), hidden_1_layer['biases'])
    l1 = tf.nn.relu(l1)

    l2 = tf.add(tf.matmul(l1, hidden_2_layer['weights']), hidden_2_layer['biases'])
    l2 = tf.nn.relu(l2)

    output = tf.add(tf.matmul(l2, output_layer['weights']), output_layer['biases'])

    return output


# Routine to train the neural network
def train_neural_network(x_ph):
    prediction = neural_network_model(x_ph)
    cost = tf.reduce_mean(tf.square(prediction - y_ph))
    optimizer = tf.train.GradientDescentOptimizer(learn_rate).minimize(cost)

    # cycles feed forward + backprop
    hm_epochs = 10

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        # Train in each epoch with the whole data
        for epoch in range(hm_epochs):
            epoch_loss = 0
            _, c = sess.run([optimizer, cost], feed_dict = {x_ph: x, y_ph: y})
            epoch_loss += c
            print('Epoch', epoch, 'completed out of', hm_epochs, 'loss:', epoch_loss)

        correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y_ph, 1))
        accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
        print('Accuracy;', accuracy.eval({x_ph: x, y_ph: x}))


# Train network
train_neural_network(x_ph)

如果您运行该程序,您将看到损失如何分歧,我不知道为什么它会像那样.谁能帮助我？

谢谢!

Answer 1

@AIdream initial learning rate convergence issue一般都是正确的.但即使有了lean_rate=1.0e-9和10000 epochs,错误仍然很大意味着问题是别的.

调试问题

运行上面的代码,给出:

Epoch 0 completed out of 10 loss: 61437.30859375
Epoch 1 completed out of 10 loss: 1.2855042406744022e+21
Epoch 2 completed out of 10 loss: inf
Epoch 3 completed out of 10 loss: nan

上面的代码试图sin function在一个范围内 近似a (0, 2*pi).由于标签(输出)将是(-1,1),因此较高的错误表示为权重初始化较大的值.将权重更改为具有较小的初始值(stddev=0.01),会导致:

Epoch 0 completed out of 10 loss: 0.5000443458557129
Epoch 1 completed out of 10 loss: 0.4999848008155823
Epoch 2 completed out of 10 loss: 0.49993154406547546
Epoch 3 completed out of 10 loss: 0.4998819828033447

损失收敛得非常快,但检查预测似乎输入都被映射到零.

问题是因为上面代码中的输入是作为a single batch而不是作为mini batches.Batch gradient decent可以导致局部最小问题,一旦达到局部最小值,它就不会出现问题.Mini batch避免这个问题,因为批量计算的梯度是嘈杂的,可以让你超出当地的最小值.随着这些变化导致:

Epoch 0 completed out of 100 loss: 456.28773515997455
Epoch 10 completed out of 100 loss: 6.713319106237066
Epoch 20 completed out of 100 loss: 0.24847120749460316
Epoch 30 completed out of 100 loss: 0.09907744570556076

通过从这里下载源可以重现上述步骤.

Answer 2

梯度下降的初始学习率太大，无法收敛到最小值（例如，请参阅有关梯度下降和学习率值的其他线程：“如果学习率太大，梯度下降会爆炸”）。

只需将其值替换为例如learn_rate = 1.0e-9，您的网络就会收敛。

痕迹：

Epoch 0 completed out of 10000 loss: 8512.4736328125
Epoch 1 completed out of 10000 loss: 8508.4677734375
...
Epoch 201 completed out of 10000 loss: 7743.56396484375
Epoch 202 completed out of 10000 loss: 7739.92431640625
...
Epoch 7000 completed out of 10000 loss: 382.22601318359375
Epoch 7001 completed out of 10000 loss: 382.08026123046875
...
Epoch 9998 completed out of 10000 loss: 147.459716796875
Epoch 9999 completed out of 10000 loss: 147.4239501953125
...