损失函数减小,但列车组的精度在张量流中不会改变

Question

我正在尝试使用tensorflow使用深度卷积神经网络实现简单的性别分类器.我找到了这个模型并实现了它.

def create_model_v2(data):

    cl1_desc = {'weights':weight_variable([7,7,3,96]), 'biases':bias_variable([96])}
    cl2_desc = {'weights':weight_variable([5,5,96,256]), 'biases':bias_variable([256])}
    cl3_desc = {'weights':weight_variable([3,3,256,384]), 'biases':bias_variable([384])}

    fc1_desc = {'weights':weight_variable([240000, 128]), 'biases':bias_variable([128])}
    fc2_desc = {'weights':weight_variable([128,128]), 'biases':bias_variable([128])}
    fc3_desc = {'weights':weight_variable([128,2]), 'biases':bias_variable([2])}

    cl1 = conv2d(data,cl1_desc['weights'] + cl1_desc['biases'])
    cl1 = tf.nn.relu(cl1)
    pl1 = max_pool_nxn(cl1,3,[1,2,2,1])
    lrm1 = tf.nn.local_response_normalization(pl1)

    cl2 = conv2d(lrm1, cl2_desc['weights'] + cl2_desc['biases'])
    cl2 = tf.nn.relu(cl2)
    pl2 = max_pool_nxn(cl2,3,[1,2,2,1])
    lrm2 = tf.nn.local_response_normalization(pl2)

    cl3 = conv2d(lrm2, cl3_desc['weights'] + cl3_desc['biases'])
    cl3 = tf.nn.relu(cl3)
    pl3 = max_pool_nxn(cl3,3,[1,2,2,1])

    fl = tf.contrib.layers.flatten(cl3)

    fc1 = tf.add(tf.matmul(fl, fc1_desc['weights']), fc1_desc['biases'])
    drp1 = tf.nn.dropout(fc1,0.5)
    fc2 = tf.add(tf.matmul(drp1, fc2_desc['weights']), fc2_desc['biases'])
    drp2 = tf.nn.dropout(fc2,0.5)
    fc3 = tf.add(tf.matmul(drp2, fc3_desc['weights']), fc3_desc['biases'])

    return fc3  

此时我需要注意的是,我还完成了论文中描述的所有预处理步骤,但是我的图像大小调整为100x100x3而不是277x277x3.

我已经[0,1]为女性和[1,0]男性定义了logits

x = tf.placeholder('float',[None,100,100,3])
y = tf.placeholder('float',[None,2])

并将培训程序定义如下:

def train(x, hm_epochs, LR):
    #prediction = create_model_v2(x)
    prediction = create_model_v2(x)
    cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits  = prediction, labels = y) )
    optimizer = tf.train.AdamOptimizer(learning_rate=LR).minimize(cost)
    batch_size = 50
    correct = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct, 'float'))
    print("hello")
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())

        for epoch in range(hm_epochs):
            epoch_loss = 0
            i = 0
            while i < (len(x_train)):
                start = i
                end = i + batch_size
                batch_x = x_train[start:end]
                batch_y = y_train[start:end]
                whatever, vigen = sess.run([optimizer, cost], feed_dict = {x:batch_x, y:batch_y})
                epoch_loss += vigen
                i+=batch_size

            print('Epoch',  epoch ,'loss:',epoch_loss/len(x_train))
            if (epoch+1) % 2 == 0:
                j = 0
                acc = []
                while j < len(x_test):
                    acc += [accuracy.eval(feed_dict = {x:x_test[j:j + 10], y:y_test[j:j+10]})]
                    j+= 10
                print ('accuracy after', epoch + 1, 'epochs on test set: ', sum(acc)/len(acc))

                j = 0
                acc = []
                while j < len(x_train):
                    acc += [accuracy.eval(feed_dict = {x:x_train[j:j + 10], y:y_train[j:j+10]})]
                    j+= 10
                print ('accuracy after', epoch, ' epochs on train set:', sum(acc)/len(acc))

上述代码的一半仅用于每2个时期输出测试和训练精度.

无论如何,损失在第一个时代开始高涨

('Epoch',0,'损失:',148.87030902462453)

('Epoch',1,'损失:',0.01549744715988636)

('精确度',2,'测试集上的'时代:',0.33052011888510396)

('精确度',1,'火车上的纪元:',0.49607501227222384)

('Epoch',2,'损失:',0.015493246909976005)

我错过了什么？

并继续像这样保持列车组的准确度为0.5.

编辑:函数权重变量,conv2d和max_pool_nn是

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def avg_pool_nxn(x, n, strides):
    return tf.nn.avg_pool(x, ksize=[1,n,n,1], strides = strides,padding = 'SAME')

def max_pool_nxn(x, n, strides):
    return tf.nn.max_pool(x, ksize=[1,n,n,1], strides = strides, padding = 'SAME')

def conv2d(x, W,stride = [1,1,1,1]):
    return tf.nn.conv2d(x, W, strides = stride, padding = 'SAME') 

编辑2 - 问题解决了

问题与参数初始化有着惊人的关系.将权重初始化从正态分布更改为Xavier初始化会产生奇迹,并且准确度最终达到约86%.如果有人对这里感兴趣的是原始论文http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf,如果有人知道并且想要解释为什么Xavier可以很好地使用回文和图像随时发布答案.

Answer 1

权重的正确初始化通常对于训练更深层次的神经网络至关重要。

Xavier 初始化的目的是确保每个神经元的输出方差预计为 1.0（参见此处）。这通常依赖于额外的假设，即您的输入已标准化为均值 0 和方差 1，因此确保这一点也很重要。

对于 ReLU 单元，我相信He 初始化实际上被认为是最佳实践。这需要从具有标准差的零均值高斯分布进行初始化：

其中n是输入单元的数量。有关其他一些激活函数的最佳实践，请参阅烤宽面条文档。

另一方面，批量归一化通常可以减少模型性能对权重初始化的依赖性。