Keras网络永远无法对最后一堂课进行分类

Question

我一直致力于我的项目深度学习语言检测,这是一个具有这些层的网络,可以从16种编程语言中识别:

这是生成网络的代码:

# Setting up the model
graph_in = Input(shape=(sequence_length, number_of_quantised_characters))
convs = []
for i in range(0, len(filter_sizes)):
    conv = Conv1D(filters=num_filters,
                  kernel_size=filter_sizes[i],
                  padding='valid',
                  activation='relu',
                  strides=1)(graph_in)
    pool = MaxPooling1D(pool_size=pooling_sizes[i])(conv)
    flatten = Flatten()(pool)
    convs.append(flatten)

if len(filter_sizes)>1:
    out = Concatenate()(convs)
else:
    out = convs[0]

graph = Model(inputs=graph_in, outputs=out)

# main sequential model
model = Sequential()


model.add(Dropout(dropout_prob[0], input_shape=(sequence_length, number_of_quantised_characters)))
model.add(graph)
model.add(Dense(hidden_dims))
model.add(Dropout(dropout_prob[1]))
model.add(Dense(number_of_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

所以我的最后一个语言类是SQL,在测试阶段,它永远无法正确预测SQL,它的得分为0%.我认为这是由于SQL样本质量差(实际上它们很差)所以我删除了这个类并开始训练15个类.令我惊讶的是,现在F#文件的检测率为0%,而F#是删除SQL后的最后一个类(即最后一个位置为1且其余为0的单热矢量).现在,如果一个训练有16的网络用于对抗15,那么它将获得98.5%的非常高的成功率.

我使用的代码非常简单,主要在defs.py和data_helper.py中提供

以下是针对16个类测试的16个课程的网络训练结果:

Final result: 14827/16016 (0.925761738262)
xml:        995/1001 (0.994005994006)
fsharp:     974/1001 (0.973026973027)
clojure:        993/1001 (0.992007992008)
java:       996/1001 (0.995004995005)
scala:      990/1001 (0.989010989011)
python:     983/1001 (0.982017982018)
sql:        0/1001 (0.0)
js:     991/1001 (0.99000999001)
cpp:        988/1001 (0.987012987013)
css:        987/1001 (0.986013986014)
csharp:     994/1001 (0.993006993007)
go:     989/1001 (0.988011988012)
php:        998/1001 (0.997002997003)
ruby:       995/1001 (0.994005994006)
powershell:     992/1001 (0.991008991009)
bash:       962/1001 (0.961038961039)

这是同一个网络(训练为16)对15个课程的结果:

Final result: 14827/15015 (0.987479187479)
xml:        995/1001 (0.994005994006)
fsharp:     974/1001 (0.973026973027)
clojure:        993/1001 (0.992007992008)
java:       996/1001 (0.995004995005)
scala:      990/1001 (0.989010989011)
python:     983/1001 (0.982017982018)
js:     991/1001 (0.99000999001)
cpp:        988/1001 (0.987012987013)
css:        987/1001 (0.986013986014)
csharp:     994/1001 (0.993006993007)
go:     989/1001 (0.988011988012)
php:        998/1001 (0.997002997003)
ruby:       995/1001 (0.994005994006)
powershell:     992/1001 (0.991008991009)
bash:       962/1001 (0.961038961039)

有没有人见过这个？我怎么能绕过它呢？

Answer 1

TL; DR:问题是您的数据在被分成训练和验证集之前不会被洗牌.因此,在训练期间,属于"sql"类的所有样本都在验证集中.如果没有在该类中给出样本,您的模型将不会学习预测最后一个类.

在get_input_and_labels(),首先加载类0的文件,然后加载类1,依此类推.既然你设置了n_max_files = 2000,就意味着

• 第一个2000(左右,取决于你实际拥有多少个文件)条目Y将是0级("go")
• 接下来的2000个参赛作品将是1级("csharp")
• ...
• 最后2000个条目将是最后一个类("sql").

不幸的是,Keras在将数据拆分为训练和验证集之前不会对数据进行混洗.因为validation_split在代码中设置为0.1,所以最后3000个样本(包含所有"sql"样本)将在验证集中.

如果设置validation_split为更高的值(例如,0.2),您将看到更多的类得分为0%:

Final result: 12426/16016 (0.7758491508491508)
go:             926/1001 (0.9250749250749251)
csharp:         966/1001 (0.965034965034965)
java:           973/1001 (0.972027972027972)
js:             929/1001 (0.9280719280719281)
cpp:            986/1001 (0.985014985014985)
ruby:           942/1001 (0.9410589410589411)
powershell:             981/1001 (0.98001998001998)
bash:           882/1001 (0.8811188811188811)
php:            977/1001 (0.9760239760239761)
css:            988/1001 (0.987012987012987)
xml:            994/1001 (0.993006993006993)
python:         986/1001 (0.985014985014985)
scala:          896/1001 (0.8951048951048951)
clojure:                0/1001 (0.0)
fsharp:         0/1001 (0.0)
sql:            0/1001 (0.0)

如果在加载后对数据进行洗牌,则可以解决该问题.看来你已经在洗牌数据了:

# Shuffle data
shuffle_indices = np.random.permutation(np.arange(len(y)))
x_shuffled = x[shuffle_indices]
y_shuffled = y[shuffle_indices].argmax(axis=1)

但是,如果你适合的模型,你通过了原有的x和y以fit()代替x_shuffled和y_shuffled.如果您将该行更改为:

model.fit(x_shuffled, y_shuffled, batch_size=batch_size,
          epochs=num_epochs, validation_split=val_split, verbose=1)

测试输出会变得更合理:

Final result: 15248/16016 (0.952047952047952)
go:             865/1001 (0.8641358641358642)
csharp:         986/1001 (0.985014985014985)
java:           977/1001 (0.9760239760239761)
js:             953/1001 (0.952047952047952)
cpp:            974/1001 (0.973026973026973)
ruby:           985/1001 (0.984015984015984)
powershell:             974/1001 (0.973026973026973)
bash:           942/1001 (0.9410589410589411)
php:            979/1001 (0.978021978021978)
css:            965/1001 (0.964035964035964)
xml:            988/1001 (0.987012987012987)
python:         857/1001 (0.8561438561438561)
scala:          955/1001 (0.954045954045954)
clojure:                985/1001 (0.984015984015984)
fsharp:         950/1001 (0.949050949050949)
sql:            913/1001 (0.9120879120879121)