Keras:加权二元交叉熵

Question

我试图用Keras实现加权二进制交叉熵,但我不确定代码是否正确.训练输出似乎有点令人困惑.在几个时代之后,我得到的精确度为~0.15.我认为这太少了(即使是随机猜测).

输出中通常有大约11%的值和89%的零,因此权重为w_zero = 0.89且w_one = 0.11.

我的代码:

def create_weighted_binary_crossentropy(zero_weight, one_weight):

    def weighted_binary_crossentropy(y_true, y_pred):

        # Original binary crossentropy (see losses.py):
        # K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)

        # Calculate the binary crossentropy
        b_ce = K.binary_crossentropy(y_true, y_pred)

        # Apply the weights
        weight_vector = y_true * one_weight + (1. - y_true) * zero_weight
        weighted_b_ce = weight_vector * b_ce

        # Return the mean error
        return K.mean(weighted_b_ce)

    return weighted_binary_crossentropy

也许有人看到什么错了？

谢谢

Answer 1

通常，少数群体的权重较高。最好使用one_weight=0.89, zero_weight=0.11（顺便说一句，您可以使用class_weight={0: 0.11, 1: 0.89}，如注释中所建议）。

在类不平衡的情况下，您的模型看到的零比零多得多。它还将学会预测比零更多的零，因为这样做可以最大程度地减少训练损失。这就是为什么您看到的精度接近比例0.11的原因。如果对模型预测取平均值，则该平均值应非常接近零。

使用类权重的目的是更改损失函数，以使“简单的解决方案”（即预测零）无法将训练损失降至最低，这就是为什么对那些使用更高的权重会更好的原因。

请注意，最佳权重不一定是0.89和0.11。有时，您可能需要尝试采用对数或平方根（或任何满足的权重one_weight > zero_weight）以使其起作用。

Answer 2

您可以使用sklearn模块自动为每个类计算权重，如下所示：

# Import
import numpy as np
from sklearn.utils import class_weight

# Example model
model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))

# Use binary crossentropy loss
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Calculate the weights for each class so that we can balance the data
weights = class_weight.compute_class_weight('balanced',
                                            np.unique(y_train),
                                            y_train)

# Add the class weights to the training                                         
model.fit(x_train, y_train, epochs=10, batch_size=32, class_weight=weights)

请注意，的输出class_weight.compute_class_weight()是一个numpy数组，如下所示：[2.57569845 0.68250928]。

Answer 3

使用class_weightsinmodel.fit略有不同：它实际上更新样本而不是计算加权损失。

我还发现，当作为 TFDataset 或生成器发送到 model.fit 时，class_weights以及sample_weights，在 TF 2.0.0 中会被忽略。x我相信它在 TF 2.1.0+ 中已修复。

这是我的多热编码标签的加权二元交叉熵函数。

import tensorflow as tf
import tensorflow.keras.backend as K
import numpy as np
# weighted loss functions


def weighted_binary_cross_entropy(weights: dict, from_logits: bool = False):
    '''
    Return a function for calculating weighted binary cross entropy
    It should be used for multi-hot encoded labels

    # Example
    y_true = tf.convert_to_tensor([1, 0, 0, 0, 0, 0], dtype=tf.int64)
    y_pred = tf.convert_to_tensor([0.6, 0.1, 0.1, 0.9, 0.1, 0.], dtype=tf.float32)
    weights = {
        0: 1.,
        1: 2.
    }
    # with weights
    loss_fn = get_loss_for_multilabels(weights=weights, from_logits=False)
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.6067193, shape=(), dtype=float32)

    # without weights
    loss_fn = get_loss_for_multilabels()
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.52158177, shape=(), dtype=float32)

    # Another example
    y_true = tf.convert_to_tensor([[0., 1.], [0., 0.]], dtype=tf.float32)
    y_pred = tf.convert_to_tensor([[0.6, 0.4], [0.4, 0.6]], dtype=tf.float32)
    weights = {
        0: 1.,
        1: 2.
    }
    # with weights
    loss_fn = get_loss_for_multilabels(weights=weights, from_logits=False)
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(1.0439969, shape=(), dtype=float32)

    # without weights
    loss_fn = get_loss_for_multilabels()
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.81492424, shape=(), dtype=float32)

    @param weights A dict setting weights for 0 and 1 label. e.g.
        {
            0: 1.
            1: 8.
        }
        For this case, we want to emphasise those true (1) label, 
        because we have many false (0) label. e.g. 
            [
                [0 1 0 0 0 0 0 0 0 1]
                [0 0 0 0 1 0 0 0 0 0]
                [0 0 0 0 1 0 0 0 0 0]
            ]

        

    @param from_logits If False, we apply sigmoid to each logit
    @return A function to calcualte (weighted) binary cross entropy
    '''
    assert 0 in weights
    assert 1 in weights

    def weighted_cross_entropy_fn(y_true, y_pred):
        tf_y_true = tf.cast(y_true, dtype=y_pred.dtype)
        tf_y_pred = tf.cast(y_pred, dtype=y_pred.dtype)

        weights_v = tf.where(tf.equal(tf_y_true, 1), weights[1], weights[0])
        weights_v = tf.cast(weights_v, dtype=y_pred.dtype)
        ce = K.binary_crossentropy(tf_y_true, tf_y_pred, from_logits=from_logits)
        loss = K.mean(tf.multiply(ce, weights_v))
        return loss

    return weighted_cross_entropy_fn