如何实现神经网络修剪？

Question

我在keras中训练了一个模型，并且正在考虑修剪我的全连接网络。我对如何修剪图层有些迷惑。

《学习有效的神经网络的权重和连接》一书的作者说，它们为图层的阈值权重添加了掩码。我可以尝试做同样的事情，并微调训练好的模型。但是，它如何减少模型大小和计算数量？

Answer 1

根据评论中的讨论，这里是修剪神经网络的一层（权重矩阵）的方法。该方法的本质是k%根据其范数选择最小的权重（矩阵的元素），并将其设置为零。这样，可以将对应的矩阵视为稀疏矩阵，并且我们可以执行密集稀疏矩阵乘法，如果修剪了足够多的权重，该乘法会更快。

def weight_pruning(w: tf.Variable, k: float) -> tf.Variable:
    """Performs pruning on a weight matrix w in the following way:

    - The absolute value of all elements in the weight matrix are computed.
    - The indices of the smallest k% elements based on their absolute values are selected.
    - All elements with the matching indices are set to 0.

    Args:
        w: The weight matrix.
        k: The percentage of values (units) that should be pruned from the matrix.

    Returns:
        The unit pruned weight matrix.

    """
    k = tf.cast(tf.round(tf.size(w, out_type=tf.float32) * tf.constant(k)), dtype=tf.int32)
    w_reshaped = tf.reshape(w, [-1])
    _, indices = tf.nn.top_k(tf.negative(tf.abs(w_reshaped)), k, sorted=True, name=None)
    mask = tf.scatter_nd_update(tf.Variable(tf.ones_like(w_reshaped, dtype=tf.float32), name="mask", trainable=False), tf.reshape(indices, [-1, 1]), tf.zeros([k], tf.float32))

    return w.assign(tf.reshape(w_reshaped * mask, tf.shape(w)))

上面的方法修剪单个连接（权重）时，下面的方法从权重矩阵修剪整个神经元。即，该方法k%基于欧几里得范数选择最小的神经元（权重矩阵的列），并将其设置为零。

def unit_pruning(w: tf.Variable, k: float) -> tf.Variable:
    """Performs pruning on a weight matrix w in the following way:

    - The euclidean norm of each column is computed.
    - The indices of smallest k% columns based on their euclidean norms are selected.
    - All elements in the columns that have the matching indices are set to 0.

    Args:
        w: The weight matrix.
        k: The percentage of columns that should be pruned from the matrix.

    Returns:
        The weight pruned weight matrix.

    """
    k = tf.cast(
        tf.round(tf.cast(tf.shape(w)[1], tf.float32) * tf.constant(k)), dtype=tf.int32
    )
    norm = tf.norm(w, axis=0)
    row_indices = tf.tile(tf.range(tf.shape(w)[0]), [k])
    _, col_indices = tf.nn.top_k(tf.negative(norm), k, sorted=True, name=None)
    col_indices = tf.reshape(
        tf.tile(tf.reshape(col_indices, [-1, 1]), [1, tf.shape(w)[0]]), [-1]
    )
    indices = tf.stack([row_indices, col_indices], axis=1)

    return w.assign(
        tf.scatter_nd_update(w, indices, tf.zeros(tf.shape(w)[0] * k, tf.float32))
    )

最后，该Github存储库将通过此处介绍的修剪方法对MNIST数据集进行实验。