在Object Detection API中使用数据扩充选项时出错

Question

我正在尝试使用.config文件中的data_augmentation_options来训练网络,特别是ssd_mobilenet_v1,但是当我激活选项random_adjust_brightness时,我会很快得到下面粘贴的错误消息(我在步骤110000之后激活该选项).

我尝试减少默认值:

optional float max_delta=1 [default=0.2];

但结果是一样的.

知道为什么吗？图像是来自png文件的RGB(来自博世小交通灯数据集).

INFO:tensorflow:global step 110011: loss = 22.7990 (0.357 sec/step)
INFO:tensorflow:global step 110012: loss = 47.8811 (0.401 sec/step)
2017-11-16 11:02:29.114785: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.114895: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.114969: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.115043: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
     [[Node: CheckNumerics = CheckNumerics[T=DT_FLOAT, message="LossTensor is inf or nan.", _device="/job:localhost/replica:0/task:0/device:CPU:0"](total_loss)]]
2017-11-16 11:02:29.115112: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
...

编辑:我找到的解决方法是这个.inf或nan是丢失的,所以检查/object_detection/core/preprocessor.py中的函数进行亮度随机化:

def random_adjust_brightness(image, max_delta=0.2):
  """Randomly adjusts brightness.

  Makes sure the output image is still between 0 and 1.

  Args:
    image: rank 3 float32 tensor contains 1 image -> [height, width, channels]
           with pixel values varying between [0, 1].
    max_delta: how much to change the brightness. A value between [0, 1).

  Returns:
    image: image which is the same shape as input image.
    boxes: boxes which is the same shape as input boxes.
  """
  with tf.name_scope('RandomAdjustBrightness', values=[image]):
    image = tf.image.random_brightness(image, max_delta)
    image = tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)
    return image

假设图像值必须介于0.0和1.0之间.图像是否可能实际到达0均值甚至不同范围？在这种情况下,剪辑会破坏它们并导致失败.长话短说:我评论了剪切线,它正在工作(我们将看到结果).

Answer 1

通常，获取LossTensor is inf or nan. : Tensor had NaN values是由于边界框/注释中的错误造成的（来源： https: //github.com/tensorflow/models/issues/1881）。

我知道博世小型交通灯数据集有一些超出图像尺寸的注释。例如，该数据集中图像的高度为 720 像素，但某些边界框的高度坐标大于 720。这种情况很常见，因为每当记录序列的汽车经过红绿灯时，某些红绿灯都是可见的，还有一部分被切断了。

我知道这不是您问题的准确答案，但希望它可以帮助您了解出现问题的可能原因。也许删除超出图像尺寸的注释将有助于解决问题；但是，我正在处理同样的问题，只是我没有使用图像预处理。在同一个数据集上，我LossTensor is inf or nan. : Tensor had NaN values每大约 8000 步就会遇到错误。