TensorFlow 对象检测 API 评估训练性能

Question

TensorFlow 对象检测 API 评估训练性能

Maz*_*har 4 python machine-learning object-detection tensorflow object-detection-api

我一直在我自己的数据集上使用Tensorflow Object Detection API。在训练时，我想知道神经网络从训练集中学习的情况如何。因此，我想对训练集和评估集进行评估，并在训练期间分别获得准确率 (mAP)。

我的配置文件：

model {
  faster_rcnn {
    num_classes: 50
    image_resizer {
      fixed_shape_resizer {
        height: 960
        width: 960
      }
    }
    number_of_stages: 3
    feature_extractor {
      type: 'faster_rcnn_resnet101'
      first_stage_features_stride: 8
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 8
        width_stride: 8
      }
    }
    first_stage_atrous_rate: 2
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.00999999977648
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.699999988079
    first_stage_max_proposals: 100
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        conv_hyperparams {
          op: CONV
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.00999999977648
            }
          }
        }
        predict_instance_masks: true
        mask_height: 33
        mask_width: 33
        mask_prediction_conv_depth: 0
        mask_prediction_num_conv_layers: 4
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.300000011921
        iou_threshold: 0.600000023842
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
    second_stage_mask_prediction_loss_weight: 4.0
  }
}
train_config: {
  batch_size: 1
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.003
          schedule {
            step: 3000
            learning_rate: 0.00075
          }
          schedule {
            step: 6000
            learning_rate: 0.000300000014249
          }
          schedule {
            step: 15000
            learning_rate: 0.000075
          }
          schedule {
            step: 18000
            learning_rate: 0.0000314249
          }
          schedule {
            step: 900000
            learning_rate: 2.99999992421e-05
          }
          schedule {
            step: 1200000
            learning_rate: 3.00000010611e-06
          }
        }
      }
      momentum_optimizer_value: 0.899999976158
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "./mask_rcnn_resnet101_atrous_coco/model.ckpt"
  from_detection_checkpoint: true
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}
train_input_reader: {
  label_map_path: "./map901_label_map.pbtxt"
  load_instance_masks: true
  mask_type: PNG_MASKS
  tf_record_input_reader {
    input_path: ["./my_coco_train.record-?????-of-00005"]
  }
}
eval_config: {
  num_examples: 8000
  max_evals: 100
  num_visualizations: 25
}
eval_input_reader: {
  label_map_path: "./map901_label_map.pbtxt"
  shuffle: false
  load_instance_masks: true
  mask_type: PNG_MASKS
  num_readers: 1
  tf_record_input_reader {
    input_path: ["./my_coco_val.record-?????-of-00001"]
  }
}

Run Code Online (Sandbox Code Playgroud)

我用这些参数运行了脚本

python model_main.py --alsologtostderr \
  --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
  --model_dir=${TRAIN_DIR} \
  --num_train_steps=24000 \
  --sample_1_of_n_eval_on_train_examples=25 \
  --num_eval_steps=100 \
  --sample_1_of_n_eval_examples=1

Run Code Online (Sandbox Code Playgroud)

我认为这将对 Eval 示例进行评估。为了评估训练数据（检查从训练中捕获了多少特征），我已将其添加 --eval_training_data=True到参数中。

我无法随时随地添加“eval_training_data”。我需要进行 2 次不同的培训课程。

有趣的是，添加了“eval_training_data”参数后，我得到了，

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.165
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.281
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.167
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.109
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.164
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.141
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.236

Run Code Online (Sandbox Code Playgroud)

而没有“eval_training_data”我得到

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.168
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.283
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.173
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.049
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.108
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.208
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.170
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.208
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.208
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.056
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.139
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.248

Run Code Online (Sandbox Code Playgroud)

我只是糊涂了。我的问题是：

“eval_training_data”不会强制对象检测 API 在训练集上运行评估吗？
为什么在我的情况下两个分数几乎相同，在某些情况下评估分数更好？
需要添加哪些参数，以便在训练期间分别评估训练和评估集并打印？

Answer 1

Sus*_*ant 5

从我快速查看回购协议可以收集到的是：

eval_training_data 仅对训练集进行评估，并从评估过程中排除 eval 集。所以它只在你的训练集上运行。
分数相同并不是坏事。这实际上很好，并且表明您的模型没有过拟合，如果训练数据分数的评估明显高于评估数据的评估分数，就会出现这种情况。在某些情况下，评估得分较高是因为它必须是一个小得多的数据集，因此即使在少数情况下预测好的或坏的，分数也会有很大的变化。此外，该模型正在学习特征并将其与类相关联，而不是学习示例，因此不要指望它在训练集上表现出色，因为它已经看过所有这些。您的模型在验证集上的表现越好意味着它的泛化性越好。
如果您设置 eval_training_data = True，它实际上已经在单独评估训练集，如果它设置为 false（默认情况下），它只会单独评估 eval 集。我不确定他们是否添加了用于同时评估两者的功能，但是您可以通过在 model_main.py 中进行非常小的更改来实现。做这个补充。它不是干净和优化的，但我想你明白这一点并可以相应地修改它。

flags.DEFINE_boolean('eval_training_data_and_eval_data', False,
                     'This will evaluate botht the training data and evaluation data sequentially')

  if FLAGS.checkpoint_dir:
    if FLAGS.eval_training_data_and_eval_data:

      name = 'training_data'
      input_fn = eval_on_train_input_fn
      if FLAGS.run_once:
        estimator.evaluate(input_fn,
                           steps=None,
                           checkpoint_path=tf.train.latest_checkpoint(
                               FLAGS.checkpoint_dir))
      else:
        model_lib.continuous_eval(estimator, FLAGS.checkpoint_dir, input_fn,
                                  train_steps, name)

      name = 'validation_data'
      # The first eval input will be evaluated.
      input_fn = eval_input_fns[0]
      if FLAGS.run_once:
        estimator.evaluate(input_fn,
                           steps=None,
                           checkpoint_path=tf.train.latest_checkpoint(
                               FLAGS.checkpoint_dir))
      else:
        model_lib.continuous_eval(estimator, FLAGS.checkpoint_dir, input_fn,
                                  train_steps, name)

  else:
    train_spec, eval_specs = model_lib.create_train_and_eval_specs(
        train_input_fn,
        eval_input_fns,
        eval_on_train_input_fn,
        predict_input_fn,
        train_steps,
        eval_on_train_data=False)

    # Currently only a single Eval Spec is allowed.
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])

Run Code Online (Sandbox Code Playgroud)

此外，请确保您也为数据集提供了正确且独特的路径。请注意，如果我们根据验证分数优化超参数，验证分数就会有偏差，并且不再是对泛化的良好估计。为了获得对泛化的正确估计，我们必须计算另一个测试集的分数。

归档时间：	6 年，2 月前
查看次数：	2584 次
最近记录：	6 年，2 月前