如何使用Keras在TensorBoard中显示自定义图像？

Question

我正在研究Keras中的分段问题,我希望在每个训练时代结束时显示分段结果.

我想要一些类似于Tensorflow:如何在Tensorboard中显示自定义图像(例如Matplotlib Plots),但使用Keras.我知道Keras有TensorBoard回调但看起来似乎有限.

我知道这会破坏Keras的后端抽象,但无论如何我对使用TensorFlow后端感兴趣.

是否有可能通过Keras + TensorFlow实现这一目标？

Answer 1

因此,以下解决方案适合我:

import tensorflow as tf

def make_image(tensor):
    """
    Convert an numpy representation image to Image protobuf.
    Copied from https://github.com/lanpa/tensorboard-pytorch/
    """
    from PIL import Image
    height, width, channel = tensor.shape
    image = Image.fromarray(tensor)
    import io
    output = io.BytesIO()
    image.save(output, format='PNG')
    image_string = output.getvalue()
    output.close()
    return tf.Summary.Image(height=height,
                         width=width,
                         colorspace=channel,
                         encoded_image_string=image_string)

class TensorBoardImage(keras.callbacks.Callback):
    def __init__(self, tag):
        super().__init__() 
        self.tag = tag

    def on_epoch_end(self, epoch, logs={}):
        # Load image
        img = data.astronaut()
        # Do something to the image
        img = (255 * skimage.util.random_noise(img)).astype('uint8')

        image = make_image(img)
        summary = tf.Summary(value=[tf.Summary.Value(tag=self.tag, image=image)])
        writer = tf.summary.FileWriter('./logs')
        writer.add_summary(summary, epoch)
        writer.close()

        return

tbi_callback = TensorBoardImage('Image Example')

只需将回调传递给fit或fit_generator.

请注意,您还可以使用model回调内部运行某些操作.例如,您可以在某些图像上运行模型以检查其性能.

Answer 2

基于以上答案和我自己的搜索，我提供以下代码来使用Keras中的TensorBoard完成以下操作：

• 问题设置：预测双目立体匹配中的视差图；
• 用输入的左图像x和地面真实视差图馈入模型gt;
• x在某个迭代时间显示输入和地面真相“ gt”；
• y在某个迭代时间显示模型的输出。

1. 首先，您必须使用制作带修饰的回调类Callback。 Note回调可以通过class属性访问其关联的模型self.model。另外Note：如果要获取并显示模型的输出，则必须使用feed_dict将输入提供给模型。

   from keras.callbacks import Callback
   import numpy as np
   from keras import backend as K
   import tensorflow as tf
   import cv2
   
   # make the 1 channel input image or disparity map look good within this color map. This function is not necessary for this Tensorboard problem shown as above. Just a function used in my own research project.
   def colormap_jet(img):
       return cv2.cvtColor(cv2.applyColorMap(np.uint8(img), 2), cv2.COLOR_BGR2RGB)
   
   class customModelCheckpoint(Callback):
       def __init__(self, log_dir='./logs/tmp/', feed_inputs_display=None):
             super(customModelCheckpoint, self).__init__()
             self.seen = 0
             self.feed_inputs_display = feed_inputs_display
             self.writer = tf.summary.FileWriter(log_dir)
   
       # this function will return the feeding data for TensorBoard visualization;
       # arguments:
       #  * feed_input_display : [(input_yourModelNeed, left_image, disparity_gt ), ..., (input_yourModelNeed, left_image, disparity_gt), ...], i.e., the list of tuples of Numpy Arrays what your model needs as input and what you want to display using TensorBoard. Note: you have to feed the input to the model with feed_dict, if you want to get and display the output of your model. 
       def custom_set_feed_input_to_display(self, feed_inputs_display):
             self.feed_inputs_display = feed_inputs_display
   
       # copied from the above answers;
       def make_image(self, numpy_img):
             from PIL import Image
             height, width, channel = numpy_img.shape
             image = Image.fromarray(numpy_img)
             import io
             output = io.BytesIO()
             image.save(output, format='PNG')
             image_string = output.getvalue()
             output.close()
             return tf.Summary.Image(height=height, width=width, colorspace= channel, encoded_image_string=image_string)
   
   
       # A callback has access to its associated model through the class property self.model.
       def on_batch_end(self, batch, logs = None):
             logs = logs or {} 
             self.seen += 1
             if self.seen % 200 == 0: # every 200 iterations or batches, plot the costumed images using TensorBorad;
                 summary_str = []
                 for i in range(len(self.feed_inputs_display)):
                     feature, disp_gt, imgl = self.feed_inputs_display[i]
                     disp_pred = np.squeeze(K.get_session().run(self.model.output, feed_dict = {self.model.input : feature}), axis = 0)
                     #disp_pred = np.squeeze(self.model.predict_on_batch(feature), axis = 0)
                     summary_str.append(tf.Summary.Value(tag= 'plot/img0/{}'.format(i), image= self.make_image( colormap_jet(imgl)))) # function colormap_jet(), defined above;
                     summary_str.append(tf.Summary.Value(tag= 'plot/disp_gt/{}'.format(i), image= self.make_image( colormap_jet(disp_gt))))
                     summary_str.append(tf.Summary.Value(tag= 'plot/disp/{}'.format(i), image= self.make_image( colormap_jet(disp_pred))))
   
                 self.writer.add_summary(tf.Summary(value = summary_str), global_step =self.seen)
   

2. 接下来，将此回调对象传递fit_generator()给您的模型，例如：

      feed_inputs_4_display = some_function_you_wrote()
      callback_mc = customModelCheckpoint( log_dir = log_save_path, feed_inputd_display = feed_inputs_4_display)
      # or 
      callback_mc.custom_set_feed_input_to_display(feed_inputs_4_display)
      yourModel.fit_generator(... callbacks = callback_mc)
      ...
   

3. 现在，您可以运行代码，并前往TensorBoard主机以查看具有特征的图像显示。例如，这就是我使用上述代码得到的：

   做完了！请享用！