有效实现Java Native Interface Webcam Feed

Question

我正在开发一个项目,该项目从网络摄像头获取视频输入并向用户显示运动区域.我在这个项目中的"beta"尝试是使用Java Media Framework来检索网络摄像头源.通过一些实用功能,JMF可以方便地将网络摄像头帧作为BufferedImages返回,我构建了大量的框架来处理.但是,我很快意识到JMF不再受Sun/Oracle的支持,并且一些较高的网络摄像头分辨率(720p)无法通过JMF接口访问.

我想继续像BufferedImages一样处理帧,并使用OpenCV(C++)来获取视频源.单独使用OpenCV的框架,我发现OpenCV可以很好地有效地返回高清摄像头帧并将它们绘制到屏幕上.

我认为将这些数据提供给Java并实现相同的效率非常简单.我刚刚编写了JNI DLL,将这些数据复制到BufferedImage中并将其返回给Java.但是,我发现我正在进行的数据复制实际上是在阻碍性能.我的目标是30 FPS,但是只需要大约100毫秒就可以将OpenCV返回的char数组中的数据复制到Java BufferedImage中.相反,我看到大约2-5 FPS.

返回帧捕获时,OpenCV提供指向1D char数组的指针.这些数据需要提供给Java,显然我没有时间复制任何数据.

我需要一个更好的解决方案来将这些帧捕获到BufferedImage中.我正在考虑的一些解决方案,我认为其中没有一个非常好(相当肯定它们也会表现不佳):

(1)覆盖BufferedImage,并通过对DLL进行本机调用,从各种BufferedImage方法返回像素数据.(而不是一次进行数组复制,我按照调用代码的请求返回单个像素).请注意,调用代码通常需要图像中的所有像素来绘制图像或对其进行处理,因此这个单独的像素抓取操作将在2D for循环中实现.

(2)指示BufferedImage使用java.nio.ByteBuffer以某种方式直接访问OpenCV返回的char数组中的数据.非常感谢有关如何完成此任务的任何提示.

(3)用C++做一切并忘记Java.好吧,是的,这听起来像是最合乎逻辑的解决方案,但是我没有时间从头开始这个为期数月的项目.

截至目前,我的JNI代码已被编写为返回BufferedImage,但此时我愿意接受返回1D char数组然后将其放入BufferedImage.

顺便说一句......这里的问题是:将1D char数组图像数据复制到BufferedImage的最有效方法是什么？

提供了(低效)代码,我用它从OpenCV中获取图像并复制到BufferedImage:

JNIEXPORT jobject JNICALL Java_graphicanalyzer_ImageFeedOpenCV_getFrame
  (JNIEnv * env, jobject jThis, jobject camera)
{
 //get the memory address of the CvCapture device, the value of which is encapsulated in the camera jobject
 jclass cameraClass = env->FindClass("graphicanalyzer/Camera");
 jfieldID fid = env->GetFieldID(cameraClass,"pCvCapture","I");

 //get the address of the CvCapture device
 int a_pCvCapture = (int)env->GetIntField(camera, fid);

 //get a pointer to the CvCapture device
    CvCapture *capture = (CvCapture*)a_pCvCapture;

 //get a frame from the CvCapture device
 IplImage *frame = cvQueryFrame( capture );

 //get a handle on the BufferedImage class
 jclass bufferedImageClass = env->FindClass("java/awt/image/BufferedImage");
 if (bufferedImageClass == NULL)
 {
  return NULL;
 }

 //get a handle on the BufferedImage(int width, int height, int imageType) constructor
 jmethodID bufferedImageConstructor = env->GetMethodID(bufferedImageClass,"<init>","(III)V");

 //get the field ID of BufferedImage.TYPE_INT_RGB
 jfieldID imageTypeFieldID = env->GetStaticFieldID(bufferedImageClass,"TYPE_INT_RGB","I");

 //get the int value from the BufferedImage.TYPE_INT_RGB field
 jint imageTypeIntRGB = env->GetStaticIntField(bufferedImageClass,imageTypeFieldID);

 //create a new BufferedImage
 jobject ret = env->NewObject(bufferedImageClass, bufferedImageConstructor, (jint)frame->width, (jint)frame->height, imageTypeIntRGB);

 //get a handle on the method BufferedImage.getRaster()
 jmethodID getWritableRasterID = env->GetMethodID(bufferedImageClass, "getRaster", "()Ljava/awt/image/WritableRaster;");

 //call the BufferedImage.getRaster() method
 jobject writableRaster = env->CallObjectMethod(ret,getWritableRasterID);

 //get a handle on the WritableRaster class
 jclass writableRasterClass = env->FindClass("java/awt/image/WritableRaster");

 //get a handle on the WritableRaster.setPixel(int x, int y, int[] rgb) method
 jmethodID setPixelID = env->GetMethodID(writableRasterClass, "setPixel", "(II[I)V"); //void setPixel(int, int, int[])

 //iterate through the frame we got above and set each pixel within the WritableRaster
 jintArray rgbArray = env->NewIntArray(3);
 jint rgb[3];
 char *px;
 for (jint x=0; x < frame->width; x++)
 {
  for (jint y=0; y < frame->height; y++)
  {
   px = frame->imageData+(frame->widthStep*y+x*frame->nChannels);
   rgb[0] = abs(px[2]);  // OpenCV returns BGR bit order
   rgb[1] = abs(px[1]);  // OpenCV returns BGR bit order
   rgb[2] = abs(px[0]);  // OpenCV returns BGR bit order
   //copy jint array into jintArray
   env->SetIntArrayRegion(rgbArray,0,3,rgb); //take values in rgb and move to rgbArray
   //call setPixel()  this is a copy operation
   env->CallVoidMethod(writableRaster,setPixelID,x,y,rgbArray);
  }
 }

 return ret;  //return the BufferedImage
}

Answer 1

使用 NIO ByteBuffer 设法加速该过程。

在 C++ JNI 方面...

JNIEXPORT jobject JNICALL Java_graphicanalyzer_ImageFeedOpenCV_getFrame
  (JNIEnv * env, jobject jThis, jobject camera)
{
    //...

    IplImage *frame = cvQueryFrame(pCaptureDevice);

    jobject byteBuf = env->NewDirectByteBuffer(frame->imageData, frame->imageSize);

    return byteBuf;
}

而在Java方面......

void getFrame(Camera cam)
{
    ByteBuffer frameData = cam.getFrame();   //NATIVE call

    byte[] imgArray = new byte[frame.data.capacity()];
    frameData.get(imgArray); //although it seems like an array copy, this call returns very quickly
    DataBufferByte frameDataBuf = new DataBufferByte(imgArray,imgArray.length);

    //determine image sample model characteristics
    int dataType = DataBuffer.TYPE_BYTE;
    int width = cam.getFrameWidth();
    int height = cam.getFrameHeight();
    int pixelStride = cam.getPixelStride();
    int scanlineStride = cam.getScanlineStride();
    int bandOffsets = new int[] {2,1,0};  //BGR

    //create a WritableRaster with the DataBufferByte
    PixelInterleavedSampleModel pism = new PixelInterleavedSampleModel
    (
        dataType,
        width,
        height,
        pixelStride,
        scanlineStride,
        bandOffsets
    );
    WritableRaster raster = new ImgFeedWritableRaster( pism, frameDataBuf, new Point(0,0) );

    //create the BufferedImage
    ColorSpace cs = ColorSpace.getInstance(ColorSpace.CS_sRGB);
    ComponentColorModel cm = new ComponentColorModel(cs, false, false, Transparency.OPAQUE, DataBuffer.TYPE_BYTE);
    BufferedImage newImg = new BufferedImage(cm,raster,false,null);

    handleNewImage(newImg);
}

使用 java.nio.ByteBuffer，我可以快速寻址 OpenCV 代码返回的 char 数组，而无需（显然）进行大量可怕的数组复制。