如何使用OpenGLES 2.0实时在libgdx中在背景上渲染Android的YUV-NV21相机图像？
Ayb*_*gür 20 shader camera android opengl-es-2.0 libgdx
与Android不同,我对GL/libgdx相对较新.我需要解决的任务,即将Android相机的YUV-NV21预览图像实时渲染到libgdx内部的屏幕背景是多方面的.以下是主要关注点:
Android相机的预览图像仅保证在YUV-NV21空间中(在类似的YV12空间中,U和V通道不交错但分组).假设大多数现代设备提供隐式RGB转换是非常错误的,例如最新的Samsung Note 10.1 2014版本仅提供YUV格式.由于除非是RGB格式,否则无法在OpenGL中向屏幕绘制任何内容,因此必须以某种方式转换颜色空间.
libgdx文档(集成libgdx和设备摄像头)中的示例使用Android表面视图,该视图低于使用GLES 1.1绘制图像的所有内容.自2014年3月初以来,由于过时而几乎所有设备都支持GLES 2.0,因此从libgdx中删除了OpenGLES 1.x支持.如果您使用GLES 2.0尝试相同的样本,则在图像上绘制的3D对象将是半透明的.由于背后的表面与GL无关,因此无法真正控制.禁用BLENDING/TRANSLUCENCY不起作用.因此,渲染此图像必须完全在GL中完成.
这必须是实时完成的,因此色彩空间转换必须非常快.使用Android位图的软件转换可能会太慢.
作为侧面功能,必须可以从Android代码访问摄像机图像,以执行除在屏幕上绘制之外的其他任务,例如通过JNI将其发送到本机图像处理器.
问题是,这项任务如何正确,尽可能快地完成？
Ayb*_*gür 74
简短的回答是将相机图像通道(Y,UV)加载到纹理中,并使用自定义片段着色器将这些纹理绘制到网格上,该着色器将为我们进行色彩空间转换.由于这个着色器将在GPU上运行,因此它将比CPU快得多,当然比Java代码快得多.由于此网格是GL的一部分,因此可以在其上方或下方安全地绘制任何其他3D形状或精灵.
我从这个答案/sf/answers/1233098751/开始解决了这个问题.我理解了使用以下链接的一般方法:如何使用OpenGL ES的摄像机视图,它是为Bada编写的,但原理是相同的.那里的转换公式有点奇怪,所以我用维基百科文章YUV转换为/从RGB替换它们.
以下是导致解决方案的步骤:
YUV-NV21解释
来自Android相机的实时图像是预览图像.默认颜色空间(以及两个保证颜色空间中的一个)是YUV-NV21,用于相机预览.这种格式的解释非常分散,所以我将在此简要解释一下:
图像数据由(宽x高)×3/2字节组成.第一个宽度x高度字节是Y通道,每个像素有1个亮度字节.以下(宽度/ 2)x(高度/ 2)x 2 =宽度x高度/ 2字节是UV平面.每两个连续字节是V,U(按照NV21规范的顺序)2×2 = 4个原始像素的色度字节.换句话说,UV平面的尺寸是(宽度/ 2)×(高度/ 2)像素,并且在每个维度中被下采样2倍.另外,U,V色度字节是交错的.
这是一个非常好的图像,解释了YUV-NV12,NV21只是U,V字节翻转:
如何将此格式转换为RGB？
正如问题中所述,如果在Android代码中完成此转换,则需要花费太多时间才能生效.幸运的是,它可以在GL着色器内完成,该着色器在GPU上运行.这将使它运行得非常快.
一般的想法是将我们图像的通道作为纹理传递到着色器,并以RGB转换的方式渲染它们.为此,我们必须首先将图像中的通道复制到可以传递给纹理的缓冲区:
byte[] image;
ByteBuffer yBuffer, uvBuffer;
...
yBuffer.put(image, 0, width*height);
yBuffer.position(0);
uvBuffer.put(image, width*height, width*height/2);
uvBuffer.position(0);
然后,我们将这些缓冲区传递给实际的GL纹理:
/*
* Prepare the Y channel texture
*/
//Set texture slot 0 as active and bind our texture object to it
Gdx.gl.glActiveTexture(GL20.GL_TEXTURE0);
yTexture.bind();
//Y texture is (width*height) in size and each pixel is one byte;
//by setting GL_LUMINANCE, OpenGL puts this byte into R,G and B
//components of the texture
Gdx.gl.glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE,
width, height, 0, GL20.GL_LUMINANCE, GL20.GL_UNSIGNED_BYTE, yBuffer);
//Use linear interpolation when magnifying/minifying the texture to
//areas larger/smaller than the texture size
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_MIN_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_MAG_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_WRAP_S, GL20.GL_CLAMP_TO_EDGE);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_WRAP_T, GL20.GL_CLAMP_TO_EDGE);
/*
* Prepare the UV channel texture
*/
//Set texture slot 1 as active and bind our texture object to it
Gdx.gl.glActiveTexture(GL20.GL_TEXTURE1);
uvTexture.bind();
//UV texture is (width/2*height/2) in size (downsampled by 2 in
//both dimensions, each pixel corresponds to 4 pixels of the Y channel)
//and each pixel is two bytes. By setting GL_LUMINANCE_ALPHA, OpenGL
//puts first byte (V) into R,G and B components and of the texture
//and the second byte (U) into the A component of the texture. That's
//why we find U and V at A and R respectively in the fragment shader code.
//Note that we could have also found V at G or B as well.
Gdx.gl.glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE_ALPHA,
width/2, height/2, 0, GL20.GL_LUMINANCE_ALPHA, GL20.GL_UNSIGNED_BYTE,
uvBuffer);
//Use linear interpolation when magnifying/minifying the texture to
//areas larger/smaller than the texture size
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_MIN_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_MAG_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_WRAP_S, GL20.GL_CLAMP_TO_EDGE);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D,
GL20.GL_TEXTURE_WRAP_T, GL20.GL_CLAMP_TO_EDGE);
接下来,我们渲染我们之前准备的网格(覆盖整个屏幕).着色器将负责渲染网格上的绑定纹理:
shader.begin();
//Set the uniform y_texture object to the texture at slot 0
shader.setUniformi("y_texture", 0);
//Set the uniform uv_texture object to the texture at slot 1
shader.setUniformi("uv_texture", 1);
mesh.render(shader, GL20.GL_TRIANGLES);
shader.end();
最后,着色器接管将纹理渲染到网格的任务.实现实际转换的片段着色器如下所示:
String fragmentShader =
"#ifdef GL_ES\n" +
"precision highp float;\n" +
"#endif\n" +
"varying vec2 v_texCoord;\n" +
"uniform sampler2D y_texture;\n" +
"uniform sampler2D uv_texture;\n" +
"void main (void){\n" +
" float r, g, b, y, u, v;\n" +
//We had put the Y values of each pixel to the R,G,B components by
//GL_LUMINANCE, that's why we're pulling it from the R component,
//we could also use G or B
" y = texture2D(y_texture, v_texCoord).r;\n" +
//We had put the U and V values of each pixel to the A and R,G,B
//components of the texture respectively using GL_LUMINANCE_ALPHA.
//Since U,V bytes are interspread in the texture, this is probably
//the fastest way to use them in the shader
" u = texture2D(uv_texture, v_texCoord).a - 0.5;\n" +
" v = texture2D(uv_texture, v_texCoord).r - 0.5;\n" +
//The numbers are just YUV to RGB conversion constants
" r = y + 1.13983*v;\n" +
" g = y - 0.39465*u - 0.58060*v;\n" +
" b = y + 2.03211*u;\n" +
//We finally set the RGB color of our pixel
" gl_FragColor = vec4(r, g, b, 1.0);\n" +
"}\n";
Please note that we are accessing the Y and UV textures using the same coordinate variable v_texCoord, this is due to v_texCoord being between -1.0 and 1.0 which scales from one end of the texture to the other as opposed to actual texture pixel coordinates. This is one of the nicest features of shaders.
The full source code
Since libgdx is cross-platform, we need an object that can be extended differently in different platforms that handles the device camera and rendering. For example, you might want to bypass YUV-RGB shader conversion altogether if you can get the hardware to provide you with RGB images. For this reason, we need a device camera controller interface that will be implemented by each different platform:
public interface PlatformDependentCameraController {
void init();
void renderBackground();
void destroy();
}
该界面的Android版本如下(假设实时摄像机图像为1280x720像素):
public class AndroidDependentCameraController implements PlatformDependentCameraController, Camera.PreviewCallback {
private static byte[] image; //The image buffer that will hold the camera image when preview callback arrives
private Camera camera; //The camera object
//The Y and UV buffers that will pass our image channel data to the textures
private ByteBuffer yBuffer;
private ByteBuffer uvBuffer;
ShaderProgram shader; //Our shader
Texture yTexture; //Our Y texture
Texture uvTexture; //Our UV texture
Mesh mesh; //Our mesh that we will draw the texture on
public AndroidDependentCameraController(){
//Our YUV image is 12 bits per pixel
image = new byte[1280*720/8*12];
}
@Override
public void init(){
/*
* Initialize the OpenGL/libgdx stuff
*/
//Do not enforce power of two texture sizes
Texture.setEnforcePotImages(false);
//Allocate textures
yTexture = new Texture(1280,720,Format.Intensity); //A 8-bit per pixel format
uvTexture = new Texture(1280/2,720/2,Format.LuminanceAlpha); //A 16-bit per pixel format
//Allocate buffers on the native memory space, not inside the JVM heap
yBuffer = ByteBuffer.allocateDirect(1280*720);
uvBuffer = ByteBuffer.allocateDirect(1280*720/2); //We have (width/2*height/2) pixels, each pixel is 2 bytes
yBuffer.order(ByteOrder.nativeOrder());
uvBuffer.order(ByteOrder.nativeOrder());
//Our vertex shader code; nothing special
String vertexShader =
"attribute vec4 a_position; \n" +
"attribute vec2 a_texCoord; \n" +
"varying vec2 v_texCoord; \n" +
"void main(){ \n" +
" gl_Position = a_position; \n" +
" v_texCoord = a_texCoord; \n" +
"} \n";
//Our fragment shader code; takes Y,U,V values for each pixel and calculates R,G,B colors,
//Effectively making YUV to RGB conversion
String fragmentShader =
"#ifdef GL_ES \n" +
"precision highp float; \n" +
"#endif \n" +
"varying vec2 v_texCoord; \n" +
"uniform sampler2D y_texture; \n" +
"uniform sampler2D uv_texture; \n" +
"void main (void){ \n" +
" float r, g, b, y, u, v; \n" +
//We had put the Y values of each pixel to the R,G,B components by GL_LUMINANCE,
//that's why we're pulling it from the R component, we could also use G or B
" y = texture2D(y_texture, v_texCoord).r; \n" +
//We had put the U and V values of each pixel to the A and R,G,B components of the
//texture respectively using GL_LUMINANCE_ALPHA. Since U,V bytes are interspread
//in the texture, this is probably the fastest way to use them in the shader
" u = texture2D(uv_texture, v_texCoord).a - 0.5; \n" +
" v = texture2D(uv_texture, v_texCoord).r - 0.5; \n" +
//The numbers are just YUV to RGB conversion constants
" r = y + 1.13983*v; \n" +
" g = y - 0.39465*u - 0.58060*v; \n" +
" b = y + 2.03211*u; \n" +
//We finally set the RGB color of our pixel
" gl_FragColor = vec4(r, g, b, 1.0); \n" +
"} \n";
//Create and compile our shader
shader = new ShaderProgram(vertexShader, fragmentShader);
//Create our mesh that we will draw on, it has 4 vertices corresponding to the 4 corners of the screen
mesh = new Mesh(true, 4, 6,
new VertexAttribute(Usage.Position, 2, "a_position"),
new VertexAttribute(Usage.TextureCoordinates, 2, "a_texCoord"));
//The vertices include the screen coordinates (between -1.0 and 1.0) and texture coordinates (between 0.0 and 1.0)
float[] vertices = {
-1.0f, 1.0f, // Position 0
0.0f, 0.0f, // TexCoord 0
-1.0f, -1.0f, // Position 1
0.0f, 1.0f, // TexCoord 1
1.0f, -1.0f, // Position 2
1.0f, 1.0f, // TexCoord 2
1.0f, 1.0f, // Position 3
1.0f, 0.0f // TexCoord 3
};
//The indices come in trios of vertex indices that describe the triangles of our mesh
short[] indices = {0, 1, 2, 0, 2, 3};
//Set vertices and indices to our mesh
mesh.setVertices(vertices);
mesh.setIndices(indices);
/*
* Initialize the Android camera
*/
camera = Camera.open(0);
//We set the buffer ourselves that will be used to hold the preview image
camera.setPreviewCallbackWithBuffer(this);
//Set the camera parameters
Camera.Parameters params = camera.getParameters();
params.setFocusMode(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO);
params.setPreviewSize(1280,720);
camera.setParameters(params);
//Start the preview
camera.startPreview();
//Set the first buffer, the preview doesn't start unless we set the buffers
camera.addCallbackBuffer(image);
}
@Override
public void onPreviewFrame(byte[] data, Camera camera) {
//Send the buffer reference to the next preview so that a new buffer is not allocated and we use the same space
camera.addCallbackBuffer(image);
}
@Override
public void renderBackground() {
/*
* Because of Java's limitations, we can't reference the middle of an array and
* we must copy the channels in our byte array into buffers before setting them to textures
*/
//Copy the Y channel of the image into its buffer, the first (width*height) bytes are the Y channel
yBuffer.put(image, 0, 1280*720);
yBuffer.position(0);
//Copy the UV channels of the image into their buffer, the following (width*height/2) bytes are the UV channel; the U and V bytes are interspread
uvBuffer.put(image, 1280*720, 1280*720/2);
uvBuffer.position(0);
/*
* Prepare the Y channel texture
*/
//Set texture slot 0 as active and bind our texture object to it
Gdx.gl.glActiveTexture(GL20.GL_TEXTURE0);
yTexture.bind();
//Y texture is (width*height) in size and each pixel is one byte; by setting GL_LUMINANCE, OpenGL puts this byte into R,G and B components of the texture
Gdx.gl.glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE, 1280, 720, 0, GL20.GL_LUMINANCE, GL20.GL_UNSIGNED_BYTE, yBuffer);
//Use linear interpolation when magnifying/minifying the texture to areas larger/smaller than the texture size
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_MIN_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_MAG_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_WRAP_S, GL20.GL_CLAMP_TO_EDGE);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_WRAP_T, GL20.GL_CLAMP_TO_EDGE);
/*
* Prepare the UV channel texture
*/
//Set texture slot 1 as active and bind our texture object to it
Gdx.gl.glActiveTexture(GL20.GL_TEXTURE1);
uvTexture.bind();
//UV texture is (width/2*height/2) in size (downsampled by 2 in both dimensions, each pixel corresponds to 4 pixels of the Y channel)
//and each pixel is two bytes. By setting GL_LUMINANCE_ALPHA, OpenGL puts first byte (V) into R,G and B components and of the texture
//and the second byte (U) into the A component of the texture. That's why we find U and V at A and R respectively in the fragment shader code.
//Note that we could have also found V at G or B as well.
Gdx.gl.glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE_ALPHA, 1280/2, 720/2, 0, GL20.GL_LUMINANCE_ALPHA, GL20.GL_UNSIGNED_BYTE, uvBuffer);
//Use linear interpolation when magnifying/minifying the texture to areas larger/smaller than the texture size
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_MIN_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_MAG_FILTER, GL20.GL_LINEAR);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_WRAP_S, GL20.GL_CLAMP_TO_EDGE);
Gdx.gl.glTexParameterf(GL20.GL_TEXTURE_2D, GL20.GL_TEXTURE_WRAP_T, GL20.GL_CLAMP_TO_EDGE);
/*
* Draw the textures onto a mesh using our shader
*/
shader.begin();
//Set the uniform y_texture object to the texture at slot 0
shader.setUniformi("y_texture", 0);
//Set the uniform uv_texture object to the texture at slot 1
shader.setUniformi("uv_texture", 1);
//Render our mesh using the shader, which in turn will use our textures to render their content on the mesh
mesh.render(shader, GL20.GL_TRIANGLES);
shader.end();
}
@Override
public void destroy() {
camera.stopPreview();
camera.setPreviewCallbackWithBuffer(null);
camera.release();
}
}
主要应用程序部分确保init()在开始时调用一次,在renderBackground()每个渲染周期destroy()调用,最后调用一次:
public class YourApplication implements ApplicationListener {
private final PlatformDependentCameraController deviceCameraControl;
public YourApplication(PlatformDependentCameraController cameraControl) {
this.deviceCameraControl = cameraControl;
}
@Override
public void create() {
deviceCameraControl.init();
}
@Override
public void render() {
Gdx.gl.glViewport(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT);
//Render the background that is the live camera image
deviceCameraControl.renderBackground();
/*
* Render anything here (sprites/models etc.) that you want to go on top of the camera image
*/
}
@Override
public void dispose() {
deviceCameraControl.destroy();
}
@Override
public void resize(int width, int height) {
}
@Override
public void pause() {
}
@Override
public void resume() {
}
}
唯一的Android特定部分是以下非常简短的主要Android代码,您只需创建一个新的Android特定设备相机处理程序并将其传递给主libgdx对象:
public class MainActivity extends AndroidApplication {
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
AndroidApplicationConfiguration cfg = new AndroidApplicationConfiguration();
cfg.useGL20 = true; //This line is obsolete in the newest libgdx version
cfg.a = 8;
cfg.b = 8;
cfg.g = 8;
cfg.r = 8;
PlatformDependentCameraController cameraControl = new AndroidDependentCameraController();
initialize(new YourApplication(cameraControl), cfg);
graphics.getView().setKeepScreenOn(true);
}
}
它有多快？
我在两台设备上测试了这个例程.虽然测量值在帧间不是恒定的,但可以观察到一般的轮廓:
三星Galaxy Note II LTE - (GT-N7105):拥有ARM Mali-400 MP4 GPU.
- 渲染一帧需要大约5-6毫秒,偶尔会每隔几秒跳转到大约15毫秒
- 实际渲染线(
mesh.render(shader, GL20.GL_TRIANGLES);)始终需要0-1 ms - 两种纹理的创建和绑定总共需要1-3毫秒
- ByteBuffer副本通常总共需要1-3毫秒,但有时会跳到7毫秒左右,这可能是因为图像缓冲区在JVM堆中移动了
三星Galaxy Note 10.1 2014 - (SM-P600):拥有ARM Mali-T628 GPU.
- 渲染一帧大约需要2-4毫秒,罕见的跳跃大约需要6-10毫秒
- 实际渲染线(
mesh.render(shader, GL20.GL_TRIANGLES);)始终需要0-1 ms - 两种纹理的创建和绑定总共需要1-3毫秒,但每隔几秒就跳到6-9毫秒左右
- ByteBuffer副本通常总共需要0-2 ms,但很少会跳到6ms左右
如果您认为可以使用其他方法更快地制作这些配置文件,请随时分享.希望这个小教程有所帮助.
- 这是绝对一流的解释和示例,谢谢! (3认同)
为了最快、最优化的方式，只需使用常见的 GL 扩展
//Fragment Shader
#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES u_Texture;
比在Java中
surfaceTexture = new SurfaceTexture(textureIDs[0]);
try {
someCamera.setPreviewTexture(surfaceTexture);
} catch (IOException t) {
Log.e(TAG, "Cannot set preview texture target!");
}
someCamera.startPreview();
private static final int GL_TEXTURE_EXTERNAL_OES = 0x8D65;
在 Java GL 线程中
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, textureIDs[0]);
GLES20.glUniform1i(uTextureHandle, 0);
颜色转换已经为您完成。您可以在片段着色器中做任何您想做的事情。
它根本不是 Libgdx 解决方案，因为它依赖于平台。您可以在包装器中初始化与平台相关的内容，然后将其发送到 Libgdx Activity。
希望这可以为您节省一些研究时间。