Java – From android.media.Image to Mat

I’m using the Android Camera2 API to get frames from the camera. I can get an ImageReader that reads the Image object using the acquireLatestImage() method.

Since I need to process the acquired frames, I have to convert each Image object to a Mat object. I guess this is a very common problem, so I was hoping to find a convenient OpenCV way to do this. But I didn’t find any.

Do you know how to get Mat objects from the camera frame using Android Camera2?
Thank you

This depends on what ImageFormat you set for the ImageReader class. The following code assumes that it is using the JPEG format.

Image image = reader.acquireLatestImage();

Mat buf = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC1);

ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
buf.put(0, 0, bytes);

// Do note that Highgui has been replaced by Imgcodecs for OpenCV 3.0 and above
Mat mat = Highgui.imdecode(buf, IMREAD_COLOR);

image.close();

Note that imdecode is very slow. I’m not entirely sure, but I think converting YUV_420_888 to Mat objects would be faster, but I haven’t figured out how to do it myself. I see this post trying to convert YUV to Mat,