Java – Use MediaExtractor and MediaCodec to skip frames during video playback

Use MediaExtractor and MediaCodec to skip frames during video playback… here is a solution to the problem.

Use MediaExtractor and MediaCodec to skip frames during video playback

I’m using MediaExtractor/MediaCodec to decode the video and render it to TextureView. As a template, I used the following code: https://github.com/vecio/MediaCodecDemo/blob/master/src/io/vec/demo/mediacodec/DecodeActivity.java

I want to be able to play videos at 2x speed. Luckily, media encoding/decoding is fast enough that I can do this by having MediaCodec decode every frame and then render only every other frame to the screen. However, this is not a good solution, especially if you want to increase the playback with an arbitrary value. For example, at 10x, the codec cannot decode frames fast enough to play every 10 frames at 30 fps.

So I want to control playback by calling MediaExtractor.advance() multiple times to skip frames that don’t need to be decoded. For example:

        ...
        mDecoder.queueInputBuffer(inIndex, 0, sampleSize, mExtractor.getSampleTime(), 0);
        for (i = 0; i < playbackSpeedIncrease; i++) {
               mExtractor.advance();
        }
        ...

Using this code, the extractor should theoretically only fetch every nth frame, where n is defined by the variable “playbackSpeedIncrease”. For example, if n = 5, this should advance to frames 1-4 and extract only frame 5.

However, this does not work in practice. When I run this code, the image rendered on the screen is distorted: Enter image description here

Does anyone know why this is? Any suggestions for a better way to increase playback of videos at any speed?

Solution

You usually can’t do this for AVC videos.

Encoded video has “key” (or “sync” or “I”) frames that contain the full image, but there are “differences” between the frames between the keyframes and the previous frames. You can find some articles on Wikipedia about video encoding methods, such as this one.

If you’ve ever seen a video fast-forward smooth but rewind clunky, such as on TiVo, here’s why: the video decoder plays forward quickly, but plays the I-frames in reverse, holding them on screen long enough to get the desired rate. At “faster” forward/reverse, it flattens because the device is just playing I-frames. You can do something similar by looking at the SAMPLE_FLAG_SYNC flag on the frame obtained from MediaExtractor.

Generally speaking, you can only play videos at the speed decoded by the device, or only keyframes. (You might do better if you know enough about the layout of a particular video in a particular encoding, such as playing I and P instead of B, but I’m not sure this will work in AVC.) )

The frequency of the I-frame is determined by the video encoder. It tends to be one to two per second, but if you get videos from different sources, then you can expect a difference in GOP size (groups of pictures, i.e. how many frames are between I-frames).

Related Problems and Solutions