Imx8m-mini H264 Encoder Data Spikes

[DanielePettenuzzo]

When encoding video on the imx8m-mini to go over an RF link we use 2mbps as our default bitrate. On average we see 2mbps but we have very high spikes in the data coming from the encoder. This causes issues on the RF side and therefore some stuttering when we decode the video on the opposite side of the RF link. We have done some comparisons with other encoders (x264enc software encoder and the hardware encoder on a raspberry pi). With these other encoders the data rate is much more constant, making the video much smoother after it goes through our RF link.

We tried to tune the gop-size parameter but this didn't have much of an effect on the spikes.

Any clues on how we could reduce the spikes?

This is a capture of the data from wireshark:

And this is how you can reproduce our pipeline using an open source rtsp server I found online: https://github.com/sfalexrog/gst-rtsp-launch

This is what I did to get it working on ubuntu 20.04 running on the ixm8m-mini. We are using a UVC/USB camera that provides raw video on /dev/video1.

apt update
apt install -y cmake libgstrtspserver-1.0-dev libglib2.0-dev libgstreamer1.0-dev git
git clone https://github.com/sfalexrog/gst-rtsp-launch.git
cd gst-rtsp-launch/
mkdir build
cd build
cmake ..
make -j4
cd src

./gst-rtsp-launch -p 8553 -e stream1 "( v4l2src device=/dev/video1 ! video/x-raw,framerate=30/1 ! queue ! videoconvert ! queue ! imxvpuenc_h264 bitrate=2048 ! queue ! rtph264pay name=pay0 pt=96 mtu=1200 )"

And then to fetch the video you can use any player with the following url:

rtsp://<IP>:8553/stream1

[dv1]

So far I've had no luck in trying to find out what is going on. A very similar issue can be found here. Fixes for this problem most likely need to be incorporated into libimxvpuapi. A potentially relevant detail is that if HRD is enabled in the Hantro encoder, then only strict CBR is possible - dynamic bitrate adjustments won't work. But perhaps HRD + CBR would yield a more consistent behavior with fewer spikes.

If you are able / willing to experiment, modify the code in your local git clone of libimxvpuapi, file imxvpuapi2/imxvpuapi2_imx8m_hantro_encoder.c, replace OMX_Video_ControlRateVariable with OMX_Video_ControlRateConstant at line 402. Then check how it behaves. In my tests, I did not see much of a change, but I may not have caught all cases.

It is also possible that the lower level APIs that lie below the Hantro OMX layer would provide extra configuration options that would help. Switching to using those APIs directly would be a big change though - effectively it would be a rewrite of the encoder.

[dv1]

Fixing this may turn out to require substantial changes in libimxvpuapi, meaning that I need to acces the underlying Hantro APIs directly instead of Hantro's OpenMAX IL layer. I will post updates here later.

[ndufresne]

Personally, I thin kthe graph looks normal, typical spikes caused by key-frames. If you can't afford these spikes, disable keyframes, and request them when strictly needed. I don't think the HW supports intra-refresh. In fact, if you look at the reference software (leaked here, https://github.com/CliveLau1990/imx-vpu-hantro) it seems like the rate control is a copy paste from libvpx, so with the same key-frame distance, the software encoder in libvpx should give you the same. If you try to match your bitrate on keyframe, image quality will decrease too drasticly. In general, its preferred to have larger keyframes, since this will be cached up over a "number of frame" window, as a better key-frame means smaller deltas.

[dv1]

@ndufresne So for transmissions over a medium that strictly requires constant bitrates (due to narrow bandwidth for example) you limit the keyframes to the absolute minimum? But then, a receiver joining in later won't get a picture for a long time.

I would have expected keyframes to be more aggressively quantized, but yeah, I guess the keyframes would then look too degraded.

The H1 encoder does seem to support intra refresh though. At least I see numerous references to "cyclic intra refresh" in the hantro code. I'll look into that, since it would help reduce bitrate variability.

[ndufresne]

Strict bitrate limit is never set to 1 frame duration. Bitrate control works by time window. Typically, STBs have a windows of multiple seconds to hit the perfect bitrate. They also introduce very large latency. In video streaming, we recommend a 100-200ms window, so at 30fps you will be able to smooth your bitrate over a windows of 3 to 6 frames.

As an example, in GStreamer, you will configure the jitterbuffer latency to that window, so that if a frame takes longer to transmit, you can buffer it and not endup with late frames.

[ndufresne]

The H1 encoder does seem to support intra refresh though. At least I see numerous references to "cyclic intra refresh" in the hantro code. I'll look into that, since it would help reduce bitrate variability.

cyclic intra refresh is the most common type of progressive refresh. I believe Venus HW from Qualcom have support for the mainline and there is V4L2 controls too.

[dv1]

I've been making efforts to finally deal with remaining issues as much as possible before making a release. Unfortunately, the fix for this won't make it into the upcoming libimxvpuapi 2.3.0 release, since there is no way to improve the H1 encoder backend without a rewrite. I'll focus on that for the release after 2.3.0.

[sandeshworld]

Are there any updates on a fix for this issue?