FFmpeg: The Open Source Multimedia Ecosystem

The concept of free and open-source software has gone through a long journey, from misunderstanding and controversy to widespread acceptance. After several years of open-source collaboration, Tencent MPS has also achieved significant accomplishments. Among them, Tencent MPS has made notable contributions to important multimedia open-source communities such as FFmpeg and SRS.

FFmpeg is one of the most famous open-source projects in the audio and video domain, often referred to as the Swiss Army Knife of multimedia and serving as the foundation for numerous audio and video applications. The release of FFmpeg included a significant number of important updates, many of which were contributed by the Tencent MPS team, showcasing interesting and valuable features. In addition to FFmpeg, the Tencent MPS team actively leads or participates in the development of various open-source audio and video projects such as SRS, SRT, and VLC, embodying the principle of mutual benefit and growth between the cloud and open-source communities.

What is FFmpeg

FFmpeg was initially initiated by French programmer Fabrice Bellard in 2000. It is feature-rich and capable of meeting various audio and video processing and development needs. In addition to its rich functionality, FFmpeg also incorporates a significant amount of acceleration optimization. It fully utilizes SIMD assembly acceleration and multi-threading acceleration on CPUs, and actively introduces heterogeneous acceleration such as GPUs to achieve optimal performance. After more than 20 years of iteration and extensive testing, FFmpeg also stands out in terms of code robustness and compatibility in media data processing.

Currently, the main components of FFmpeg include command-line tools and basic libraries.

1. Fftools command-line tool

• ffmpeg: a multimedia data processing tool that includes features such as format conversion and filter processing. It can automatically perform complex editing functions.
• ffprobe: a multimedia data analysis tool.
• ffplay: A simple yet powerful player.

2. Basic library

• libavutil: Foundation library that includes data structures, string handling, mathematical calculations, memory management, logging system, and more.
• libavcodec: Audio, video, and subtitle encoding/decoding library that includes over 800 codecs. It also includes related functionalities like parsers and bitstream filters.
• libavformat: Implementation of transport protocols, container formats, and demuxing/muxing. It supports over 500 media container formats and more than 50 transport protocols.
• libswscale: Image scaling and format conversion library.
• libswresample: Audio format conversion and resampling library.
• libavfilter: Audio and video filtering framework that implements various functionalities like cropping, watermarking, mixing, etc.
• libavdevice: Library for audio and video capture and rendering output.
• libpostproc: Video post-processing library.

Tencent MPS and the FFmpeg community

Since 2019, Tencent MPS has been actively contributing to the FFmpeg community, with over 800 patches submitted. More than 350 of these patches have been merged into the FFmpeg mainline. Tencent MPS also has two FFmpeg maintainers who actively participate in development and community building. They are deeply involved in patch reviews, GSOC mentoring, new feature development, bug fixes, and other community activities. This demonstrates the symbiotic relationship and collaborative development between the cloud and the community. Tencent MPS is among the largest contributors to the FFmpeg community internationally.

In addition to numerous bug fixes and feature enhancements, the Tencent MPS team has implemented and optimized various aspects of FFmpeg. This includes mobile hardware acceleration for encoding and decoding, support for codec standards, integration of the SRT transmission protocol, and enhancements to the MP4 container format.

1. Mobile hardware encoding and decoding acceleration

The Tencent MPS team has helped FFmpeg implement MediaCodec encoding on the Android platform and MediaCodec decoding for the AV1 format (supporting both Java MediaCodec and NDK MediaCodec). This allows developers to utilize MediaCodec encoding and decoding through FFmpeg's command-line interface, enabling zero-copy transcoding. For example:

ffmpeg -hwaccel mediacodec \
            -init_hw_device mediacodec=mediacodec,create_window=1 \
            -i input.mp4 -c:a copy -c:v hevc_mediacodec \
            -tag:v hvc1 output.mp4

The zero-copy transcoding method provides the best performance but does not support various filters available in libavfilter. On the other hand, using the decoding-to-buffer approach introduces some performance loss but makes the overall media processing pipeline more flexible.

ffmpeg -hwaccel mediacodec -i input.mp4 \
            -c:a copy -c:v hevc_mediacodec \
            -tag:v hvc1 output.mp4

If libavcodec is integrated using a library and OpenGL is used for filter processing, it can achieve the best optimization in terms of performance and speed. Due to certain limitations of OpenGL, FFmpeg itself does not implement OpenGL filters but instead supports Vulkan.

The significance of hardware encoding and decoding support in FFmpeg's libavcodec lies in:

• Providing a unified encoding and decoding interface, allowing applications to use a single API to support various platforms and hardware acceleration methods.
• Enabling easy implementation of fallback strategies, allowing flexible switching between FFmpeg's software and hardware codecs.
• Seamless integration with the complete media processing pipeline, ensuring smooth transitions between encoding, decoding, and bitstream filtering.
• Connecting with libavformat, transmission, unpacking, decoding, encoding, packaging and streaming are all done in one go.

Here's an example: for videos with non-16-aligned width and height resolutions, Android MediaCodec encoding has compatibility issues. FFmpeg's h264_metadata/hevc_metadata bitstream filters can be used to work around these compatibility problems. The solution has already been integrated into the MediaCodec wrapper. For more details, please refer to: https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/mediacodecenc.c

VideoToolbox serves as the video encoding and decoding acceleration interface for macOS/iOS systems. Tencent MPS has integrated the FFmpeg VideoToolbox H.265 transparent video encoding pipeline. Here's an example of encoding the first video as the base layer and the second video as the alpha layer:

./ffmpeg -i base.mp4 -i alpha.mp4 \
     -filter_complex '[0:v][1:v]alphamerge[v2]' \
     -map '[v2]' -pix_fmt bgra \
     -c:v hevc_videotoolbox -alpha_quality 0.3 \
     -tag:v hvc1 output.mp4

The support for transparent video decoding of H.265 is still under continuous iteration and will be further improved in the future. The open-source full pipeline for H.265 is gradually being perfected. Chrome 105 already supports H.265 playback. You can use FFmpeg to push SRT streams to an SRS server, convert SRT to HTTP-TS, and then directly play the live stream using VLC/ffplay or Chrome/mpegts.js.

2. SRT (Secure Reliable Transport) Protocol

Tencent MPS, as a member of the SRT Alliance, has participated in the development and customization of libsrt and maintains the FFmpeg libsrt wrapper and VLC player's libsrt wrapper. This enriches the ecosystem of SRT. The excellent resilience to weak networks and low-latency characteristics of SRT can be utilized in FFmpeg, gaining widespread application in transmission acceleration, especially in live streaming scenarios.

SRT is widely used in low-latency live streaming scenarios, such as low-latency switching. It can achieve delays of 300 to 500ms. You can choose to use FFmpeg/OBS/vMix to push SRT streams to an SRS server and then use ffplay/vMix to pull the stream or watch it on a webpage using WebRTC. In addition to the above solutions, Tencent Cloud's TMIO can also be used. It not only optimizes the latency issues specifically for remote transmission but also provides the following features to better adapt to different network conditions:

• Connection optimization: Optimizes connection time through techniques like 0RTT and connection reuse.
• Retransmission optimization: Solves the issue of excessive retransmission rates through parameter optimization and sequence disorder control strategies, compared to the open-source version of libsrt.
• Congestion control optimization: Incorporates the BBR congestion control algorithm to adapt to different network scenarios with packet loss and high BDP (Bottleneck Bandwidth and Round-trip propagation time).
• Multi-link transmission: Optimizes multi-network transmission mechanisms for live streaming scenarios, effectively addressing the problem of transmission quality being affected by the instability of a single network.
• Terminal proxy acceleration: Implements targeted protocol acceleration for live streaming push scenarios, improving the quality of upstream streaming.

3. Enhancements to the MP4 container format

MP4 is one of the most widely used container formats. In addition to fixing numerous compatibility bugs, Tencent MPS has enhanced support for the fMP4 format and MP4 Common Encryption. They have implemented the ISO/IEC 23003-5 standard for uncompressed audio in MPEG-4 file format and the ISO/IEC 23001-8 standard for coding-independent code points (during the patch review process). These enhancements have enabled the broader application of the MP4 format in both on-demand and live streaming scenarios.

Conclusion

Multimedia technology is a field that heavily relies on accumulated experience and involves intricate details. As a hub for multimedia technology, FFmpeg has attracted numerous top developers in the industry to collaborate and jointly develop the "Swiss Army Knife" of multimedia technology. Projects such as FFmpeg and SRS have lowered technical barriers in audio and video development, promoting the prosperity of industries such as live streaming and short video. Tencent MPS team stands on the shoulders of giants and will continue to actively contribute to the open-source community, working alongside the community. Welcome to contact us for consultation.