1 VideoMost Research. White Paper VIDEO CONSUMES OVER 70% OF TOTAL INTERNET BANDWIDTH Videos have become an integral part of our daily life. Video is the most effective form of advertising. Every time we enjoy a movie on our TV or watch a funny video on our smartphone or video conference while commuting to the office, video codecs are working behind the scenes to make this possible. According to the Cisco White Paper: “Visual Networking Index: Forecast and Trends, 2017-2022”, by 2022 more than 80% of internet traffic will be videos. It’s not surprising that the industry’s demand for enhanced video codecs is growing year by year. VIDEO NEEDS EFFECTIVE CODECS Video codecs are two-part (encoder and decoder) compression tools that allow content distributors and creators (like Netflix, Apple and Amazon Twitch), internet companies (like Facebook, Google and Snap) and conferencing vendors (like Microsoft, Cisco and Amazon Chime) to condense a video for delivery across the internet via a process called video encoding. Codecs enable us to so easily stream Facebook videos and FaceTime loved ones, even with limited bandwidth on mobile. Thanks to codecs, Netflix manages to stream more than 97,000 hours of content every minute. And in order to get those streams to you — no matter the device — Netflix must use both new and tried-and-true codecs. Codecs allow to tightly compress a bulky video for delivery and storage and later decompress it for viewing. Video encoding refers to the process of converting raw video into a digital format and container that’s compatible with many devices. When it comes to streaming, videos are often compressed from gigabytes of data down to megabytes of data. Encoding is done by the sender and decoding is done by the receiver. We are now able to enjoy any kind of video, almost everywhere, on different devices, by simply pressing the play button. This seems to happen seamlessly, but what we don’t get to see is the great complexity behind this simple gesture: a wide range of tools – from image processing and partitioning to motion compensation – work together to compress video data and, at the same time, achieve the highest video quality at substantially lower costs. Innovation needs tools, and their evolution over time – together with the increasing performance they achieve – lead to different codecs, tailored to address specific needs of different industries – from broadcasting to distributed medical consultations – all to the benefit of consumers and those that provide devices and services to consumers. The 60 years history of video compression standardization (starting from TV broadcasting and to mobile video streaming and multi-way communication) has been dominated by efforts to maximize compression ratio, where an increasing variety of services, the growing popularity of HD video, and the emerging beyond-HD formats (e.g. 4K resolution) are creating even stronger needs for compression capabilities. Mobile devices now need to receive and display HD video as well. The advent of stereo and multi-view displays (and corresponding camera systems) further increases the amount of data. As increasing video traffic is seen as a major cause of the “spectrum crunch” that could occur in wireless networks, increased video compression ratio can be a factor to mitigate this as well. Moreover, video now comprises over 70% of Internet data traffic worldwide, and its percentage of that traffic is continually increasing. Proprietary and Confidential. © 2020 VideoMost Research 2 CODECS STANDARDISATION International organizations and global professional associations document and thereby make widely available current technological standards, promoting a broad diffusion in the marketplace. Ensuring the world can communicate in the same language is hard, even more when the language is coding. Innovation requires collaboration: each encoded video must play on each enabled device. That’s why international and independent standard-setting bodies are so important within the innovation ecosystem: their work makes the interoperability between content and devices possible, resulting in products which are able to talk to each other. Standard-setting bodies, like ITU (International Telecommunication Union) and ISO (International Standardization Organization), include experts from firms, users, interest groups and governments. Their scope is to create open, interoperable and universal international standards. ITU and ISO have also specific branches dedicated to the creation of new video codec standards, respectively MPEG (Moving Picture Expert Group) and VCEG (Video Coding Experts Group), and their work over decades (both alone and as part of joint teams) brought to life the most used video codecs, from MPEG-2 to H.264/AVC and H.265/HEVC. Standards for television. Standard development in broadcasting scenarios was critical for the success of television: interoperability ensures that a consumer is able to buy a TV from any manufacturer and be sure the equipment will decode television signals from any broadcaster in the world. Standardization bodies and industry fora including ITU, ISO, DVB, ATSC, have been working hard on the development of standards with global relevance such as MPEG-2, AVC and HEVC, enabling the creation of horizontal markets in the field of multimedia services, continuing improving and developing them in order to provide the proper support to the needs of consumers and the media industry. Hundreds of companies accessed this standard and elected to implement the technology in their products. Without this fully-functional ecosystem, TV manufacturers would have had to implement each single transmission technology adopted in each country to ensure interoperability within the broadcast infrastructure. This would have required long, complex and expensive implementations, leading to an increased price being passed to the final user for access to the technology. Mobile communication standards. Mobile telephony has reshaped the entire system. The players in the telecommunication industry have addressed together the complexity behind mobile communications by developing sets of standards to build a system intended to be deployed on a global scale. As a result, fixed and mobile networks are now connected and make up an unite communication system allowing everyone to enjoy communication services everywhere in the world through our gadgets. THE MOST COMMON AND FUTURE CODECS H.264 / AVC STANDARD : 82% OF THE MARKET, Developed in 2003 According to “2019 Global Media Format Report” by Encoding.com: https://1yy04i3k9fyt3vqjsf2mv610yvm-wpengine.netdna-ssl.com/files/2019-Global-Media-Formats-Report.pdf The majority (82%) of encoding output today takes the form of H.264 files, also referred to as AVC (Advanced Video Coding). This widely supported codec was developed by the International Telecommunications Union (ITU) and the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group (MPEG) in 2003, 17 years ago. H.264 also has significant penetration into markets outside of streaming, such as Blu-ray disks and cable broadcasting. H.264 plays on virtually any device, delivers quality video streams, and comes with the least concerns surrounding royalties. Proprietary and Confidential. © 2020 VideoMost Research 3 H.265 / HEVC STANDARD : 12% OF THE MARKET, Developed in 2013 H.265 also called HEVC (High Efficiency Video Coding) is a video compression standard developed jointly by ITU-T VCEG and ISO/IEC MPEG through their Joint Collaborative Team on Video Coding (JCT-VC) as the successor to H.264. The first version of HEVC was finalized in January 2013, 7 years ago, and the second version in July 2014 provided additional functionality. The HEVC project was launched to achieve major savings for comparable visual quality relative to the bit rate needed by the widely used H.264 standard. H.265 aims to improve compression efficiency and support 8K resolution. After 10 years of innovation following release of H.264, the new H.265 codec generates up to 50% smaller file volumes than H.264, thus decreasing the bandwidth required to view these streams. According to “2019 Global Media Format Report” by Encoding.com H.265 is now 12% of the global market. Technology and key features. HEVC has been designed to address essentially all existing applications of previous standards and to particularly focus on two key issues: increased video resolution and increased use of parallel processing architectures. The syntax of HEVC is generic, and its design elements could also be attractive for other application domains that have not used the preceding standards. The video coding layer of HEVC employs essentially the same “hybrid” approach (inter- / intra-picture prediction and 2D transform plus entropic coding) used in all video compression standards since H.261. A prediction residual is computed from previously decoded information (either previously decoded pictures for inter-picture motion compensated based prediction or previously decoded samples from the same picture for intra-picture spatial prediction). The residual is then processed by a block transform, and the transform frequency coefficients are quantized and entropy coded. Side information data such as motion vectors and mode switching parameters are also encoded and transmitted. PATENTS Innovation takes years. In the case of video codecs, it may take 10 years from preliminary work to standard publication and the patentable technologies that result from innovation, with expenditures of substantial resources. All standard codecs are protected by hundreds of patents from dozens of different global players. In order to ship a product with HEVC support, you need to acquire licenses from at least four patent pools (MPEG LA, HEVC Advance, Technicolor, and Velos Media) as well as numerous other companies, many of which do not offer standard licensing terms (instead requiring you to negotiate terms), which can potentially cost hundreds of millions of dollars (and that’s after the recent drastic cuts to HEVC royalty fees). The H.265 codec for compressing video isn’t getting market traction because costs of licensi ng patents for it are both high and unclear. Proprietary and Confidential. © 2020 VideoMost Research 4 With some proposed standards, project members agree on licensing terms before work begins, rather than leaving the job to lawyers sometimes years after a standard is set. Others implement fair, reasonable, and non-discriminatory (FRAND) licensing, an often-used concept that has proved to be dangerously vague. There is yet no single method that guarantees a worthwhile return for developing new technologies while preventing abuse from opportunists and monopolies, given the industry’s diversity in business models, technologies and markets. And some see in standards-essential patents (SEPs) a Gordian knot that cannot be untied or cut. Patent wars in video codecs have raged off and on for years. With the combination of the limited monopoly granted in a patent, and the de facto monopoly of a standard, the normal competitive forces that would determine a reasonable royalty don’t apply. Tod ay, SEPs are key to the high-profile Apple / Qualcomm dispute over cellular, and SEPs are the hot spot for H.265, too. Tensions reached a peak when three patent pools emerged for HEVC. One of them, Velos Media, has not yet disclosed its licensing terms, and more than a dozen large patent holders have not joined any of the pools. That’s a complete nightmare to navigate, and the uncertainty has kept the previous codec, H.264, dominant with an estimated 82% market share. The newer H.265 has about 12% of the market, despite the fact it has been available for some seven years and is technically superior to H.264. The core problem is the MPEG group behind video codec standards focuses solely on technology, leaving others to sort out patent licensing terms. MPEG creates video standards that matter but they fall down by relying on FRAND. The crisis got so bad that the old MPEG business model is now broke. While those steep royalties were already problematic for products like Google Chrome, Opera, Netflix, Amazon Video, Cisco WebEx, Skype, and others, they completely exclude HEVC as an option for projects like Mozilla Firefox, both on an economic level (Firefox simply cannot afford to waste hundreds of millions of dollars on royalties and hundreds of man hours negotiating all the necessary licensing agreements), on a practical level (Firefox needs to be royalty-free in order to ship in many Free and Open Source projects), Proprietary and Confidential. © 2020 VideoMost Research 5 and on an ideological level (Mozilla believes in a free and open web, and that isn’t possible if you promote patent-encumbered standards). Those issues prevented Firefox (and Chromium) from even including native H.264 playback on many platforms until a couple years ago (with it still requiring a plugin on Linux), and will likely prevent Firefox from supporting HEVC until after its patents expire in the 2030s (or possibly even later). Even to this day, Firefox only supports H.264 natively thanks to Cisco couple of years ago offering to pay all of the licensing costs for Mozilla through OpenH264, in order to standardize H.264 for streaming across the market until the next generation codec was ready. Uncertainties surrounding royalties have stifled adoption. Specifically, content distributors are frustrated by the lacking transparency into what they’ll have to pay when using this codec. GOOGLE VP9 CODEC, Developed in 2013 And that opened the door for VP9 codec. Google released VP9 as a royalty-free alternative to HEVC. It actually took Google buying On2 company with 15 years of video codec development history for $116M in 2010 and investing 3 more years in the codec R&D. Every Android phone and Chrome browser supports VP9, as well as Google’s video platform YouTube. By being royalty free, VP9 was able to be implemented on any platform or service that wanted it, and it is seeing substantial hardware acceleration support as well. Beyond YouTube using it on any device that can support it (as the reduced bandwidth usage is a huge cost savings for YouTube), the WebM container (which supports VP9 video and audio in either Opus or Vorbis) is also replacing .gifs with silent videos that are substantially smaller on sites like imgur and gfycat, it’s being used throughout Wikipedia, it has been adopted by Skype (who were a driving force behind Opus’ voice codec development), and it’s even being adopted by Netflix (starting with their downloads for offline viewing, and moving to their regular streaming in the future). VP9 offers almost similar quality at the same bitrate as H.265/HEVC. While the majority of browsers support it, Apple devices do not. Alliance for Open Media AV1 CODEC, Developed in 2018 VP9 codec is thought of as AV0, or an earlier version of AV1. For the time being, it’s also a better alternative to AV1 since more devices support it. However, VP9 alone was not enough. Google wants even better compression, especially for YouTube and Duo, where a tiny increase in video compression can result in huge cost savings and a major improvement in user experience. So Google put together a plan to rapidly update their VPx codec line, like they do with Chrome and some of their other products. Google announced that they planned to release VP10 in 2016, and then would release an update every 18 months to ensure a steady progression. It got to the point where Google even started to release code for VP10, and then suddenly Google announced the cancellation of VP10 and formed the Alliance for Open Media (AOMedia) together with Apple, Amazon, Netflix, Microsoft, Cisco, Mozilla and others who were frustrated about the royalties associated with H.265. The goal? Create an open-source, royalty-free alternative called AV1. Proprietary and Confidential. © 2020 VideoMost Research 6 The Alliance for Open Media includes everyone from processor designers (AMD, ARM, Broadcom, Chips&Media, Intel, Nvidia, etc.) to browser developers (Google, Microsoft, and Mozilla) to streaming and videoconferencing services (Adobe, Amazon, BBC R&D, Cisco, Netflix, YouTube, etc.). Those companies are expected to bring their substantial strength to play in rolling out AV1 support, with the first streaming services expected to be ready within just 6 months after the bitstream format is finalized, and the first hardware decoders are expected to ready within 12 months. That alone may bring substantial hardware support for AV1 fairly quickly, however if everything lines up, we may even see partial hardware acceleration backported to some already existing hardware, like what happened with VP9, which could be a huge boost for compatibility. Despite HEVC and VP9 being the two latest codecs, they weren’t the only ones. Cisco was developing Thor for use in their videoconferencing products, and Xiph was developing Daala (a codec designed to be substantially different from all previous codecs, in order to prevent any possibility of patent claims). All three codecs (Thor, Daala, and VP9/VP10) were looking quite promising, but the split efforts were stifling their development and adoption, so the three organizations came together and merged their codecs into one (AV1), and created the Alliance for Open Media to further the development and adoption of this joint codec. AV1 aims to take the best parts of each of those three codecs, and merge them into a royalty-free package that anyone can implement. Video streaming is a massive chunk of total internet traffic, and even a couple percent improvement in compression can have massive effects on both the network as a whole, and on user experience for that specific application. Proprietary and Confidential. © 2020 VideoMost Research 7 AV1 aims to make it possible to have decent quality video on lower throughput connections (opening up video streaming for more situations and more markets), and will enable even better quality than before on high throughput connections. It was designed with use over cellular networks in mind, with AV1 bringing massive improvements in how well they scale as connection speeds change, not to mention the higher resolutions, higher frame rates, expanded color space, HDR support (which will be vital for services like Netflix, YouTube, and Amazon Video to take full advantage of the new displays on modern smartphones, now being able to take advantage of Netflix’s recently-added HDR support in mobile. AV1’s improvements will bring better live casting of events, better video chatting (via WebRTC), smaller files for local storage, previously unheard of quality for video streaming (such as high quality 4K HDR while on a cellular network), and potentially other uses that we haven’t yet thought of, especially when paired with the improved speeds of 5G mobile networks and 802.11ax WiFi. Of course, the groups promoting HEVC won’t sit idly by while this is happening. They have already begun making threats about starting patent litigation against AV1 once it is released, and the Alliance for Open Media are going to great lengths to make sure that it does not happen. They are performing an extensive legal code review of AV1 to make sure it does not infringe on any patents held by MPEG LA, HEVC Advance, Technicolor, Velos Media, and others. That form of code review was highly successful for VP8 and VP9, both of which survived all legal challenges. MPEG LA’s legal actions against VP8 and VP9 were seen as potentially not having any legal grounding and instead being purely anti-competitive. The DoJ was investigating MPEG LA’s actions until they agreed to drop the lawsuit and give Google permission to sub-license out MPEG LA’s patent pool to any users of VP8 or VP9. While we likely will see similar attempts at stopping AV1, Google’s substantially expanded patent pool and the substantially increased number of companies supporting the codec (thanks to the Alliance for Open Media) should both go a long way towards ensuring that they are dealt with in short order. AV1 touts itself as being 30 percent more efficient than HEVC / VP9, but these claims still need to be verified by independent sources. On the other hand, growing complexity of AV1 development has made the codec 1000 times hungrier for computing power than it was in its early days. AV1 is patent royalty-free but this approach has not been immune from attacks from patent holders, nor has it been sufficiently motivating to attract some top technology developers, especially companies with significant licensing businesses. It will take some time before AV1 hardware decoding capabilities are integrated on a mass scale. While leaders at Netflix, Facebook, Google, and more are planning to make the move to AV1, playback limitations can’t be ignored. Even Apple devices are lacking support for the codec. “The one disadvantage at this point is that AV1 is just new,” explains Anne Aaron, director of encoding technologies at Netflix. “H.264 is a really good codec that’s been developed over more than ten years — and AV1 is new, so there’s still kinks in implementations.” In other words, the industry is still in flux when it comes to AV1. STANDARD H.266/VVC CODEC (won't be finalized before end of 2020) Intended to usurp H.265, VVC (Versatile Video Coding) comes with the same royalty issues as its predecessor. According to Beamr’s chief technology officer Dror Gill, “It’s okay to pay royalties as long as you know how much you need to pay and when. With H.264, it was very clear how much you need to pay, there was one Proprietary and Confidential. © 2020 VideoMost Research 8 body collecting all the royalties, and this became the world’s most prominent video codec. The same can happen with VVC, if they get their act together before releasing the standard”. Still a nascent technology, we’re waiting to see how adoption of VVC pans out. This codec isn’t intended to ship until the end of 2020 and unforeseen challenges on the licensing front are up in the air. SPIRIT ULTRA PERFORMANCE VIDEO CODEC BASED ON MACHINE LEARNING So the higher is video compression rate delivered by a codec, the less bandwidth (or bitrate) is required to transmit the content. All current developments are based on more and more complex coding algorithms. This traditional approach to coding has now practically reached its fundamental limit. According to MPEG-LA, improvement in video compression rate over past 30 years has drastically slowed down so that after 10 years of development the new generation of video coding standard brings only up to 50% of improvement: Standard MPEG-1 Required bitrate vs previous Year of approval 1992 MPEG-2 MPEG-4 Visual H.264 / AVC H. 265 / HEVC 25% less 25% less 30% less 50% less 1994 1998 2003 2013 Even though over 70% of internet traffic today is digital media, the way images and video are represented and transmitted has not evolved much in the past 20 years (apart from Pied Piper's Middle-Out algorithm). It has been challenging for existing commercial compression algorithms to adapt to the growing demand and the changing landscape of transmission settings and requirements. The available codecs are ''onesize-fits-all'': they are hard-coded, and cannot be customized to particular use cases beyond high-level hyperparameter tuning. The old paradigm to codec development created by Claude Shannon on the base of rate-distortion theory, has reached its fundamental limit. SPIRIT DSP has developed a fundamentally new concept of video coding that is based on trained neural networks, which increases video’s compression ratio by about 3 times, not 30%, compared to traditional video codecs, by reducing statistical, psychophysical and applied redundancy. In other words, the new paradigm proposed by SPIRIT DSP is all about removing a bulk of data that were mandatory for keeping in the old concept while handling sufficient minimum of information during its transmitting and storing so that the original video could be restored by a trained ANN. The shift in paradigm proposed by SPIRIT DSP has been recognized by granting the company a patent No. RU 2 698 414 C1. SPIRIT DSP has also filed an international application No. PCT/RU2019/050108 that has been accepted and registered with the priority date September 21, 2018. Proprietary and Confidential. © 2020 VideoMost Research 9 In the below table we compare1 Data Compression Ratio (CR) of SPIRIT VideoMost Research codec with the most advanced codecs discussed above assuming the subjective metric of “Mean Opinion Score” (MOS) is equal to 4 that correlates with high subjective video quality. Codec Fees, USD SPIRIT VideoMost Н.264 / AVC H.265 / HEVC H.266 / VVC Google VP9 AOM AV1 $0.1 per unit*/annual cap $9.75M $0.2** per unit/annual cap $9.75M** $0.2*** per unit/annual cap $25M*** N/A Free Free Data compression ratio / range 240 / 150-700 45 / 10-60 55 / 10-100 65 / 10-150 50 / 10-80 65 / 10-150 Required bitrate, Mbps 0.1-1 0.5-6 0.3-5 0.25-5 0.4-6 0.2-4 *) MPEG-LA only: A “unit” means a decoder (example: a TV set), an encoder (example: a camera) or a product consisting of one decoder and one encoder (example: a smartphone). **) MPEG-LA only: current term through December 31, 2020. Royalty rates for specific license grants may increase by 10% at 2 renewals. Annual royalty caps are not subject to the 10% limitation. ***) MPEG-LA only: current term through December 31, 2020. Royalties for license grant or royalty cap may increase by 20% at 3 each renewal. SPIRIT DSP’s innovative video codec: pre-processes a video stream or video file; transforms a video into a sketch form; adapts the frame rate to the dynamics of the objects in a scene; performs frame-by-frame entropy coding; decodes sketch frames and restores the original frame rate; restores video stream using a trained neural network; provides an optional search for an object of interest, faces detection, stylistic coloring, etc. The advantages of SPIRIT DSP’s innovation: significant reduction in costs of video data transmissions; streaming of highly compressed video over narrowband communication channels. COMPANY SPIRIT DSP is developing and licensing software products to the global market for more than 25 years. Between 1999 and 2012 different voice, audio and video codecs were the main product line of the company. SPIRIT communication software products are licensed to/built into popular products from more than 250 global technology leaders including Apple, Avaya, Adobe, AT&T, Blizzard, BT, China Mobile, Ericsson, HP, HTC, Huawei, Korea Telecom, LG, Microsoft, Oracle, Samsung, Skype, Toshiba, Viber, ZTE, etc. More than 1 billion people in more than 100 countries use SPIRIT software. SPIRIT VideoMost® is a software videoconferencing server product with PC and mobile clients that support all the popular communication protocols and international standards, including H.323, H.239, SIP, BFCP, XMPP, H.264, G.7xx, WebRTC, and delivers interoperability with the legacy videoconferencing hardware. VideoMost works in popular browsers (IE, Firefox, Chrome, Safari) and is available on both iOS and Android mobile platforms. It also provides a full range of tools for teamwork, including mobile messaging, document sharing and polling. 1 https://www.researchgate.net/publication/327638411_Comparison_of_Compression_Efficiency_between_HEVCH265_VP9_and_AV1_based_on_Subjective_Quality_Assessments https://netflixtechblog.com/performance-comparison-of-video-coding-standards-an-adaptive-streaming-perspective-d45d0183ca95 http://compression.ru/video/codec_comparison/introduction.html https://bitmovin.com/vvc-video-codec/ 2 https://www.mpegla.com/wp-content/uploads/avcweb.pdf 3 https://www.mpegla.com/wp-content/uploads/HEVCweb.pdf Proprietary and Confidential. © 2020 VideoMost Research
