All Articles Film Making
The Past and Future of Video Codecs
Iain Anderson on Mon, January 18th 1 comments
If you consider how far video compression has come in a relatively short space of time it's mind-blowing. But,there's more to come. Iain Anderson looks back and forward in terms of video codecs.

Today we take compressed video on our computing devices for granted, but it wasn’t always the case. Once upon a time, even a video the size of a postage stamp on a computer was an amazing thing to behold—even compressing a still image was a big deal once. Here, we’ll take a look back, and then a look forward to what’s coming next.

Before we dive into video, we’ll look at still images, because many compression schemes used by moving images are similar.

Lossless compression

This PNG is the same as the original—nothing but grain all the way down.

This PNG is the same as the original—nothing but grain all the way down.

Back when we didn’t have a whole lot of data to compress, or a whole lot of colors to express ourselves with, compression was usually lossless. Instead of writing a chunk of data for each color: “white, white, white, red, white, white, white” and so on, you can use a scheme like Run-Length Encoding to write something along the lines of “three whites, red, three whites”. This saves a little space, and lossless encoding is still used today in GIF, PNG and some TIFF files. 

How lossy compression works

JPEG throws away data, but discard too much and the image is compromised.

JPEG throws away data, but discard too much and the image is compromised.

Because lossless compression doesn’t usually save a whole lot of space, and the finer image details aren’t always important, lossy compression throws away data to save more space. JPEG breaks the entire image up into 8x8 pixel blocks, figures out the average color of each block, then describes the pixels in each block in relation to that average color. That’s why you see blockiness in high contrast areas of heavily compressed images.

Temporal compression 

The sky doesn’t change much in these shots, but water changes completely from frame to frame, and is usually a torture test.

The sky doesn’t change much in these shots, but water changes completely from frame to frame, and is usually a torture test.

When you throw video into the mix, file sizes become huge. Some professional workflows do indeed use uncompressed video or image sequences, but almost all video is captured and delivered in a compressed way. A key difference is whether the video is compressed as intra-frame (also known as I-Frame, essentially just a series of still images) or inter-frame (where each frame is described in relation to past frames or even future frames). If a scene isn’t changing much, then this inter-frame strategy can save a lot of space, and variations of this are dominant today. Advanced codecs use blocks of variable sizes and predictive algorithms to guess where content will move from frame to frame.

Easy editing

Slower Macs will have an easier time if you optimize, but it’s far from essential.

Slower Macs will have an easier time if you optimize, but it’s far from essential.

A computer will have to work harder to decode inter-frame video (like H.264) than it would to decode intra-frame video (like ProRes 422 or DNxHD), but modern Macs running FCP X or Premiere Pro can handle inter-frame H.264 just fine. Older Macs may struggle, which is why FCP X offers an option to “optimize” video to the easier intra-frame ProRes 422. It’s less efficient, and takes more space on disk, but is easier to interpret. Remember, though, that an intra-frame codec needs a higher bitrate than an inter-frame one to maintain the same quality.

QuickTime 1.0

Not quite version 1.0, but still “back in the day”.

Not quite version 1.0, but still “back in the day”.

Now we’ve got the basics down, let’s go back in time to QuickTime 1.0, released back in 1991. I still remember seeing this for the first time, in a demo from an Apple rep, with tiny videos of Apollo rockets taking off. Here’s a very geeky introduction, and a very early video. It allowed anyone with a color Mac to watch moving video, and to copy and paste videos for simple editing. While digital video capture was still a long way off, this was very obviously the way of the future, and a revelation at the time.

Containers and Codecs

Back when DV tape was king of the hill, you could get glitches like this without using effects.

Back when DV tape was king of the hill, you could get glitches like this without using effects.

QuickTime brought a container (the .mov file, or at the time, a “MooV” file type) which could contain audio, video, or other tracks. A wide range of codecs could be used to compress the video and audio, including Animation, Video and Graphics in version 1.0, and then Cinepak in version 1.5. This new codec became dominant for many years, and you can read more about it here.

Later versions of QuickTime brought newer codecs, including Sorenson Video, and MPEG-1, but it really got interesting in 2001 with the DV codec, DV camcorders, and the iMac DV. Suddenly it was possible to edit home videos on a home computer without external capture devices, and this was a big deal. 

Other systems 

Ah, Motion JPEG A, from 1997.

Ah, Motion JPEG A, from 1997.

Of course, QuickTime wasn’t the only game in town. Microsoft introduced the AVI container format, supporting the same Cinepak, Motion JPEG and DV codecs that QuickTime offered. RealVideo was important for streaming video for a time, as was Flash video, but for a long time it was difficult to provide a single movie file that everyone could play. It was looking a bit crazy there for a while, but standards have eventually, largely, prevailed.

The rise of standards 

A video and an audio track, synced together—finally not hard to do.

A video and an audio track, synced together—finally not hard to do.

QuickTime’s container format was used as the basis for the MPEG-4 container format, released alongside the MPEG-4 codec by MPEG, the Motion Picture Experts Group. Their previous standard-based efforts included MPEG-1 (popular for a while) and MPEG-2, used on DVDs. Oh, and MPEG Audio Layer III—MP3—a very popular audio format you may have heard of. The MPEG-4 video codec eventually evolved into today’s dominant video standard, H.264, also known as MPEG-4 AVC.

Why are standards important? Since most cameras capture data in a flavor of H.264, and most delivery takes place in an H.264 codec, our devices can include hardware-based support for encoding and decoding it. For phones and laptops, that means vastly increased battery life, speeds, or both. That doesn’t mean that there aren’t good reasons for choosing something else, though.

Other codecs and containers

If you’re working with RED footage, each clip is not just a single file, but several.

If you’re working with RED footage, each clip is not just a single file, but several.

When image quality is critical, there are good reasons to capture in a raw format like REDCODE RAW or ARRIRAW, or in a more generic intermediate format like ProRes. These formats can avoid compression artifacts and capture more color information for better results in post production. There are also many more container types in which to package your video: MXF, Matroska, and more, and some are easier to work with than others.

Choosing a non-standard codec or container means you’ll need special software support to be able to edit or view your footage. For advanced workflows, it can be worth the effort, but it’s going to be easier to work with an H.264 MP4 file or a ProRes QuickTime. Sure, a new codec can be more efficient, but it’s disappointingly common for new cameras to use slightly different codecs and/or container formats for apparently no good reason.

Patent Issues

From the MPEG-LA website, if you have more than 100,000 paid subscribers or sell videos longer than 12 minutes, you should pay [http://www.mpegla.com/main/programs/avc/Documents/avcweb.pdf]

From the MPEG-LA website, if you have more than 100,000 paid subscribers or sell videos longer than 12 minutes, you should pay [http://www.mpegla.com/main/programs/avc/Documents/avcweb.pdf]

In the video game world, an otherwise-obscure codec called Bink Video is popular for pre-rendered cutscenes. It’s popular because there’s no per-unit royalty on its use, so it’s much cheaper than MPEG-4 for game developers. Wait, what? Royalties?

Indeed, patent and royalty issues have been problematic for MPEG in recent years. If you sell an MPEG-4 file for profit, you’re meant to pay a per-unit fee, and there are costs for providers of encoding and decoding software too. What if you don’t want to pay, or can’t?

Google tried to offer a free solution with the WebM codec, which is claimed to be patent-free, but it’s not yet been widely adopted outside the open source community. (This is also the case with Ogg Vorbis, an audio codec that competes with MP3.) Phones need H.264, computers make H.264 quickly and easily, Microsoft and Apple pay the playback fees, and for a lot of users, that’s more or less that.

Also tricky: since most of the methods behind video codecs are similar, it’s likely that a new codec will infringe on some part of an MPEG-owned patent. This creates a legally challenging environment for new codecs. 

H.265 and onward 

The iPhone 6 and up use H.265 for FaceTime over cellular, but that’s all.

The iPhone 6 and up use H.265 for FaceTime over cellular, but that’s all.

The latest flavor of MPEG is H.265 (HEVC), which uses larger variable block sizes and better motion prediction to achieve higher quality at the same data rate as H.264, or the same quality at approximately half the data rate. (Here are some technical details.) This is especially important for the delivery of 4K video, as users may not have enough bandwidth to support an increased data rate. H.265 is not yet common as a capture codec, though the (recently discontinued) Samsung NX1 used it, and Premiere Pro now supports it as an editing format.

Hardware H.265 decoding support apparently exists in iPhone 6 and 6s models, though today it’s only used in FaceTime with H.264 being much more common. Given hardware support is present today, I’d expect to see wider adoption over the next few years.

Conclusion

Upload whatever you like, and Vimeo or YouTube will encode everything else.

Upload whatever you like, and Vimeo or YouTube will encode everything else.

If you’re delivering video, you probably don’t need to worry too much about the finer points of codecs—YouTube or Vimeo will take care of the final compression for you. They may even recompress your source file to H.265 if that becomes popular in the future. It is important, though, to know the limitations of your capture codec, to know when you need to use something better, and to know how to deal with weird files your clients may give to you.

As codecs and cameras evolve, they’ll become more efficient, but will demand more resources too. When you pick a new camera, it’s a good idea to stay with standard codecs if you can, and be sure your other gear is up to the challenge if you decide to move to something new. Good luck!

Comments (1)

You must be logged in to comment.

  • cedesse
    Given that the license fees mentioned in the article must be very costly for the hardware and software vendors, I can't help wondering why there is still no industry-wide initiative to start adding native support for the new standardized royalty-free open formats like VP9 video, FFv1 lossless video, FLAC and Opus audio, and of course the MKV and WebM containers? None of these formats are patented. They are absolutely free to implement for any commercial purpose. So from a simple commercial point of view this doesn't make much sense. Why not add VP9 and Opus decoding support to my smart TV firmware, if the cost is limited to adding the standard codecs? And why not add the FFv1 and FLAC codecs to my AV editing software, when my end users are asking for it? Support for these most common modern open standards would clearly be a selling point towards public instutitions and even private sector businesses who are afraid of vendor lock-in. The actual implementation costs can't possibly exceed the annual license fees to MPEG-LA and other patent holder organizations. Are there other reasons that can explain this loyalty towards the heavily patented industry standards?
    • 2 years ago
    • By: cedesse
    Reply
Feedback
Course Advisor
Don't Know Where To Start?
Ask A Course Advisor
Ask Us!
Copy the link below and paste it into an email, forum, or Facebook to share this with your friends.
Make money when you share our links
Become a macProVideo.com Affiliate!
The current affiliate rate is: 50%
Classes Start Next Week!
Live 8-week Online Certification Classes for: