Today we take compressed video on our computing devices for granted, but it wasn’t always the case. Once upon a time, even a video the size of a postage stamp on a computer was an amazing thing to behold—even compressing a still image was a big deal once. Here, we’ll take a look back, and then a look forward to what’s coming next.
Before we dive into video, we’ll look at still images, because many compression schemes used by moving images are similar.
Back when we didn’t have a whole lot of data to compress, or a whole lot of colors to express ourselves with, compression was usually lossless. Instead of writing a chunk of data for each color: “white, white, white, red, white, white, white” and so on, you can use a scheme like Run-Length Encoding to write something along the lines of “three whites, red, three whites”. This saves a little space, and lossless encoding is still used today in GIF, PNG and some TIFF files.
Because lossless compression doesn’t usually save a whole lot of space, and the finer image details aren’t always important, lossy compression throws away data to save more space. JPEG breaks the entire image up into 8x8 pixel blocks, figures out the average color of each block, then describes the pixels in each block in relation to that average color. That’s why you see blockiness in high contrast areas of heavily compressed images.
When you throw video into the mix, file sizes become huge. Some professional workflows do indeed use uncompressed video or image sequences, but almost all video is captured and delivered in a compressed way. A key difference is whether the video is compressed as intra-frame (also known as I-Frame, essentially just a series of still images) or inter-frame (where each frame is described in relation to past frames or even future frames). If a scene isn’t changing much, then this inter-frame strategy can save a lot of space, and variations of this are dominant today. Advanced codecs use blocks of variable sizes and predictive algorithms to guess where content will move from frame to frame.
A computer will have to work harder to decode inter-frame video (like H.264) than it would to decode intra-frame video (like ProRes 422 or DNxHD), but modern Macs running FCP X or Premiere Pro can handle inter-frame H.264 just fine. Older Macs may struggle, which is why FCP X offers an option to “optimize” video to the easier intra-frame ProRes 422. It’s less efficient, and takes more space on disk, but is easier to interpret. Remember, though, that an intra-frame codec needs a higher bitrate than an inter-frame one to maintain the same quality.
Now we’ve got the basics down, let’s go back in time to QuickTime 1.0, released back in 1991. I still remember seeing this for the first time, in a demo from an Apple rep, with tiny videos of Apollo rockets taking off. Here’s a very geeky introduction, and a very early video. It allowed anyone with a color Mac to watch moving video, and to copy and paste videos for simple editing. While digital video capture was still a long way off, this was very obviously the way of the future, and a revelation at the time.
QuickTime brought a container (the .mov file, or at the time, a “MooV” file type) which could contain audio, video, or other tracks. A wide range of codecs could be used to compress the video and audio, including Animation, Video and Graphics in version 1.0, and then Cinepak in version 1.5. This new codec became dominant for many years, and you can read more about it here.
Later versions of QuickTime brought newer codecs, including Sorenson Video, and MPEG-1, but it really got interesting in 2001 with the DV codec, DV camcorders, and the iMac DV. Suddenly it was possible to edit home videos on a home computer without external capture devices, and this was a big deal.
Of course, QuickTime wasn’t the only game in town. Microsoft introduced the AVI container format, supporting the same Cinepak, Motion JPEG and DV codecs that QuickTime offered. RealVideo was important for streaming video for a time, as was Flash video, but for a long time it was difficult to provide a single movie file that everyone could play. It was looking a bit crazy there for a while, but standards have eventually, largely, prevailed.
QuickTime’s container format was used as the basis for the MPEG-4 container format, released alongside the MPEG-4 codec by MPEG, the Motion Picture Experts Group. Their previous standard-based efforts included MPEG-1 (popular for a while) and MPEG-2, used on DVDs. Oh, and MPEG Audio Layer III—MP3—a very popular audio format you may have heard of. The MPEG-4 video codec eventually evolved into today’s dominant video standard, H.264, also known as MPEG-4 AVC.
Why are standards important? Since most cameras capture data in a flavor of H.264, and most delivery takes place in an H.264 codec, our devices can include hardware-based support for encoding and decoding it. For phones and laptops, that means vastly increased battery life, speeds, or both. That doesn’t mean that there aren’t good reasons for choosing something else, though.
If you’re working with RED footage, each clip is not just a single file, but several.
When image quality is critical, there are good reasons to capture in a raw format like REDCODE RAW or ARRIRAW, or in a more generic intermediate format like ProRes. These formats can avoid compression artifacts and capture more color information for better results in post production. There are also many more container types in which to package your video: MXF, Matroska, and more, and some are easier to work with than others.
Choosing a non-standard codec or container means you’ll need special software support to be able to edit or view your footage. For advanced workflows, it can be worth the effort, but it’s going to be easier to work with an H.264 MP4 file or a ProRes QuickTime. Sure, a new codec can be more efficient, but it’s disappointingly common for new cameras to use slightly different codecs and/or container formats for apparently no good reason.
From the MPEG-LA website, if you have more than 100,000 paid subscribers or sell videos longer than 12 minutes, you should pay [http://www.mpegla.com/main/programs/avc/Documents/avcweb.pdf]
In the video game world, an otherwise-obscure codec called Bink Video is popular for pre-rendered cutscenes. It’s popular because there’s no per-unit royalty on its use, so it’s much cheaper than MPEG-4 for game developers. Wait, what? Royalties?
Indeed, patent and royalty issues have been problematic for MPEG in recent years. If you sell an MPEG-4 file for profit, you’re meant to pay a per-unit fee, and there are costs for providers of encoding and decoding software too. What if you don’t want to pay, or can’t?
Google tried to offer a free solution with the WebM codec, which is claimed to be patent-free, but it’s not yet been widely adopted outside the open source community. (This is also the case with Ogg Vorbis, an audio codec that competes with MP3.) Phones need H.264, computers make H.264 quickly and easily, Microsoft and Apple pay the playback fees, and for a lot of users, that’s more or less that.
Also tricky: since most of the methods behind video codecs are similar, it’s likely that a new codec will infringe on some part of an MPEG-owned patent. This creates a legally challenging environment for new codecs.
The latest flavor of MPEG is H.265 (HEVC), which uses larger variable block sizes and better motion prediction to achieve higher quality at the same data rate as H.264, or the same quality at approximately half the data rate. (Here are some technical details.) This is especially important for the delivery of 4K video, as users may not have enough bandwidth to support an increased data rate. H.265 is not yet common as a capture codec, though the (recently discontinued) Samsung NX1 used it, and Premiere Pro now supports it as an editing format.
Hardware H.265 decoding support apparently exists in iPhone 6 and 6s models, though today it’s only used in FaceTime with H.264 being much more common. Given hardware support is present today, I’d expect to see wider adoption over the next few years.
If you’re delivering video, you probably don’t need to worry too much about the finer points of codecs—YouTube or Vimeo will take care of the final compression for you. They may even recompress your source file to H.265 if that becomes popular in the future. It is important, though, to know the limitations of your capture codec, to know when you need to use something better, and to know how to deal with weird files your clients may give to you.
As codecs and cameras evolve, they’ll become more efficient, but will demand more resources too. When you pick a new camera, it’s a good idea to stay with standard codecs if you can, and be sure your other gear is up to the challenge if you decide to move to something new. Good luck!