X Course Advisor FREE Course Advisor
Watch Tutorials
After Effects: The Roto brush Tool

Check out our After Effects: Motion Tracking and Stabilization course!

Rotoscoping, the art of tracing moving characters and objects in film and video footage to isolate them from their backgrounds, has a long history, going back to the earliest days of cartoon animation. (For more about rotoscoping, check out my HUB article "A Brief Technical History of Compositing".

Nowadays, rotoscoping is mainly used in situations where characters need to be isolated but weren't originally shot against green or blue, which would allow them to be easily color-keyed. (Rotoscoping, in fact, is how many current films that were originally shot in 2D are converted into a form of 3D for theatrical screening.)  In such cases, the only choice is generally to trace the outlines of the characters by hand, often frame by frame, which can be a terribly tedious and time-consuming job. 

But there are some tools available to us that can make jobs like that go a little smoother. In this article, I'm going to introduce you to After Effects' Roto brush tool, which can largely automate the process of rotoscoping and let us easily create mattes for compositing footage that isn't suitable for color-keying. Let's get started.

Step 1 -  Set Up Your Project 

For best results with the Roto brush tool, your footage should have its subject well-outlined against a plain solid background, with high contrast between the foreground and background. I'm going to be using a short shot of me speaking against a plain white background, which I've imported and dragged onto the Create a New Composition button to add to a new comp:

Even though this shot is only about 3.5 seconds long, it's still over 100 frames, so it's something we definitely wouldn't want to trace by hand. And even though my head is well-defined against the background, the shot isn't well-lit enough to allow for color or luminance keying. Here's what happens when you try:

trying luminance keying

So, it's on to the Roto brush tool.

Step 2 -  Select and Apply the Roto Brush Tool

With your new comp active, select the Roto brush tool:

Rotobrush tool

Double-click with it on the image in the Comp window to open the footage's Layer window (most Roto brush work happens in the Layer Window).

Step 3 -  Draw Your First Stroke

Next, locate your base frame (a frame where the subject is clearly defined against the background) where you'll be drawing your first Roto brush stroke. The first frame of your shot often works well as a base frame, and that's what I'm going to be using here. 

Now, drag the Roto brush tool (which should display as a green circle with a plus in it) down through the center of your image from edge to edge, as shown (you don't use the Roto Brush tool to draw around the outline of the image, as you would with the Pen tool)...

An outline

...and release the mouse button to create the segmentation boundary, the magenta line that will define your subject's mask:

Define the subject's mask

As you can see, the Roto brush has pretty smoothly outlined at least part of the image. Our next step is to add to this basic boundary to outline the entire image.

Step 4 -  Add to the Segmentation Boundary

Still working in our Base Frame, position the Roto brush cursor in the areas of the subject that weren't outlined with the first stroke, and just drag through them to add to the boundary until you get a solid magenta line that completely outlines the subject:

A complete outline of the subject

Step 5 -  Refine the Segmentation Boundary

Now you can see, in a couple of places at the top of my head, that the Roto brush has selected a little too much of the background, which won't give up a clean matte. However, if we hold down the Option key (the Roto brush tool will change to a red circle with a minus sign in it) and drag in those areas (zooming in can help here), we can tighten up the boundary by subtracting those areas from the selection:

Subtract unwanted areas

My wispy hair is a bit of a problem here, but we can improve that a little later. For now, let's leave it as is. Continue on if necessary by adding to and subtracting from the Base Frame's boundary until you have a clean, solid selection. Click the Toggle Alpha:

Toggle Alpha

and Toggle Alpha Overlay:

Toggle Alpha Overlay

These will show us our alpha channel alone and the subject with a red overlay in the background, respectively, so we can check the state of the matte.

Step 6 - Propagate the Boundary

Once the Base Frame matte is to our liking, it's time to propagate the current outline to the rest of the footage. This is where the magic happens: when we propagate the rest of the frames, the Roto brush tool will copy and automatically adjust the boundary to match the movement of the subject. 

Now, the range of frames within which the propagation occurs is called the span, and we adjust the span to set the propagation range. With complex subjects, we often want to set a short span, so we can precisely tweak the segmentation boundary. In this case though, because there's relatively little motion in the clip, we can set the span to cover the entire clip, and tweak later if necessary. 

Drag the span (the row of gray blocks under the Time ruler) out to the end of the Time Ruler, then hit the Space bar to begin the propagation. 

Drag the Span

If the boundary loses track of the subject's outline, just stop the propagation and adjust the boundary as you did on the base frame, backing up in the Time Ruler if necessary, then continue the propagation. Eventually, all your frames will be outlined. 

Step 7 - Freeze the Boundary

The last step in the Layer window is to freeze the segmentation, which will cache the boundary data to disk to both prevent further mask updating and to speed up interaction with the comp. To do that, just click the Freeze button at the bottom of the window:

Click the Freeze button

After a bit, you'll see an indication in the Layer window that the segmentation is frozen and you can close the Layer window.

It's frozen

Step 8 - Composite the Layer and Adjust the Matte

Our final step is to composite the layer we just masked and tweak the edge of its alpha channel to smooth it against the background. In my case, I've added a layer of flame imagery behind my masked head to make the edge clear:

Smoothing it out

As you can see, the masked edge of my head is pretty hard and doesn't blend smoothly with the background. To clean that up, we'll use the Roto brush's Matte tools which, since the Roto brush is an effect, are found in the Effect Controls window for the selected layer:

The Roto Brush will smooth it out

Smooth, Feather, and Choke work interactively to refine the edge of matte:

Smooth, as you'd expect, reduces the roughness of the matte edge; Feather softens the matte edge by making it more transparent, so it blends more into the background; Choke pulls the edge of the matte in towards the center of the subject to tighten it up. 

Adjusting these three together will give you a much smoother final composite:

And there you have it.  

The Roto brush tool is one of After Effects' lesser-known features, but a powerful one that anyone doing compositing special effects will find useful. I haven't gone into all the details of the tool by any means, so as always, I encourage you to experiment. And above all, have fun!

Check out our After Effects: Motion Tracking and Stabilization course!

Richard Lainhart

Richard Lainhart | Articles by this author

Richard Lainhart is an award-winning composer, filmmaker, and author. His compositions have been performed in the US, Europe Asia, and Australia, and recordings of his music have appeared on the Periodic Music, Vacant Lot, XI Records, Airglow Music, Tobira Records, Infrequency, VICMOD, and ExOvo labels. His animations and short films have been shown in festivals in the US, Europe, and Asia, and online at ResFest, The New Venue, The Bitscreen, and Streaming Cinema 2.0. He has authored over a dozen technical manuals for music and video hardware and software, served as Contributing Editor for Interactivity and 3D Design Magazines, and contributed to books on digital media production published by IDG, Peachpit Press, McGraw Hill, and Miller Freeman Books. Previously an Adobe Certified Expert in After Effects and Premiere, a demo artist for Adobe Systems, and co-founder of the official New York City After Effects User Group, he was, from 2000-2009, Technical Director for Total Training Productions, an innovative digital media training company based in New York and California.


You must be logged in to post a comment.
Create an Account  Login Now

What is macProVideo.com?

macProVideo.com is an online education community featuring Tutorial-Videos & Training for popular Audio & Video Applications including Adobe CS, Logic Studio, Final Cut Studio, and more.
© 2018 macProVideo.com
a division of NonLinear Educating Inc.