Photoshop: Finessing a Mask with Selection Tricks

There are many ways to create a mask in Photoshop, and even more ways to improve that mask. We’ve covered some of these before, but here we’ll look at how to move seamlessly between selections and masks, using one to improve the other.

Start from a basic selection

The content will define the strategy you need to use to get your basic selection. Sometimes the Quick Selection Tool will be the best option, sometimes Select > Color Range. Even the tools I love to hate, the Magic Wand Tool and the Magnetic Lasso Tool, can have a place. Mostly, it comes down to a few things:

If the object has crisp edges and is a different color to the background, use the Quick Selection Tool, or the Magic Wand Tool if that fails.

If the object has soft edges and is a different color to the background, use the Color Range Tool, or start by Command-clicking one of the Channels (R/G/B) in your image.

If the object is at least partly the same color as the background, you’ll have to use the Pen Tool to define a clipping path, or paint a selection in Quick Mask or a Layer Mask. That’s where it can get messy.

Cutting out this awesome birthday cake is a bit of a challenge — the Quick Selection Tool makes a mess of it, there are too many colors for Color Range, and the edges are soft — a mask would need to be manually painted.

Finesse the selection with Refine Edge

If your mask is nearly there, one of the easiest ways to clean up a selection is to use Refine Edge. You can:

• use Feather to soften the selection.
  
• use Contrast to crispen the selection.
  
• use Smooth to take sharp direction changes (like spikes) out of the selection edges — but this will ruin soft edges like hair.
  
• use Shift Edge to move edges in or out.
  

For a smoother look, you could use settings like these:

Alternatively, leave all those settings at 0, then:

• Use the Refine Radius Tool to paint over transition areas like hair. Here’s what you can do with a little work with the Refine Radius tool:
  

• Export with decontamination, to a new layer.
  
• Finally, you should choose to output to a New Layer with Layer Mask, and Decontaminate Colors to correct any for thin edges of your foreground object that have taken on the color of your background. On a Layer Mask, painting with black will hide your image, and painting with white will show it again — this is the key to making the mask perfect.
  

A layer with a layer mask.

Dodge and Burn

Now that you have a Layer Mask, you can paint the selection to make it good, or you can use the Dodge Tool (set to Highlights) and Burn Tool (set to Shadows) to make your job easier. This technique was discussed in an earlier article here so check it out now.

Dodge the Highlights, Burn the Shadows.

Using fancy selections to finesse your mask

Even the best mask isn’t perfect. Maybe part of it could be sharper, or maybe you need to punch a hole in it — or reveal the transparent part of a glass. To tweak part of your mask, the easiest way is often to select part of your image with Channels. If you Command-click the Red, Green or Blue channel in your Channels panel, you’ll load it as a selection. Select > Color Range... can also work well here if the areas you want to select are more complex.

Clicking just once on the back of the glass in the main image was enough to select it — use Fuzziness to get it right.

Enter Quick Mask

Press Q to enter Quick Mask, converting your selection back into a temporary mask. In a Mask, white and black map to selected and not selected, with a grey range connecting the two smoothly. However, it’s unlikely that the areas you want to change on your mask are all white.

Here’s the selection viewed as a Quick Mask, with RGB hidden (and note, it’s possible to invert this display if desired).

Press the eye next to the RGB channel to hide it, and you’ll see your Quick Mask alone, as a greyscale image. Press Command-L to access old-school destructive Levels, then tweak to make the part of the image you want to adjust white and the rest black. Use Command-I to invert the mask if it’s all wrong.

Here’s the old-school Levels command — only good for masks! — in action.

Exit Quick Mask

Press Q again to exit Quick Mask, and you’ll be able to use your new selection to limit your painting. Click on your Layer Mask, then paint with black or white through your selection. Black hides and white reveals your image, and you can press Command-H to hide the marching ants around the selection if they prove distracting.

With the glass interior selected, paint black on the existing Layer Mask to hide it.

Using the image data like this is ideal for fixing up any problems with your mask. Also, don’t be afraid to invert your selection with Command-Shift-I if it’s the wrong way round.

Masks are wonderful things, yet few masks are perfect out of the box. The more tools you have at your disposal, the better your chances of making a mask as good as you can.