Creating Smooth Mask Using Blend
This tutorial is For Flash 8 and above only.
Example

Background

One of the limitation of the traditional mask layer in Flash prior to version 8 was the inability to define masks with semi-transparent areas. (If you're unfamiliar with masking, read more about it here.)

The usual way to create the smooth edges is applying Soften Fill Edges to a shape like below.

But when you create a shape to be used as a mask, the masked pixel are either on or off without any in between - so you cannot have say, a mask with 50% transparency or feathered edges.

For example, this soft edged circle below when used as a mask shape, still result in still sharp edged mask like seen on the right side. This is because the masking layer treat all shapes inside it as opaque and does not care about the opacity.

                  

The irony is that Flash 8 still treats masking layers the same way (but in fact, we can use this to our benefit as you see later). Flash 8 however has some useful new features to make smooth/soft edged masking that retains the alpha values by utilizing some of the newly introduced Blend modes. Using this method, you can re-do the mask above with the kind of result seen in the Example section above.

The catch in that the method is not intuitive and takes more steps than regular masks. For example, it requires you to create a separate Movie Clip for the mask; and it does not make use of the Mask Layer in a straightforward manner.

Blend Modes

Since the method involves Blend modes, let's talk about it briefly.

Blend mode affects how a MovieClip is displayed with respect to the content underneath it. The term content here could mean another MovieClip on the layer(s) below it or another MovieClip on the same layer.) For example, If you select Invert, then the area of the image below the movie clip will be inverted (as in negative-film effect).

To set a blend mode, put a MovieClip onto a stage, and bring up the Property dialog. You can also set it programatically - read more here.

The blends are straightforward to use with the exception of Erase and Alpha which we will discuss in this tutorial. You can read what the others do in the Flash Help file (search for BlendMode).

Step by Step Tutorial
 
  1. Because Blend modes only work on MovieClips, you need to create a MovieClip for any shape you want to use as a mask. So create a MovieClip New Symbol, name it Shape and put your mask shape there. For example, I created a circle.
      
  2. You can add soft edges to the shape using the usual way using Soften Fill Edges. Or you can skip this part and use the Blur Filter to make the soft edges. Using Filter makes it easier to change the amount of smoothing (but it is potentially slower when running the movie). Pick one.
      
  3. Ceate another MovieClip Symbol, name it BlendedShape. Name the existing layer to BlendShapeLayer. Put an instance of the Shape that you created earlier onto the stage.
      
  4. If you skipped Step 2, you can add can add soft edges by selecting the movie clip and open up the Filters window. Add (using the plus icon) a Blur filter. Play around with the Blur X and Blur Y value to get what you want. Leave the Quality to Low for now.
      

      
  5. Create a new layer below the existing layer and insert the clip that you want to mask there (I use the grass image below). Position it to the location you want.
      
  6. Reselect the Shape movie (the circle) and change its Blend to Alpha.
      

    Don't fret when the image dissapear, this is how Alpha blend works.
      
  7. Now we're ready to see what the result looks like. Go back to the main timeline (or create a new symbol). Insert an instance of BlendedShape symbol and set its Blend mode to Layer.
      

      
    What we have done here is to use the Alpha blend to mask out our original image. You can put anything underneath this shape and it will show through. Try tchanging the background color.

    But there's a big problem: the outside of the shape is still showing! This is because Alpha blend does nothing to the hollow area. Since the outside of the circle is hollow, it is not blended.

    There are several options to solve this problem:
    • Create a shape that enclosed the outer area and fill it with any color of alpha value 0. This will be easier if you had used the Step 2 above. In our example the shape will be a rectangle with a circular hole on it. I personally don't like this solution because it is not very flexible (for example, if the stage dimension changes, then the encloding rectange needs to be modified.
    • Or, you can use the same shape as a mask.
    I will do the second options because it is more flexible and it reuses the shape that we already have.
  8. Go back to the BlendedShape symbol. Create a new layer above the BlendShapeLayer and name it MaskLayer (that's right, we're going back to the old Masking layer again!).
     
  9. Copy yhe shape from the BlendedShape layer (this is the circle) and copy it onto the exact same location on the MaskLayer. Use Menu->Edit->Paste In Place.
      
  10. Now make the MaskLayer a real mask layer to mask the two layers below it. The layers configuration should look like this:
      

      
  11. Go back to the stage and see how the shape looks now. This is mine:
      

      
So what we have done here is combining the traditional mask technique with Blend mode to create a smooth mask.
More Fun Stuff

Add another shape behind and see that the soft edges are really transparent.

Play around with other Blend modes. For example, if you change the Blend mode to Erase (instead of Alpha), you can get the reverse effect like below which might be desirable in some situations and with that, you do not need to use a masking-layer at all.

Download Example FLA.