I programmed the masking many years ago. I strongly suspect that you can't have a mask within the range of another mask.

You don't have to use a single masking shape throughout the entire movie. If you insert a tween of zero somewhere (so there isn't a morphing problem), you can change masks at that point.