I think the problem is that you try to translate the image after you scale it, something flash can't do. In flash, matrices are always translated before they are scaled, no matter what order you call the functions or set the variables in.

To fix it, you need to multiply the tx and ty with the scale. Try the following code instead (I'm not certain it will work, but it might):

matrix.tx = (colX*matrix.a*scaleX)+(colZ*matrix.c);
matrix.ty = (colX*matrix.b*scaleX)+(colZ*matrix.d);