Decimal Point equations

[flosculus]

Im surprized this is the first time i've noticed this, but for some reason:

1.5 - 1.1 = 0.3999999999999999

I know how to fix this; multiple by 100 or so, round it, then divide by 100 or so.

However in my case, the number of decimal points the resulting number has it not up to me, its completely dynamic. so the 100 or so could be 1 million or so.

I have found a solution to this problem, but it took like 7 lines of code, and i'd rather do it in just one.

does anyone know something i dont?

thank you

Flos

[5TonsOfFlax]

This is standard floating point error. It occurs in all systems which use a floating point representation. AS3 Number is a 64 bit IEEE-754 standard format number.

If you just want your output to be represented to a certain number of decimal places, you can use toFixed or toPrecision to format your number as a String to either a certain number of decimal places, or a certain precision.

If you are doing say, money calculations, then do your calculations in cents rather than dollars, and use int (assuming you don't need to represent amounts bigger than about 400 million dollars.) Or you can still use Number to represent your cents and be sure you do integer math stuff and you can represent 2^53 or so cents, which is about 10 times bigger than the world GDP if I looked things up correctly.

[flosculus]

This is what i did to fix the problem. I havent noticed any bugs with it, it seems to do what i want it to:

private function integralSubtraction(high:Number, low:Number):Number {
    var decimal:uint = Math.pow(10, decimalPrecision(high, low));
    return Math.round(decimal*(high - low)) / decimal;
}

private function integralAddition(x:Number, y:Number):Number {
    var decimal:uint = Math.pow(10, decimalPrecision(x, y));
    return Math.round(decimal*(x + y)) / decimal;;
}

private function decimalPrecision(x:Number, y:Number):Number {
    var xPoints:uint = x.toString().replace(/^.+\./, '').length;
    var yPoints:uint = y.toString().replace(/^.+\./, '').length;
    
    if(xPoints > yPoints)  return xPoints;
    else return yPoints;
}

So now:

trace(integralSubtraction(1.5, 1.1));  // returns 0.4
trace(integralAddition(1.5, 1.1));     // returns 2.6

Can you spot any mistakes?

Thank you

Flos