Why can't Flash 8 do math correctly?

[ushiroda80]

//thanks deadbeat.

Hi, I am running the following script. The problems is that the answer I get it something like 0.0499999999999998 instead of 0.05 and I was wondering why.

function checkChange(cost, moneyTendered, moneyReturned){
var correctChange:Number = 0;
var difference:Number = 0;
trace ("I am checking my change...");
correctChange = moneyTendered - cost;
difference = correctChange - moneyReturned;
if (correctChange == moneyReturned) {
trace("\tCorrect change given");
} else {
trace("\tIncorrect amount! by $" + difference);
}
}

checkChange(12.5,15,2.55);

[jasonsplace]

It looks like you may have found a bug unless someone else can come up with a reason that 2.5-2.55 = 4.999999998

[The_lone1]

I would suggest using Math.round().

[jasonsplace]

I would suggest using Math.round().

That was my first thought as well but it doesn't leave it as a decimal and in this case just rounds to 0....which i'm guessing won't work in this situation.

[The_lone1]

I reckon this is probably the same limitation every single computer will encounter. It's based on the fact that all data is stored in binary, and the smaller the number is, the less likely you'll be able to store it exactly.

[rynoe]

It gets even more strange:
[CODE]
var a:Number = 0.5 - 0.55;
var b:Number = 1.5 - 1.55;
var c:Number = 2.5 - 2.55;
var d:Number = 3.5 - 3.55;
var e:Number = 4.5 - 4.55;
trace( "zero: " + a);
trace( "one: " + b);
trace( "two: " + c);
trace( "three: " + d);
trace( "four: " + e);
[CODE]


Output:
zero: -0.05
one: -0.05
two: -0.0499999999999998
three: -0.0499999999999998

[The_lone1]

hmm. looks more like the smaller proportion the remainder is of the original number, the more likely it is to get a math error...

[IMS]

Math.round won't work, .5 would round up to 1.
this is strange, I have mx2004pro, and it did the same thing.
if you put trace(2.5-2 );it will give you .5

I got this to work once, all the other tries the answer just kept getting worse

IMS

[deadbeat]

Sounds like you've run into the rounding error:

http://www.adobe.com/cfusion/knowled...fm?id=tn_13989

K.

[IMS]

If you give back too much change it seems to work fine, but if it is correct or less, I've been getting errors.... I'm bored outta my cap right now, so this is helping the day go a little quicker

IMS

[dudeqwerty]

meh, nevermind

[The_lone1]

Mate, what are you talking about?

it's impossible to represent some numbers in binary. most of these are small negative decimals., as evidenced in this case.

[dudeqwerty]

yeah i got you, i though you were saying something else, hence my edit

[The_lone1]

oh right.

[drfrankius]

zero: -0.050000000000000044
one: -0.050000000000000044
two: -0.04999999999999982
three: -0.04999999999999982
four: -0.04999999999999982

results in flash 9 ... even weirder

var a:Number = 0.5 - 0.55;
var b:Number = 1.5 - 1.55;
var c:Number = 2.5 - 2.55;
var d:Number = 3.5 - 3.55;
var e:Number = 4.5 - 4.55;
trace( "zero: " + a);
trace( "one: " + b);
trace( "two: " + c);
trace( "three: " + d);
trace( "four: " + e);

[kaeru]

after reading that roundoff error article on the adobe site, it does seem like thats the case...

if Math.round only works on the units level, then just multiply your result by 100 first, then use Math.round(), then divide it by 100 again.

does that make sense D:?

kae

[cancerinform]

You probably solved this thread. I was wondering how one can do a simple computer calculation without loosing money