Calculation problems

[namliam]

Has anyone ever experienced this? I think it has to do with floating point stuff...

?int(Abs(-9.95) * 100)
994

?Abs(-9.95) * 100
995

?int(round(Abs(-9.95) * 100,0))
995

Why oh why is that first one not 995??? Brrrrrr... Anyway I added the round to make sure, but AAaaaaaargh....

 

[modest]

Mailman, I hardly ever use the int() just because it doesn't really evaluate to an integer. It returns the number that is not a fraction (meaning it drops the decimals), sort of similar to a string truncation.

While the anomaly you've found has sparked my curiosity, because it should still evaluate to the correct number, I think I've come across the same problem before.

Instead of using Int, try using cInt. Notice the following tests I have done:

    Dim varN As Variant, intVarN As Variant, cintVarN As Variant
    Dim intN As Integer, intIntN As Integer, cintIntN As Integer
    Dim dblN As Double, intDblN As Double, cintDblN As Double
    Dim num
     
    num = -9.95                 'Value After    Variable Type
    varN = Abs(num) * 100       'returns 995    Variant/Double
    intN = Abs(num) * 100       'returns 995    Integer
    dblN = Abs(num) * 100       'returns 995    Double
    
    intVarN = Int(varN)         'returns 994    Variant/Double
    intIntN = Int(intN)         'returns 995    Integer
    intDblN = Int(dblN)         'returns 994    Double
    
    cintVarN = CInt(varN)       'returns 995    Variant/Integer
    cintIntN = CInt(intN)       'returns 995    Integer
    cintDblN = CInt(dblN)       'returns 995    Double

It appears as though Int() has problems handling a double. Although Int(CDbl(995)) and Int(995#) both work as intended.

 

[modest]

In case you were wondering:
intIntN = Int(varN)
intIntN = Int(dblN)
both returned 994 so it has nothing to do with what datatype is storing them.

 

[namliam]

This is very disturbing, I work for financial departments where cents matter! Sometimes even hundreds of cents (yes they are cheap, but the bookkeeping must be correct right?)

Which is why I am doing int(9.95 * 100), to make sure I have exactly 995 and not 994.99999999999 which can happen in a double. And which is probably what is happening here.

Still this does disturb me

 

[modest]

The question is why is it becoming 994.999, it was clearly declared 9.95 (the problem occurs with or without the * 100 --> ive tested with starting the number at 995). The absolute value function doesn't effect this (i've removed it)... so,..

We have a double: 995
We use int() to drop the fractional portion (should be .0000000000)
It should leave us with 995, but instead 994 :::

Pat, please tell your buddies at Microsoft about their malfunction if they don't already know =)