Numbers... Sorry but bear with me!

[hardy1976]

Right.. on my trusty calculator...

265*0.7 = 185.5 integer = 185
90*0.7 = 63 integer = 63
350*0.7 = 245 integer = 245
170*0.7 = 119 integer = 119

However when I use vba I get...
265*0.7 = 185.5 integer = 185
90*0.7 = 63 integer = 62!??!?!
350*0.7 = 245 integer = 244!??!?!
170*0.7 = 119 integer = 118!??!?!

what am i doing wrong?! how do I get vba to hit my calculated numbers?!

 

[Galaxiom]

The Integer function you are using is triming the part after the decimal place rather than rounding. Use the Round() function.

Combined with the slightly imprecise calculation of 90*0.7 = 62.9999999999 you get the unexpected result.

If you are working with decimals it is best to use Double precision variables rather than single. It is absolutely essential if you want accuracy beyond three decimal places.

Also note that the Round function uses "banker's rounding" which rounds 5 in the Least Significant Place to the even numbered result in the next column.

1.25 and 1.15 both round to 1.2
This is designed to even out the rounding errors.

 

[hardy1976]

Galaxiomathome thanks, so if I use the int of this, would i all ways hit the right number, or do I have to use bankers rounding to?! All what the hell does prec do?!

Function CRound(MyVal As Double, Prec As Integer) As Double

CRound = Int(MyVal * 10 ^ Prec + 0.5) / 10 ^ Prec

End Function

 

[BamaColtsFan]

The Integer function you are using is triming the part after the decimal place rather than rounding. Use the Round() function.

Combined with the slightly imprecise calculation of 90*0.7 = 62.9999999999 you get the unexpected result.

If you are working with decimals it is best to use Double precision variables rather than single. It is absolutely essential if you want accuracy beyond three decimal places.

Also note that the Round function uses "banker's rounding" which rounds 5 in the Least Significant Place to the even numbered result in the next column.

1.25 and 1.15 both round to 1.2
This is designed to even out the rounding errors.

This solved my problem with simply eliminating decimals when I call the value into a text file! Thanks Galaxiom!!

 

[raskew]

Hi -

I gotta ask:

Combined with the slightly imprecise calculation of 90*0.7 = 62.9999999999 you get the unexpected result.

How does a thing like that work? 90*0.7 = 63 there's no rounding involved.

Bob

 

[gemma-the-husky]

the trouble is, the int function truncates a number, not rounds it

so int(62.99999999) is 62


----------
the problem arises because not every real number can be representedso in your example perhaps 0.7 cannot truly be represented, and 90 * 0.7 therefore doesnt compute to 63 exactly

having said that, i tend to agree with raskew, and i dont really understand why it doesnt work, since ? 90 * .7 produced the correct answer

 

[raskew]

Hi -

Dave, help me out here:

so int(62.99999999) is 62

How do you get 90 * 07 reutrns anythinyg other than 63?

Bob

 

[hardy1976]

I'm glad i'm not the only one who just doesnt get it!

also what exactly is this statement doing?!

CRound = Int(MyVal * 10 ^ Prec + 0.5) / 10 ^ Prec

 

[Rick Stanich]

Hi -

Dave, help me out here:

so int(62.99999999) is 62

How do you get 90 * 07 reutrns anythinyg other than 63?

Bob

By using "int", Integer.

in·te·ger
n. Mathematics

1. A member of the set of positive whole numbers {1, 2, 3, . . . }, negative whole numbers {-1, -2, -3, . . . }, and zero {0}.

The "int" is reporting the integer, 62. No rounding or other calculations.
A "real" or "double" would round the value to some defined decimal place, thus, reporting the entire value.

 

[Rick Stanich]

Combined with the slightly imprecise calculation of 90*0.7 = 62.9999999999 you get the unexpected result.

I didn't catch this in my intial reply.

There is no "sligthly imprecise" about "90*0.7", it is the same as "90*7.0", both should always result in a WHOLE number.

Using excel I set up a formula "=SUM(90*0.7)" and set the cell to 100 decimal places.
This number is the result of the same equations except to 100 decimal places, it is a whole number, how does Access get anything else?