Bremen
Novice
Joined: 26 Dec 2002
Posts: 33
Location: USA
|
 Posted: Sun Jun 26, 2005 9:53 pm
Floating point Range vs Precision Bug
|
I noticed this in 7.05, I'm not sure if it existed in earlier versions. I've had it tested on 7.13 and it still exists. Floating point precision is improperly handled as the range gets larger.
Example:
#echo %format("&9.0f",200009000)
200008992
#echo %format("&12.0f",200009000000)
200009007104
And so on. The higher the number (range), the lower the offered precision. This works for &f, &n, and probably the rest. It's not uncommon to deal with transactions throughout a day totalling 200M, so I noticed this from a script keeping track of gold use/deposits/withdrawals throughout a day. |
|
