| Paul van Wamelen on Wed, 28 Aug 2002 01:12:35 -0500 (CDT) |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
| Question about precision |
Dear all,
In the file gen3.c there is a function called "precision" where the
precision of a real number x is computed with the following code:
k=2-(expo(x)>>TWOPOTBITS_IN_LONG);
l=lg(x); if (l>k) k=l;
return k;
so essentially the number of code words used to represent x, except if the
exponent is small and then essentially 2 plus the power of 2^32 that gives
10^(-exponent). But shouldn't this be 2 plus the power of 2^32 that gives
10^(-exponent+relative precision as number of decimal digits), or
k=lg(x)-(expo(x)>>TWOPOTBITS_IN_LONG);
?
If I am correct, why aren't all kinds of things that use the precision
function broken?
Thanks!
Paul van Wamelen