Peter Bruin on Wed, 12 Feb 2014 13:24:01 +0100 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
Re: precision of result when adding t_REAL with lots of cancellation |
Hi Jeroen, >> Would it be reasonable to round the precision _down_ in such cases? > > If you would that, then you would very quickly end up with a precision > of 0 bits after a few such additions. That is why I only suggest to do this if the precision is _extremely_ small. If the precision of the result of an addition is 31 bits (in a 32-bit word) it doesn't hurt too much if you return this result with 32 bits of precision, but it's a different thing when you even do this when the result has only 1 bit of precision. I'm not proposing this as an optimal solution and would be happy with any alternative suggestions. Peter