Comment 19 for bug 882036

Revision history for this message
Pieter J. Kersten (EduSense BV) (pieterj) wrote : Re: rounding error

Some comparisons:

def r1(f, p=0):
    return round(f + cmp(f, 0) * pow(2, -52), p)

def r2(f, p=0):
    return round(f + cmp(f, 0.0) * (.1 / pow(10, p+2)), p)

for s in [-1, 1]:
    for f in [.674, .675, .676]:
        for p in [0.0, 1.0, 2.0, 3.0]:
            g = s*(f+p)
            print g, round(g,2), r1(g,2), r2(g,2)

-0.674 -0.67 -0.67 -0.67
-1.674 -1.67 -1.67 -1.67
-2.674 -2.67 -2.67 -2.67
-3.674 -3.67 -3.67 -3.67
-0.675 -0.68 -0.68 -0.68
-1.675 -1.68 -1.68 -1.68
-2.675 -2.67 -2.67 -2.68
-3.675 -3.67 -3.67 -3.68
-0.676 -0.68 -0.68 -0.68
-1.676 -1.68 -1.68 -1.68
-2.676 -2.68 -2.68 -2.68
-3.676 -3.68 -3.68 -3.68
0.674 0.67 0.67 0.67
1.674 1.67 1.67 1.67
2.674 2.67 2.67 2.67
3.674 3.67 3.67 3.67
0.675 0.68 0.68 0.68
1.675 1.68 1.68 1.68
2.675 2.67 2.67 2.68
3.675 3.67 3.67 3.68
0.676 0.68 0.68 0.68
1.676 1.68 1.68 1.68
2.676 2.68 2.68 2.68
3.676 3.68 3.68 3.68