Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2011-11

rounding precision error with integer and floating point division

12 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Seth Grover

In a project I'm working on making platform portable, I was getting a different answer for a calculation on i386 vs. x86_64. I boiled it down to this:

===============================
program project1;
{$mode objfpc}{$H+}

uses
Classes,

  SysUtils;
var
  iHour : longword;
  r64 : double;
begin
  iHour := 17608383;
  r64 := (iHour / 24.0);
  Writeln (FloatToStrf(r64,ffFixed,15,4));
  r64 := (double(iHour) / 24.0);
  Writeln (FloatToStrf(r64,ffFixed,15,4));
end.
===============================

Output when compiled for i386-linux:
733682.6250
733682.6250

Output when compiled for x86_64-linux:
733682.6875
733682.6250

Notice that in x86_64, unless you explicitly cast the longword as a double before the division, you get some sort of rounding error or loss of precision. For what it's worth, in C (compiled with gcc) both 64-bit and 32-bit give the same (correct) answer, 733682.6250.

In 32-bit, the two assignments generate the following machine code (both give the same answer):

# [17] r64 := (iHour / 24.0);
movl %eax,U_P$PROJECT1_IHOUR
movl %eax,-48(%ebp)
movl $0,-44(%ebp)
fildq -48(%ebp)
fdivs _$PROJECT1$_Ld2
fstpl U_P$PROJECT1_R64

# [20] r64 := (double(iHour) / 24.0);
movl U_P$PROJECT1_IHOUR,%eax
movl %eax,-48(%ebp)
movl $0,-44(%ebp)
fildq -48(%ebp)
fdivl _$PROJECT1$_Ld3
fstpl U_P$PROJECT1_R64

In 64-bit, here is the machine code generated (the first one gives the wrong answer, the second one is right):

# [17] r64 := (iHour / 24.0);
movl U_P$PROJECT1_IHOUR,%eax
cvtsi2ssq %rax,%xmm0
divss _$PROJECT1$_Ld2,%xmm0
cvtss2sd %xmm0,%xmm0
movsd %xmm0,U_P$PROJECT1_R64

# [20] r64 := (double(iHour) / 24.0);
movl U_P$PROJECT1_IHOUR,%eax
cvtsi2sdq %rax,%xmm0
divsd _$PROJECT1$_Ld3,%xmm0
movsd %xmm0,U_P$PROJECT1_R64

Although I can avoid the problem by casting the longword to a double point before the division, this seems like it should be done automatically to make results across architectures consistent. What are your thoughts? Should I log an issue in mantis?

Thanks,

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover

Reply by email

Graeme Geldenhuys
to Seth Grover
Seth Grover wrote
For what it's worth, in C (compiled with gcc) both 64-bit and 32-bit give the same (correct) answer, 733682.6250.

For what it's worth, a Java test (compiled under 64-bit Linux) also gives the answer of 733682.6250.

Regards,   - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://fpgui.sourceforge.net

Reply by email

Florian Klaempfl
to Seth Grover: In a project I'm working on making platform portable, I was getting a different answer…
Seth Grover wrote
Although I can avoid the problem by casting the longword to a double

How should the compiler know that the division should be done using double? Using always double slows down calculations. Because the range of longword fits into a single as well as 24, fpc uses a single division.

point before the division, this seems like it should be done automatically to make results across architectures consistent.

This is not possible. Especially i386/x87 behaves really strange because it does calculations always with extended.

Reply by email

Graeme Geldenhuys
to Jonas Maebe
Jonas Maebe wrote

The description of the "new behaviour" should change then. The part that says "...which doesn't cause data loss" is now false, given Seth's example. He lost precision compared to previous behaviour - so there is a small data loss, and for some type of applications that could be significant.

Regards,
- Graeme -

fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://fpgui.sourceforge.net/

Reply by email

Jonas Maebe
to Graeme Geldenhuys
Graeme Geldenhuys wrote 9 lines
On 2011-11-18 13:04, Jonas Maebe wrote: > > http://wiki.freepascal.org/ > User_Changes_2.2.0#Floating_point_constants The description of the "new behaviour" should change then. The part that says "...which doesn't cause data loss" is now false, given Seth's example.

It says that the floating point *constant* (in Seth's example: 24.0) is converted to the lowest precision that does not cause data loss. It does not say that the resulting calculation will have the same precision as before. In fact, the "effect" part says exactly the opposite.

Jonas

Reply by email

Michael Schnell
to Jonas Maebe: http://wiki.freepascal.org/User_Changes_2.2.0#Floating_point_constants Jonas
Jonas Maebe wrote

Taking a look at the "go" language:

Here all constants are considered to be of arbitrary precision. This also holds for integers !

A constant can be "10<<1000" there.

Compiler magic provides that when assigning such a constant to a really existing variable the appropriate thing happens (which of course can be an exception).

Seems like nice to me.

-Michael

Reply by email

Seth Grover

Fair enough. Jonas, thanks for the link and explanation. I'll just be more aware of what I'm doing when it involves calculations like this and do the cast when needed.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover

Reply by email

Jonas Maebe
to Seth Grover
Seth Grover wrote
Fair enough. Jonas, thanks for the link and explanation. I'll just be more aware of what I'm doing when it involves calculations like this and do the cast when needed.

One command line parameter was added later on to facilite dealing with such situations: -CF32 and -CF64. Those specify the minimal precision that will be used for floating point constants (-CF80 is not supported due to implementation reasons).

Jonas

Reply by email

Seth Grover
Florian wrote
How should the compiler know that the division should be done using double? Using always double slows down calculations. Because the range of longword fits into a single as well as 24, fpc uses a single division.

...

This is not possible. Especially i386/x87 behaves really strange because it does calculations always with extended.

Thank you for the explanation. That, together with the link to the "user changes in 2.2.0" page on the wiki, explains the behavior. I appreciate your taking the time to respond. Have a great weekend!

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover

Reply by email

Seth Grover
to Seth Grover
Jonas wrote
One command line parameter was added later on to facilitate dealing with such situations: -CF32 and -CF64. Those specify the minimal precision that will be used for floating point constants

Terrific, thanks. I'll check out using that as a possibility as well.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover

Reply by email