Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2006-01

Incorrect/Strange precision in FloatToStr and FloatToStrF

4 messages · 3 people ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Graeme Geldenhuys

Hi,

[posted this again, as I didn't see it appear on the list]

Is this a bug? The actual value of 234.502 is not greater than either data types (extended or double), but the conversion to string result using the Double is not what I expected.

Strangely enough 234.501 seem to work. 7.502 and 8.502 is another strange one!

procedure TForm1.Button4Click(Sender: TObject);
var
 Result: string;
 e: extended;
 r: double;
begin
 e := 234.502;
 Result := FloatToStrF(e, ffGeneral, 15, 0);
 Memo1.Lines.Add(Result);      { prints 234.502  }
 r := 234.502;
 Result := FloatToStrF(r, ffGeneral, 15, 0);
 Memo1.Lines.Add(Result);  { prints 234.50200000000001 }
 r := 234.501;
 Result := FloatToStrF(r, ffGeneral, 15, 0);
 Memo1.Lines.Add(Result);  { prints 234.501  Why does this work? }
 r := 7.502;
 Result := FloatToStrF(r, ffGeneral, 15, 0);
 Memo1.Lines.Add(Result);  { prints 7.502 }
 r := 8.502;
 Result := FloatToStrF(r, ffGeneral, 15, 0);
 Memo1.Lines.Add(Result);  { prints 8.502000000000001 }
end;

Regards,
- Graeme -

Reply by email

Marco van de Voort
to Graeme Geldenhuys
Show quoted text 6 lines
Is this a bug? The actual value of 234.502 is not greater than either data types (extended or double), but the conversion to string result using the Double is not what I expected. Strangely enough 234.501 seem to work. 7.502 and 8.502 is another strange one!

This is all normal. It has to do with the fact that floating point numbers variables are internally base 2 (binary) floating point notation.

So if you assign a certain decimal value, say 0.1234, is converted to the closest binary notation and assigned to the float. Then when you print it is converted back to decimal. Both operations can cause rounding

Reply by email

Graeme Geldenhuys
to Marco van de Voort

Any idea how Delphi handles this differently then without rounding errors? This behaviour was picked up in my unit tests I am porting from Delphi to FPC.

Using Double or Real under Delphi worked. Extended was the only type I could use under FPC that gave me the correct results.

Regards,
- Graeme -

Marco van de Voort wrote 20 lines
> > Is this a bug? The actual value of 234.502 is not greater than either > data types (extended or double), but the conversion to string result > using the Double is not what I expected. > > Strangely enough 234.501 seem to work. 7.502 and 8.502 is another strange one! This is all normal. It has to do with the fact that floating point numbers variables are internally base 2 (binary) floating point notation. So if you assign a certain decimal value, say 0.1234, is converted to the closest binary notation and assigned to the float. Then when you print it is converted back to decimal. Both operations can cause rounding _______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

Reply by email

Jonas Maebe
to Graeme Geldenhuys
Graeme Geldenhuys wrote
Any idea how Delphi handles this differently then without rounding errors? This behaviour was picked up in my unit tests I am porting from Delphi to FPC.

I think the reason is that FloatToStrF() is not overloaded for double and single, so even if you pass a double, it will be written out with extended precision (so you go past the maximum precision of the value and get rounding errors). Please submit a bug.

Jonas

Reply by email