Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2008-05

I get duplicate GUIDs under Linux

45 messages · 11 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Graeme Geldenhuys

Hi,

I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I constantly got PK violations. I then wrote the code show below to create 50,000 GUID's and add them to a StringList (no duplicates allowed). Every time I run this I get a duplicate GUID at iteration 1021. My original test is supposed to create 5mil records, but I can't get past 1000! :-(

Actual program output:

./index_performance.elf
Duplicate value detected: {FD3DF19D-A878-F8BA-F9D1-108658451E05} List.Count = 1021

GUID's are supposed to be very unique! Wikipedia says something like: Create a billion every second for 10 years or something before you deplete the range.
So what is wrong with my code or maybe the Free Pascal implementation?

--------------------[ code ]------------------------------
procedure TMainForm.btnTestClicked(Sender: TObject);
var
  sl: TStringList;
  s: string;
  i: integer;
  d: integer;
  guid: TGUID;
begin
  sl := TStringList.Create;
  try
    try
      sl.Sorted := True;
      sl.Duplicates := dupError;
      for i := 1 to 50000 do
      begin
        CreateGUID(guid);
        s := GUIDToString(guid);
        sl.Add(s);
      end;
      writeln('Created 50,000 GUIDs');
    except
      on e: EStringListError do
      begin
        writeln('Duplicate value detected: ' + s);
        writeln('  List.Count = ', sl.Count);
      end;
      on e: Exception do
        writeln('Unknown error: ', e.Message);
    end;
  finally
    sl.Free;
  end;
end;
--------------------[ end ]--------------------------------

Changing the line that adds the GUID to the StringList by prefixing the counter, all is fine. So I can't see a problem in my code.

   sl.Add(IntToStr(i) + s);

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Graeme Geldenhuys

Hi,

I just tested that same code under Windows 2000 and I don't have any errors or duplicates. As I would have expected. So there seems to be a bug in the CreateGUID implementation of Linux.

BTW:
I'm using Ubuntu 7.10. My 'uname -a' output:
Linux graemeg 2.6.22-14-generic #1 SMP Tue Feb 12 07:42:25 UTC 2008 i686 GNU/Linux

Regards,
- Graeme -

Graeme Geldenhuys wrote 71 lines
Hi, I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I constantly got PK violations. I then wrote the code show below to create 50,000 GUID's and add them to a StringList (no duplicates allowed). Every time I run this I get a duplicate GUID at iteration 1021. My original test is supposed to create 5mil records, but I can't get past 1000! :-( Actual program output: ./index_performance.elf Duplicate value detected: {FD3DF19D-A878-F8BA-F9D1-108658451E05} List.Count = 1021 GUID's are supposed to be very unique! Wikipedia says something like: Create a billion every second for 10 years or something before you deplete the range. So what is wrong with my code or maybe the Free Pascal implementation? --------------------[ code ]------------------------------ procedure TMainForm.btnTestClicked(Sender: TObject); var sl: TStringList; s: string; i: integer; d: integer; guid: TGUID; begin sl := TStringList.Create; try try sl.Sorted := True; sl.Duplicates := dupError; for i := 1 to 50000 do begin CreateGUID(guid); s := GUIDToString(guid); sl.Add(s); end; writeln('Created 50,000 GUIDs'); except on e: EStringListError do begin writeln('Duplicate value detected: ' + s); writeln(' List.Count = ', sl.Count); end; on e: Exception do writeln('Unknown error: ', e.Message); end; finally sl.Free; end; end; --------------------[ end ]-------------------------------- Changing the line that adds the GUID to the StringList by prefixing the counter, all is fine. So I can't see a problem in my code. sl.Add(IntToStr(i) + s); Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/
Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/

Reply by email

Mattias Gärtner
to Graeme Geldenhuys: I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I…

Zitat von Graeme Geldenhuys :

Show quoted text 8 lines
Hi, I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I constantly got PK violations. I then wrote the code show below to create 50,000 GUID's and add them to a StringList (no duplicates allowed). Every time I run this I get a duplicate GUID at iteration 1021. My original test is supposed to create 5mil records, but I can't get past 1000! :-(

Sounds like the Debian openssl key fiasko.
Add it to the list:
http://xkcd.com/424/

Mattias

Show quoted text 65 lines
Actual program output: ./index_performance.elf Duplicate value detected: {FD3DF19D-A878-F8BA-F9D1-108658451E05} List.Count = 1021 GUID's are supposed to be very unique! Wikipedia says something like: Create a billion every second for 10 years or something before you deplete the range. So what is wrong with my code or maybe the Free Pascal implementation? --------------------[ code ]------------------------------ procedure TMainForm.btnTestClicked(Sender: TObject); var sl: TStringList; s: string; i: integer; d: integer; guid: TGUID; begin sl := TStringList.Create; try try sl.Sorted := True; sl.Duplicates := dupError; for i := 1 to 50000 do begin CreateGUID(guid); s := GUIDToString(guid); sl.Add(s); end; writeln('Created 50,000 GUIDs'); except on e: EStringListError do begin writeln('Duplicate value detected: ' + s); writeln(' List.Count = ', sl.Count); end; on e: Exception do writeln('Unknown error: ', e.Message); end; finally sl.Free; end; end; --------------------[ end ]-------------------------------- Changing the line that adds the GUID to the StringList by prefixing the counter, all is fine. So I can't see a problem in my code. sl.Add(IntToStr(i) + s); Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ _______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

------------------
Powered by NetMail

Reply by email

Marco van de Voort
to Michael Van Canneyt: As a short workaround: The default implementation on Linux uses a random() call to…
Show quoted text 12 lines
On Thu, 29 May 2008, Graeme Geldenhuys wrote: > I'm trying to do performance testing in Firebird RDBMS using GUID's > and primary keys. I constantly got PK violations. I then wrote the > code show below to create 50,000 GUID's and add them to a StringList > (no duplicates allowed). Every time I run this I get a duplicate GUID > at iteration 1021. My original test is supposed to create 5mil > records, but I can't get past 1000! :-( As a short workaround: The default implementation on Linux uses a random() call to generate a guid, combined with a timestamp, if I recall correctly.

rtl/linux/suuid.inc reads from '/proc/sys/kernel/random/uuid' Maybe it is not used though.

You can use the kernel mechanism by inluding libuuid in your uses clause. That should ensure uniqueness.

It's unportable though. It is wiser to try to find out why the default (and portable call) doesn't work on Linux, using a system generator.

Reply by email

Michael Van Canneyt
to Graeme Geldenhuys: I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I…
Graeme Geldenhuys wrote 8 lines
Hi, I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I constantly got PK violations. I then wrote the code show below to create 50,000 GUID's and add them to a StringList (no duplicates allowed). Every time I run this I get a duplicate GUID at iteration 1021. My original test is supposed to create 5mil records, but I can't get past 1000! :-(

As a short workaround:

The default implementation on Linux uses a random() call to generate a guid, combined with a timestamp, if I recall correctly. You can use the kernel mechanism by inluding libuuid in your uses clause. That should ensure uniqueness.

Michael.

Reply by email

Jonas Maebe
to Michael Van Canneyt
Michael Van Canneyt wrote
The default implementation on Linux uses a random() call to generate a guid, combined with a timestamp, if I recall correctly.

Note that random() will always return the same sequence if you don't call randomize() once in the program.

Jonas

Reply by email

Graeme Geldenhuys
to Mattias Gärtner: Zitat von Graeme Geldenhuys : Sounds like the Debian openssl key fiasko. Add it to the…

2008/5/29 Mattias Gärtner :

Sounds like the Debian openssl key fiasko.

I saw Ubuntu had some updates to resolve this issue on my system. Got the updates, rebooted, recompiled my application and still 1021 is my limit before I get duplicates.

What is really weird is that once I hit the 1021 limit, I can't run that method again. I get a duplicate after 1 add! It seems to behave exactly like random() without calling randomize(). It seems to remember the last "position" it it's seed or something and tries to continue where it left off. The same pattern, over and over.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Michael Van Canneyt: As a short workaround: The default implementation on Linux uses a random() call to…

2008/5/29 Michael Van Canneyt :

The default implementation on Linux uses a random() call to generate a guid, combined with a timestamp, if I recall correctly. You can use the kernel mechanism by inluding libuuid in your uses clause. That should ensure uniqueness.

Yup, that works fine, but like Marco said... The underlying problem still persists.
Thanks for the workaround though.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Mattias Gärtner
to Graeme Geldenhuys: 2008/5/29 Mattias Gärtner : I saw Ubuntu had some updates to resolve this issue on my…

Zitat von Graeme Geldenhuys :

Show quoted text 7 lines
2008/5/29 Mattias Gärtner : > > Sounds like the Debian openssl key fiasko. I saw Ubuntu had some updates to resolve this issue on my system. Got the updates, rebooted, recompiled my application and still 1021 is my limit before I get duplicates.

Sorry. Openssl has *nothing* to do with the CreateGUID function. But the CreateGUID calls the kernel function and if this does not work it calls Randomize and calls 16 times random(256).
But Randomize only work from time to time, so don't call it often - Bug in GetRandomBytes.
IMO GetRandomBytes needs to check if Randomize has been called. Or even better: Randomize should not reset the sequence if no random number is available.

Show quoted text 5 lines
What is really weird is that once I hit the 1021 limit, I can't run that method again. I get a duplicate after 1 add! It seems to behave exactly like random() without calling randomize(). It seems to remember the last "position" it it's seed or something and tries to continue where it left off. The same pattern, over and over.

Maybe the 1021 is the limit of /dev/random.

Mattias

Reply by email

Graeme Geldenhuys
to Marco van de Voort: rtl/linux/suuid.inc reads from '/proc/sys/kernel/random/uuid' Maybe it is not used though.

2008/5/29 Marco van de Voort :

rtl/linux/suuid.inc reads from '/proc/sys/kernel/random/uuid' Maybe it is not used though.

How do I know if it's using Random or '/proc/sys/kernel/random/uuid'? Browsing the RTL source code with Lazarus, I believe I am using the latter, but not 100% sure.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Florian Klaempfl
to Graeme Geldenhuys
Show quoted text 7 lines
2008/5/29 Marco van de Voort : > rtl/linux/suuid.inc reads from '/proc/sys/kernel/random/uuid' Maybe it is > not used though. How do I know if it's using Random or '/proc/sys/kernel/random/uuid'? Browsing the RTL source code with Lazarus, I believe I am using the latter, but not 100% sure.

If it uses /proc/sys/kernel/random/uuid, strace shows it.

Reply by email

Graeme Geldenhuys
to Mattias Gärtner: Zitat von Graeme Geldenhuys : Sorry. Openssl has *nothing* to do with the CreateGUID…

2008/5/29 Mattias Gärtner :

Sorry. Openssl has *nothing* to do with the CreateGUID function.

I thought that much, but then thought it wouldn't hurt installing the patch anyway.

Show quoted text 7 lines
> What is really weird is that once I hit the 1021 limit, I can't run > that method again. I get a duplicate after 1 add! It seems to behave > exactly like random() without calling randomize(). It seems to > remember the last "position" it it's seed or something and tries to > continue where it left off. The same pattern, over and over. Maybe the 1021 is the limit of /dev/random.

I gather that you can reproduce the issue and also get stuck on 1021?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Florian Klaempfl: If it uses /proc/sys/kernel/random/uuid, strace shows it.

2008/5/29 Florian Klaempfl :

Show quoted text 5 lines
> How do I know if it's using Random or '/proc/sys/kernel/random/uuid'? > Browsing the RTL source code with Lazarus, I believe I am using the > latter, but not 100% sure. If it uses /proc/sys/kernel/random/uuid, strace shows it.

Thanks Florian. I am using the '.../random/uuid'.

nanosleep({0, 700000000}, {3082747920, 3219878756}) = 0 open("/proc/sys/kernel/random/uuid", O_RDONLY|O_LARGEFILE) = 872 read(872, "2c4bd506-c58f-4b6d-b88c-eec01a05"..., 36) = 36 ioctl(3, FIONREAD, [0]) = 0
nanosleep({0, 700000000}, {3082747920, 3219878756}) = 0 open("/proc/sys/kernel/random/uuid", O_RDONLY|O_LARGEFILE) = 873 read(873, "ea14fbdc-8d6b-45de-ae7a-5e6e1908"..., 36) = 36 ioctl(3, FIONREAD, [0]) = 0
nanosleep({0, 700000000}, <unfinished ...>
Process 6582 detached

I'm not putting a 700ms sleep between each CreateGUID call, to see what happens. I noticed that once I hit the 1021 limit at normal speed (no sleep). Every time I click the button quickly (2-3 times per second) it instantly generates the same GUID. If I wait a second between clicks, I get a duplicate on the 2nd add to stringlist. Weird, but then, I got no idea what this will prove! :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Graeme Geldenhuys

2008/5/29 Graeme Geldenhuys :

Show quoted text 7 lines
I'm not putting a 700ms sleep between each CreateGUID call, to see what happens. I noticed that once I hit the 1021 limit at normal speed (no sleep). Every time I click the button quickly (2-3 times per second) it instantly generates the same GUID. If I wait a second between clicks, I get a duplicate on the 2nd add to stringlist. Weird, but then, I got no idea what this will prove! :-)

Finally, here is the results....

At normal speed (no sleep between CreateGUID calls:

./index_performance.elf
Duplicate value detected: {7D8EB078-7FD2-8EED-1D39-B5E4F17766FD} List.Count = 1021

... And here I have a sleep(700) between CreateGUID calls... As you can see, I managed to get two extra GUID's in my list before it hits a duplicate.

./index_performance.elf
Duplicate value detected: {486D85FC-DC44-5180-A60D-3E47D00709A4} List.Count = 1023

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Vincent Snijders
to Graeme Geldenhuys: 2008/5/29 Florian Klaempfl : Thanks Florian. I am using the '.../random/uuid'.…
Graeme Geldenhuys wrote 8 lines
2008/5/29 Florian Klaempfl : >> How do I know if it's using Random or '/proc/sys/kernel/random/uuid'? >> Browsing the RTL source code with Lazarus, I believe I am using the >> latter, but not 100% sure. > If it uses /proc/sys/kernel/random/uuid, strace shows it. Thanks Florian. I am using the '.../random/uuid'.

See also:
http://forum.soft32.com/linux/PATCH-Time-based-RFC-4122-UUID-generator-ftopict467475.html

random/uuid doesn't seem to support reading lot's of UUID is a short period.

Vincent

Reply by email

Mattias Gärtner
to Graeme Geldenhuys: 2008/5/29 Mattias Gärtner : I thought that much, but then thought it wouldn't hurt…

Zitat von Graeme Geldenhuys :

Show quoted text 6 lines
2008/5/29 Mattias Gärtner : > > Sorry. Openssl has *nothing* to do with the CreateGUID function. I thought that much, but then thought it wouldn't hurt installing the patch anyway.

It would hurt a lot *not* installing the patch. The bigger problem is that the other distributions including windows, Mac OS X, BSD, ... server must now install the blacklist too. I guess non debian systems now have bigger problems than debian.

Show quoted text 9 lines
>> What is really weird is that once I hit the 1021 limit, I can't run >> that method again. I get a duplicate after 1 add! It seems to behave >> exactly like random() without calling randomize(). It seems to >> remember the last "position" it it's seed or something and tries to >> continue where it left off. The same pattern, over and over. > > Maybe the 1021 is the limit of /dev/random. I gather that you can reproduce the issue and also get stuck on 1021?

Yes. 1019

And then pango gave:
(testproject:27449): Pango-WARNING **: error opening config file '/etc/pango/pangorc': Too many open files
Apparently CreateGUID is buggy and should not be used under Linux.

Mattias

Reply by email

Marco van de Voort
to Graeme Geldenhuys: I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I…
GUID's are supposed to be very unique! Wikipedia says something like: Create a billion every second for 10 years or something before you deplete the range. So what is wrong with my code or maybe the Free Pascal implementation?

I quicktested the FreeBSD implementation, verified it used the syscall, and the first half million are no problem under either 6.3 or (the very recent) 7.0

Reply by email

Michael Van Canneyt
to Mattias Gärtner: Zitat von Graeme Geldenhuys : It would hurt a lot *not* installing the patch. The bigger…
Mattias Gärtner wrote 31 lines
Zitat von Graeme Geldenhuys : > 2008/5/29 Mattias Gärtner : > > > > Sorry. Openssl has *nothing* to do with the CreateGUID function. > > I thought that much, but then thought it wouldn't hurt installing the > patch anyway. It would hurt a lot *not* installing the patch. The bigger problem is that the other distributions including windows, Mac OS X, BSD, ... server must now install the blacklist too. I guess non debian systems now have bigger problems than debian. > >> What is really weird is that once I hit the 1021 limit, I can't run > >> that method again. I get a duplicate after 1 add! It seems to behave > >> exactly like random() without calling randomize(). It seems to > >> remember the last "position" it it's seed or something and tries to > >> continue where it left off. The same pattern, over and over. > > > > Maybe the 1021 is the limit of /dev/random. > > I gather that you can reproduce the issue and also get stuck on 1021? Yes. 1019 And then pango gave: (testproject:27449): Pango-WARNING **: error opening config file '/etc/pango/pangorc': Too many open files Apparently CreateGUID is buggy and should not be used under Linux.

Fixed the file handle leak. Merging to fixes now.

Michael.

Reply by email

Michael Van Canneyt
to Graeme Geldenhuys: 2008/5/29 Mattias Gärtner : I thought that much, but then thought it wouldn't hurt…
Graeme Geldenhuys wrote 16 lines
2008/5/29 Mattias Gärtner : > > Sorry. Openssl has *nothing* to do with the CreateGUID function. I thought that much, but then thought it wouldn't hurt installing the patch anyway. >> What is really weird is that once I hit the 1021 limit, I can't run >> that method again. I get a duplicate after 1 add! It seems to behave >> exactly like random() without calling randomize(). It seems to >> remember the last "position" it it's seed or something and tries to >> continue where it left off. The same pattern, over and over. > > Maybe the 1021 is the limit of /dev/random. I gather that you can reproduce the issue and also get stuck on 1021?

Graeme,

can you send me your test program, so I can test my fixes ?

Michael.

Reply by email

Graeme Geldenhuys
to Mattias Gärtner: Zitat von Graeme Geldenhuys : It would hurt a lot *not* installing the patch. The bigger…

2008/5/29 Mattias Gärtner :

Show quoted text 5 lines
It would hurt a lot *not* installing the patch. The bigger problem is that the other distributions including windows, Mac OS X, BSD, ... server must now install the blacklist too. I guess non debian systems now have bigger problems than debian.

Yeah, I google'd the debian fiasco to get a better understanding. It's a *huge* screw-up!

Show quoted text 8 lines
> I gather that you can reproduce the issue and also get stuck on 1021? Yes. 1019 And then pango gave: (testproject:27449): Pango-WARNING **: error opening config file '/etc/pango/pangorc': Too many open files Apparently CreateGUID is buggy and should not be used under Linux.

I need to convert existing data to our new system, so I am in need of a lot of GUID's quickly. At least in the interim I have a workaround via the 'libuuid' unit, otherwise I would be in big trouble.

How can I force my fpc to use the 'random' option, instead of '.../random/uuid'? Or must I copy and paste the code from RTL into my own function?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Vincent Snijders: See also: http://forum.soft32.com/linux/PATCH-Time-based-RFC-4122-UUID-generator-ftopict46…

2008/5/29 Vincent Snijders :

See also: http://forum.soft32.com/linux/PATCH-Time-based-RFC-4122-UUID-generator-ftopict467475.html random/uuid doesn't seem to support reading lot's of UUID is a short period.

Unfortunately I'm on kernel 2.6.22 and that patch is only for 2.6.25. :-(

Could FPC under Linux maybe default to using 'libuuid' instead, until the '../random/uuid' issue is resolved some time in the future kernels..?
That way CreateGUID() will at least be usable now (under Linux).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Michael Van Canneyt: can you send me your test program, so I can test my fixes ? Michael.

2008/5/29 Michael Van Canneyt :

can you send me your test program, so I can test my fixes ?

Here you go. It is pretty much what I sent before, except now I made it a console application instead of a fpGUI one. Below it tries to create 50,000 GUID's. My SQL version tries 5mil.

-------------------------[ project1.pas ]----------------------------------
program Project1;
{$mode objfpc}{$H+}
uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Classes, SysUtils;
procedure GUIDTest;
var
  sl: TStringList;
  s: string;
  i: integer;
  d: integer;
  guid: TGUID;
begin
  sl := TStringList.Create;
  try
    try
      sl.Sorted := True;
      sl.Duplicates := dupError;
      for i := 1 to 50000 do
      begin
        CreateGUID(guid);
        s := GUIDToString(guid);
        sl.Add(s);
//        sl.Add(IntToStr(i) + s);
      end;
      writeln('Created ', sl.Count, ' GUIDs');
    except
      on e: EStringListError do
      begin
        writeln('Duplicate value detected: ' + s);
        writeln('  List.Count = ', sl.Count);
      end;
      on e: Exception do
        writeln('Unknown error: ', e.Message);
    end;
  finally
    sl.Free;
  end;
end;
begin
  GUIDTest;
end.

------------------------------[ end ]-----------------------------------

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Michael Van Canneyt
to Graeme Geldenhuys
Graeme Geldenhuys wrote 8 lines
2008/5/29 Michael Van Canneyt : > > can you send me your test program, so I can test my fixes ? Here you go. It is pretty much what I sent before, except now I made it a console application instead of a fpGUI one. Below it tries to create 50,000 GUID's. My SQL version tries 5mil.

Well, after my patch, it creates 500.000 as well, no errors. I had to modify your algoritm a bit to make it more efficient, and I'm too lazy to wait for 5 million.

So I guess the bug is fixed. It's merged to the fixbranch.

If you want to use your own GUID creation routine, you can always set the OnCreateGUID handler in sysutils.

Michael.

Reply by email

Vincent Snijders
to Michael Van Canneyt
Michael Van Canneyt wrote 16 lines
On Thu, 29 May 2008, Graeme Geldenhuys wrote: > 2008/5/29 Michael Van Canneyt : >> can you send me your test program, so I can test my fixes ? > > Here you go. It is pretty much what I sent before, except now I made > it a console application instead of a fpGUI one. Below it tries to > create 50,000 GUID's. My SQL version tries 5mil. Well, after my patch, it creates 500.000 as well, no errors. I had to modify your algoritm a bit to make it more efficient, and I'm too lazy to wait for 5 million. So I guess the bug is fixed. It's merged to the fixbranch.

The bug was probably there, because GetRandomBytes (which was called if the kernel function didn't work) return duplicates if it is called within the same tick count. Can you fix that too, as suggested by Mattias?

Vincent

Reply by email

Michael Van Canneyt
to Vincent Snijders
Vincent Snijders wrote 20 lines
Michael Van Canneyt schreef: > > On Thu, 29 May 2008, Graeme Geldenhuys wrote: > > > 2008/5/29 Michael Van Canneyt : > > > can you send me your test program, so I can test my fixes ? > > > > Here you go. It is pretty much what I sent before, except now I made > > it a console application instead of a fpGUI one. Below it tries to > > create 50,000 GUID's. My SQL version tries 5mil. > > Well, after my patch, it creates 500.000 as well, no errors. I had to modify > your algoritm a bit to make it more efficient, > and I'm too lazy to wait for 5 million. > So I guess the bug is fixed. It's merged to the fixbranch. > The bug was probably there, because GetRandomBytes (which was called if the kernel function didn't work) return duplicates if it is called within the same tick count. Can you fix that too, as suggested by Mattias?

I patched it so Randomize is called only once.
Merged to fixes.

Michael.

Reply by email

Mattias Gärtner
to Graeme Geldenhuys: 2008/5/29 Mattias Gärtner : Yeah, I google'd the debian fiasco to get a better…

Zitat von Graeme Geldenhuys :

Show quoted text 28 lines
2008/5/29 Mattias Gärtner : > > It would hurt a lot *not* installing the patch. The bigger problem is that the > other distributions including windows, Mac OS X, BSD, ... server must now > install the blacklist too. I guess non debian systems now have bigger problems > than debian. Yeah, I google'd the debian fiasco to get a better understanding. It's a *huge* screw-up! >> I gather that you can reproduce the issue and also get stuck on 1021? > > Yes. 1019 > > And then pango gave: > (testproject:27449): Pango-WARNING **: error opening config file > '/etc/pango/pangorc': Too many open files > Apparently CreateGUID is buggy and should not be used under Linux. I need to convert existing data to our new system, so I am in need of a lot of GUID's quickly. At least in the interim I have a workaround via the 'libuuid' unit, otherwise I would be in big trouble. How can I force my fpc to use the 'random' option, instead of '.../random/uuid'? Or must I copy and paste the code from RTL into my own function?

If the GUID are used in security contexts, then you *must* wait and gather some real random numbers.
Otherwise you can simply call Randomize once at the start of the program and use:

Procedure GetRandomBytes(Var Buf; NBytes : Integer);
Var
  I : Integer;
  P : PByte;
begin
  ;
  For I:=0 to NBytes-1 do
    P[i]:=Random(256);
end;
  Randomize; // call this only once every few seconds
  for i:=0 to 5000000 do begin
    GetRandomBytes(GUID,SizeOf(Guid));

This gave me 5 mil unique guids.

Mattias

Reply by email

Florian Klaempfl
to Vincent Snijders
Show quoted text 9 lines
Michael Van Canneyt schreef: > > I patched it so Randomize is called only once. > Merged to fixes. Thanks. BTW, the log message made me think you introduced a new bug :-) "GetRandomBytes calls random only once" instead of "GetRandomBytes calls randomize only once"

... and it is not MT safe ;) because the variable is set before the actual work is done.

Reply by email

Michael Van Canneyt
to Florian Klaempfl
Florian Klaempfl wrote 13 lines
Vincent Snijders schrieb: > Michael Van Canneyt schreef: > > > > I patched it so Randomize is called only once. > > Merged to fixes. > > Thanks. BTW, the log message made me think you introduced a new bug :-) > "GetRandomBytes calls random only once" > instead of > "GetRandomBytes calls randomize only once" ... and it is not MT safe ;) because the variable is set before the actual work is done.

.. So in the worst case, it's called twice instead of once...

But I've fixed it, with a nice log message.

Michael.

Reply by email

Graeme Geldenhuys
to Graeme Geldenhuys: I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I…

Thanks to Michael and everybody else that helped out with this issue. I'll get a FPC (svn update) and try out the fixes tomorrow.

Regards,
- Graeme -

Graeme Geldenhuys wrote 71 lines
Hi, I'm trying to do performance testing in Firebird RDBMS using GUID's and primary keys. I constantly got PK violations. I then wrote the code show below to create 50,000 GUID's and add them to a StringList (no duplicates allowed). Every time I run this I get a duplicate GUID at iteration 1021. My original test is supposed to create 5mil records, but I can't get past 1000! :-( Actual program output: ./index_performance.elf Duplicate value detected: {FD3DF19D-A878-F8BA-F9D1-108658451E05} List.Count = 1021 GUID's are supposed to be very unique! Wikipedia says something like: Create a billion every second for 10 years or something before you deplete the range. So what is wrong with my code or maybe the Free Pascal implementation? --------------------[ code ]------------------------------ procedure TMainForm.btnTestClicked(Sender: TObject); var sl: TStringList; s: string; i: integer; d: integer; guid: TGUID; begin sl := TStringList.Create; try try sl.Sorted := True; sl.Duplicates := dupError; for i := 1 to 50000 do begin CreateGUID(guid); s := GUIDToString(guid); sl.Add(s); end; writeln('Created 50,000 GUIDs'); except on e: EStringListError do begin writeln('Duplicate value detected: ' + s); writeln(' List.Count = ', sl.Count); end; on e: Exception do writeln('Unknown error: ', e.Message); end; finally sl.Free; end; end; --------------------[ end ]-------------------------------- Changing the line that adds the GUID to the StringList by prefixing the counter, all is fine. So I can't see a problem in my code. sl.Add(IntToStr(i) + s); Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/
Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/

Reply by email

Vinzent Höfler
to Jonas Maebe: Note that random() will always return the same sequence if you don't call randomize()…
Jonas Maebe wrote 7 lines
On 29 May 2008, at 13:26, Michael Van Canneyt wrote: > The default implementation on Linux uses a random() call to generate a > guid, combined with a timestamp, if I recall correctly. Note that random() will always return the same sequence if you don't call randomize() once in the program.

Yes. But don't make the mistake to call it more than once.

I've seen code that calls it before each and every Random() call to make the number "more" random, thus obviously not considering the fact that consecutive calls to Randomize() are much less random that any generated Pseudo-Random-Sequence, because the output of Randomize() usually only depends on time - and last time I checked, time was still a pretty linear thing.

If Randomize() were "truly" and "more" random, one wouldn't need Random() after all, would one?

Vinzent.

Reply by email

Graeme Geldenhuys
to Vinzent Höfler: Yes. But don't make the mistake to call it more than once. I've seen code that calls it…
Vinzent Höfler wrote 6 lines
> > Note that random() will always return the same sequence if you don't call randomize() once in the program. > Yes. But don't make the mistake to call it more than once.

Why can't FPC automatically call randomize() in the RTL. Put it in some initialization section. That way, at application startup, randomize() is already called and only Random() needs to be used?

I could never figure out why this wasn't done in Delphi either? After all, if you use Random(), you want something random, yet many developers make the common mistakes of not calling Randomize() or calling it to often. If FPC handled that for us, nobody would every make those mistakes again!

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Marco van de Voort
to Graeme Geldenhuys

[ Charset ISO-8859-1 unsupported, converting... ]

Show quoted text 11 lines
On 31/05/2008, Vinzent H?fler wrote: > > > > Note that random() will always return the same sequence if you don't call > randomize() once in the program. > > > > Yes. But don't make the mistake to call it more than once. Why can't FPC automatically call randomize() in the RTL. Put it in some initialization section. That way, at application startup, randomize() is already called and only Random() needs to be used?

Then the whole random system is linked in, even when not used.

Also when you would want to set a different randseed, you'd call it twice (though I can't imagine that being such a problem nowadays)

Reply by email

Florian Klaempfl
to Graeme Geldenhuys: Why can't FPC automatically call randomize() in the RTL. Put it in some initialization…
Show quoted text 14 lines
On 31/05/2008, Vinzent Höfler wrote: >> Note that random() will always return the same sequence if you don't call > randomize() once in the program. > Yes. But don't make the mistake to call it more than once. Why can't FPC automatically call randomize() in the RTL. Put it in some initialization section. That way, at application startup, randomize() is already called and only Random() needs to be used? I could never figure out why this wasn't done in Delphi either? After all, if you use Random(), you want something random, yet many developers make the common mistakes of not calling Randomize() or calling it to often. If FPC handled that for us, nobody would every make those mistakes again!

People might want to start with a defined randseed to reproduce behaviour.

Reply by email

Jonas Maebe
to Marco van de Voort: [ Charset ISO-8859-1 unsupported, converting... ] Then the whole random system is linked…
Marco van de Voort wrote 5 lines
> Why can't FPC automatically call randomize() in the RTL. Put it in > some initialization section. That way, at application startup, > randomize() is already called and only Random() needs to be used? Then the whole random system is linked in, even when not used.

Randomize only causes the randseed variable to be initialised/used.

Jonas

Reply by email

Thaddy de Koning
to Marco van de Voort: [ Charset ISO-8859-1 unsupported, converting... ] Then the whole random system is linked…
> Why can't FPC automatically call randomize() in the RTL. Put it in > some initialization section. That way, at application startup, > randomize() is already called and only Random() needs to be used?

Please not! This would hamper all kinds of scientific, statistical and financial modelling. For all those (and more) it is sometimes necessary to have a *reproducable* random distribution.
There's more to random than random ;-)

Reply by email

Graeme Geldenhuys
to Marco van de Voort: [ Charset ISO-8859-1 unsupported, converting... ] Then the whole random system is linked…
Marco van de Voort wrote
Then the whole random system is linked in, even when not used.

I would have thought this is already linked in as part of the RTL. Simply not initialized until the first Randomize() call.

Also when you would want to set a different randseed, you'd call it twice (though I can't imagine that being such a problem nowadays)

Ok, valid point. Even so, I still think many people use Randomize() and Random() incorrectly which causes more harm (unexpected results) compared to *maybe* having to set a different randseed and calling randomize() twice.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Daniël Mantione
to Florian Klaempfl: People might want to start with a defined randseed to reproduce behaviour.

Op Mon, 2 Jun 2008, schreef Florian Klaempfl:

Show quoted text 6 lines
> all, if you use Random(), you want something random, yet many > developers make the common mistakes of not calling Randomize() or > calling it to often. If FPC handled that for us, nobody would every > make those mistakes again! People might want to start with a defined randseed to reproduce behaviour.

This implies createguid should not call randomize automatically either, it prevents you having deterministic behaviour, especially in a program where guids and normal use of the randomizer are combined.

Daniël

Reply by email

Vincent Snijders
to Daniël Mantione
Daniël Mantione wrote 15 lines
Op Mon, 2 Jun 2008, schreef Florian Klaempfl: >> all, if you use Random(), you want something random, yet many >> developers make the common mistakes of not calling Randomize() or >> calling it to often. If FPC handled that for us, nobody would every >> make those mistakes again! > > People might want to start with a defined randseed to reproduce > behaviour. This implies createguid should not call randomize automatically either, it prevents you having deterministic behaviour, especially in a program where guids and normal use of the randomizer are combined.

It is not a problem anymore. Call CreateGuid once, set the randseed and you get predictable guids afterwards.

Vincent

Reply by email

Michael Van Canneyt
to Daniël Mantione: Op Mon, 2 Jun 2008, schreef Florian Klaempfl: This implies createguid should not call…
Daniël Mantione wrote 14 lines
Op Mon, 2 Jun 2008, schreef Florian Klaempfl: > > all, if you use Random(), you want something random, yet many > > developers make the common mistakes of not calling Randomize() or > > calling it to often. If FPC handled that for us, nobody would every > > make those mistakes again! > > People might want to start with a defined randseed to reproduce behaviour. This implies createguid should not call randomize automatically either, it prevents you having deterministic behaviour, especially in a program where guids and normal use of the randomizer are combined.

Creating GUIDs should never be deterministic.. ?

Michael.

Reply by email

Daniël Mantione
to Michael Van Canneyt

Op Mon, 2 Jun 2008, schreef Michael Van Canneyt:

Show quoted text 7 lines
>> People might want to start with a defined randseed to reproduce behaviour. > > This implies createguid should not call randomize automatically either, it > prevents you having deterministic behaviour, especially in a program where > guids and normal use of the randomizer are combined. Creating GUIDs should never be deterministic.. ?

For debugging it can be usefull to have deterministic behaviour. But that was not the point. The point was, that a call to randomize "disrupts" the current state of the random generator. That means that if you use random outside guid creation, creating a guid can have unintended side effects on your random generation.

I agree it is nitpicking, but strictly speaking, it is a bad idea to automatically initialize the random generator.

Daniël

Reply by email

Klaus Hartnegg
to Daniël Mantione
Daniël Mantione wrote 8 lines
Op Mon, 2 Jun 2008, schreef Michael Van Canneyt: > Creating GUIDs should never be deterministic.. ? a call to randomize "disrupts" the current state of the random generator. That means that if you use random outside guid creation, creating a guid can have unintended side effects on your random generation.

Maybe the GUID generator should have an automatic call to randomize, but use its own randseed (or preserve the original value):

randseed_old := randseed;
if not guid_rnd_initialized then begin
    randomize;
    guid_rnd_initialized := true;
end
else
    randseed := guid_randseed;
result := rnd2guid(random);
guid_randseed := randseed;
randseed := randseed_old;

Reply by email

Vincent Snijders
to Klaus Hartnegg
Klaus Hartnegg wrote 18 lines
> a call to randomize "disrupts" the current state of the random > generator. That means that if you use random outside guid creation, > creating a guid can have unintended side effects on your random > generation. Maybe the GUID generator should have an automatic call to randomize, but use its own randseed (or preserve the original value): randseed_old := randseed; if not guid_rnd_initialized then begin randomize; guid_rnd_initialized := true; end else randseed := guid_randseed; result := rnd2guid(random); guid_randseed := randseed; randseed := randseed_old;

It is not that easy to restore the state of random number generator.

Vincent

Reply by email

Vinzent Höfler
to Vincent Snijders
Vincent Snijders wrote 21 lines
Klaus Hartnegg schreef: >> a call to randomize "disrupts" the current state of the random >> generator. That means that if you use random outside guid creation, >> creating a guid can have unintended side effects on your random >> generation. > > Maybe the GUID generator should have an automatic call to randomize, > but use its own randseed (or preserve the original value): > > randseed_old := randseed; > if not guid_rnd_initialized then begin > randomize; > guid_rnd_initialized := true; > end > else > randseed := guid_randseed; > result := rnd2guid(random); > guid_randseed := randseed; > randseed := randseed_old; It is not that easy to restore the state of random number generator.

"Not that easy" is quite an understatement here. ;) It's very close to impossible without changing the System-unit. That was one issue I had. The other one was simple that System's RNG is not thread-safe, so I finally decided to use an RNG on a per object basis here.

This makes the necessary locking of a system wide RNG in threading context much easier and additionally I can still use a "deterministic" RNG in contexts where I need it, because now I can have as many "dedicated" RNGs as I like. :)

Maybe, such Random class should be incorporated in the FCL, too?

Vinzent.

Reply by email