Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2016-08

Are the TEncoding singletons thread-safe?

15 messages · 4 people ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

silvioprog

Hello,

I've used the useful singletons available in the TEncoding class, and I don't get any problem with them, unless using threads.

I have an application that does many tests in a code compilable in FPC and Delphi, however, the FPC's tests raise memory leak at the finalization tests. The problem can be a little bit hard to be reproduce, but I think the code below can do that:

=== begin code [project1.dpr] ===

program project1;
{$IFDEF FPC}
{$MODE DELPHI}
{$ENDIF}
{$IFDEF MSWINDOWS}
{$APPTYPE CONSOLE}
{$ENDIF}
uses
//  HeapTrc, the FPC staff recomends to enable HeapTrc in the lpi or
passing the -gh compiler parameter
{$IFDEF UNIX}
  CThreads,
{$ENDIF} SysUtils, Classes;
type
  TTestThread = class(TThread)
  public
    constructor Create;
    procedure Execute; override;
  end;
  constructor TTestThread.Create;
  begin
    inherited Create(True);
    FreeOnTerminate := False;
  end;
  procedure TTestThread.Execute;
  begin
    WriteLn(TEncoding.Default.GetString(TBytes.Create(65, 66, 67)));
  end;
var
  VThread1, VThread2: TTestThread;
begin
{$IFNDEF FPC}
  ReportMemoryLeaksOnShutdown := True;
{$ENDIF}
  VThread1 := TTestThread.Create;
  VThread2 := TTestThread.Create;
  try
    Sleep(1000);
    VThread1.Start;
    VThread2.Start;
    VThread1.WaitFor;
    VThread2.WaitFor;
  finally
    VThread1.Free;
    VThread2.Free;
  end;
end.

=== end code ===

If you compile and run the code above, you may get the following trace:

=== begin log ===

ABC
ABC
Heap dump by heaptrc unit
44 memory blocks allocated : 2119/2144
43 memory blocks freed : 2087/2112
1 unfreed memory blocks : 32
True heap size : 131072
True free heap : 131072
Should be : 130912
Call trace for block $00007FFFF7FAB0C0 size 32

=== end log ===

Researching the FPC's TEncoding implementation, I think it doesn't do any lock before instantiate its singletons, so I decided to research how Delphi implements it, because I can't get any error on it, and I've seen it implements a "lock-free initialization" technique
<http://stackoverflow.com/a/24657705/3268398> using the XE's AtomicCmpExchange() function before return any TEncoding global instance.

That memory leak can be really a problem, because I can't prevent how many threads will call the TEncoding's singletons at same time, I could initialize the TEncoding before starting any thread of my application, but it doesn't sound good, because I can't prevent if the threads will instantiate the ANSI or UTF8 singleton. The best way to solve this problem would be making the TEncoding become thread-safe. I could send a patch to solve it, but I don't know if FPC provides some global locker like AtomicCmpExchange.

Silvio Clécio

Reply by email

Bart
to silvioprog
silvioprog wrote 5 lines
... so I decided to research how Delphi implements it, because I can't get any error on it, and I've seen it implements a "lock-free initialization" technique <http://stackoverflow.com/a/24657705/3268398> using the XE's AtomicCmpExchange() function before return any TEncoding global instance.

......

I could send a patch to solve it,

I hope you did not look at the Delphi sourcecode to see how it was implemented, because that would now make a "clean room" implementation on our side almost impossible.

Bart

Reply by email

Jonas Maebe
to Bart
Bart wrote 13 lines
On 8/11/16, silvioprog wrote: > ... so I decided to research how Delphi > implements it, because I can't get any error on it, and I've seen it > implements a "lock-free initialization" technique > <http://stackoverflow.com/a/24657705/3268398> using the XE's > AtomicCmpExchange() function before return any TEncoding global instance. ...... > I could send a patch to solve it, I hope you did not look at the Delphi sourcecode to see how it was implemented, because that would now make a "clean room" implementation on our side almost impossible.

It would make a clean room implementation by Silvio himself impossible, but anyone that reads his description and writes code based on that description is still clean (as long as they did not also look at the Delphi source code).

Jonas

Reply by email

silvioprog
to Jonas Maebe
Jonas Maebe wrote 20 lines
Bart wrote: > On 8/11/16, silvioprog wrote: > >> ... so I decided to research how Delphi >> implements it, because I can't get any error on it, and I've seen it >> implements a "lock-free initialization" technique >> <http://stackoverflow.com/a/24657705/3268398> using the XE's >> AtomicCmpExchange() function before return any TEncoding global instance. > ...... >> I could send a patch to solve it, > > I hope you did not look at the Delphi sourcecode to see how it was > implemented, because that would now make a "clean room" implementation > on our side almost impossible. It would make a clean room implementation by Silvio himself impossible, but anyone that reads his description and writes code based on that description is still clean (as long as they did not also look at the Delphi source code).

I've opened a new issue sending a possible patch: http://bugs.freepascal. org/view.php?id=30462.

Silvio Clécio

Reply by email

silvioprog
to Jonas Maebe
Jonas Maebe wrote 8 lines
On 11/08/16 15:17, silvioprog wrote: > I've opened a new issue sending a possible > patch: http://bugs.freepascal.org/view.php?id=30462 > <http://bugs.freepascal.org/view.php?id=30462>. > Did you look at the Delphi source code for TEncoding?

No, I didn't. Sorry if I made some misunderstood, I look at Delphi source just for check why it doesn't raise memory leak, I'm fully against piracy! I look at other sources from another languages too, like DotGNU Encoding, but just for analysis. The proposed possible patch is 100% genuine FPC implementation, and unlike Delphi, I've used a plain critical section for cross platform purposes.

Silvio Clécio

Reply by email

Jonas Maebe
to silvioprog
silvioprog wrote 6 lines
No, I didn't. Sorry if I made some misunderstood, I look at Delphi source just for check why it doesn't raise memory leak, I'm fully against piracy! I look at other sources from another languages too, like DotGNU Encoding, but just for analysis. The proposed possible patch is 100% genuine FPC implementation, and unlike Delphi, I've used a plain critical section for cross platform purposes.

How do you know that Delphi doesn't use a critical section if you did not look at its source code?

Jonas

Reply by email

silvioprog
to Jonas Maebe
Jonas Maebe wrote 12 lines
On 11/08/16 15:38, silvioprog wrote: > No, I didn't. Sorry if I made some misunderstood, I look at Delphi > source just for check why it doesn't raise memory leak, I'm fully > against piracy! I look at other sources from another languages too, like > DotGNU Encoding, but just for analysis. The proposed possible patch is > 100% genuine FPC implementation, and unlike Delphi, I've used a plain > critical section for cross platform purposes. > How do you know that Delphi doesn't use a critical section if you did not look at its source code?

Now I understand what you meant. When you asked me if I looked at Delphi's code I thought you were suggesting that I had written the patch based on it to make a kind of copy, if so, I haven't. But if the question is about looking at Delphi code, yes, I do, I bought a professional Delphi copy and its license allows me to see any unit that they've sent me. Finally, I can analyse any code of any tool if its license allows me, can't I?

Well, to avoid more confusion I'll not send more patches to solve this problem and I'm going to close my issue.

Thank you!

Silvio Clécio

Reply by email

Jonas Maebe
to silvioprog
silvioprog wrote 7 lines
Now I understand what you meant. When you asked me if I looked at Delphi's code I thought you were suggesting that I had written the patch based on it to make a kind of copy, if so, I haven't. But if the question is about looking at Delphi code, yes, I do, I bought a professional Delphi copy and its license allows me to see any unit that they've sent me. Finally, I can analyse any code of any tool if its license allows me, can't I?

Yes, but we cannot integrate any code that has been written by you based on the Delphi code you looked at, because the license does not allow you to rewrite that code into something that is licensed under the GPLv2 or later.

While in theory the issue is not as black and white (there is no definition in copyright of when exactly a work is independent and when it is derivative), to be on the safe side we do not accept code that has been written by someone after they looked at the Delphi code for the same functionality.

Well, to avoid more confusion I'll not send more patches to solve this problem and I'm going to close my issue.

I have simply removed your code from the bug report, that is enough. Your description of how to fix it does not pose any problems.

Jonas

Reply by email

silvioprog
to Jonas Maebe
Jonas Maebe wrote 5 lines
Yes, but we cannot integrate any code that has been written by you based on the Delphi code you looked at, because the license does not allow you to rewrite that code into something that is licensed under the GPLv2 or later.

While in theory the issue is not as black and white (there is no definition

in copyright of when exactly a work is independent and when it is derivative), to be on the safe side we do not accept code that has been written by someone after they looked at the Delphi code for the same functionality.

I understood and agreed.

But now I have a question: many Free Pascal programmers are Delphi programmers too, having code that can be compiled in both compilers, and sometimes they need to debug deeply, entering into the compiler code and looking at it, so, can't a programmer contribute with Free Pascal in that situation? If so, I have no more questions.

Well, to avoid more confusion I'll not send more patches to solve this

Show quoted text 5 lines
> problem and I'm going to close my issue. > I have simply removed your code from the bug report, that is enough. Your description of how to fix it does not pose any problems.

No problem. Thank you!

Silvio Clécio

Reply by email

Jonas Maebe
to silvioprog
silvioprog wrote 5 lines
But now I have a question: many Free Pascal programmers are Delphi programmers too, having code that can be compiled in both compilers, and sometimes they need to debug deeply, entering into the compiler code and looking at it, so, can't a programmer contribute with Free Pascal in that situation? If so, I have no more questions.

The compiler code is no problem, since the Delphi compiler code is not open source. For the RTL: if they looked at parts of the Delphi RTL source code, then they cannot contribute to the same functionality in the FPC RTL.

Jonas

Reply by email

silvioprog
to Jonas Maebe
Jonas Maebe wrote 12 lines
On 11/08/16 20:25, silvioprog wrote: > But now I have a question: many Free Pascal programmers are Delphi > programmers too, having code that can be compiled in both compilers, and > sometimes they need to debug deeply, entering into the compiler code and > looking at it, so, can't a programmer contribute with Free Pascal in > that situation? If so, I have no more questions. > The compiler code is no problem, since the Delphi compiler code is not open source. For the RTL: if they looked at parts of the Delphi RTL source code, then they cannot contribute to the same functionality in the FPC RTL.

Hm... now I understood and fully agreed. Since Delphi's TEncoding is part of its RTL and I've looked at its implementation to understand why I can't get leaks compiling my code using it, and there is no definition in copyright of when exactly a work is independent and when it is derivative, so anything I do can be understood as rewrite.

Silvio Clécio

Reply by email

Marcos Douglas
to Jonas Maebe: The compiler code is no problem, since the Delphi compiler code is not open source. For…
Jonas Maebe wrote 12 lines
On 11/08/16 20:25, silvioprog wrote: > > But now I have a question: many Free Pascal programmers are Delphi > programmers too, having code that can be compiled in both compilers, and > sometimes they need to debug deeply, entering into the compiler code and > looking at it, so, can't a programmer contribute with Free Pascal in > that situation? If so, I have no more questions. The compiler code is no problem, since the Delphi compiler code is not open source. For the RTL: if they looked at parts of the Delphi RTL source code, then they cannot contribute to the same functionality in the FPC RTL.

A question:
If FPC use the same names, Classes and methods, how we know if a patch was or not a copy?

Best regards,
Marcos Douglas

Reply by email

Marcos Douglas
to Jonas Maebe
Jonas Maebe wrote 7 lines
On 11/08/16 21:17, Marcos Douglas wrote: > > If FPC use the same names, Classes and methods, how we know if a patch > was or not a copy? We ask in case of doubt.

Understood, just for the record.

Reply by email