Dear All.

Currently to implement logger we use the following approach (the code was implemented for Delphi/Kylix/FPC).
Logger is intended for displaying in the GUI outputs of several threads and child processes and also for storage into internal bases.

Logic is the following:

Get stdout handle (duplicate it under linux),
create pipe,
replace the stdout (keeping the old stdout) for current process with write handle of pipe,

There are following problems with FPC
Under Windows: I have to call *rewrite(output) *for every thread which wants to use new (captured) stdout

So the questions:
How to force all the threads of process and all DLLs write into same captured stdout?
How to revert stdout back (stop capturing)?

procedure TLogger.Start;
{$IFDEF LINUX}
{$ifdef fpc}
type TPipeDescriptors=TFilDes;
{$endif}
var
   pds:TPipeDescriptors;
   {$ENDIF}
begin
  if fCaptureStdout then
   {$IFDEF MSWINDOWS}
     hConsole := GetStdHandle(STD_OUTPUT_HANDLE);
   {$ELSE}
     {$ifdef fpc}
     hConsole := fpdup(StdOutputHandle);;//stdout;
     {$else}
     hConsole := dup(STDOUT_FILENO);;//stdout;
     {$endif}
   {$ENDIF}
   {$IFDEF MSWINDOWS}
     CreatePipe(hReadPipe, hWritePipe, nil, 0);
   {$ENDIF}
   {$IFDEF LINUX}
     {$ifdef fpc}
     fppipe(pds);
     hReadPipe:=pds[0];
     hWritePipe:=pds[1];
     {$else}
     pipe(pds);
     hReadPipe:=pds.ReadDes;
     hWritePipe:=pds.WriteDes;
     {$endif}
   {$ENDIF}
   if fCaptureStdout then
   begin
     {$IFDEF MSWINDOWS}
       if not SetStdHandle(STD_OUTPUT_HANDLE, hWritePipe)
       then MessageBox(0,PChar(Format('Function: SetStdHandle(%x,%x),
Error: %d',[STD_OUTPUT_HANDLE, hWritePipe, GetLastError])),'Failed',0);
      {$IFDEF fpc}
* StdOutputHandle:=hWritePipe; // modify global runtime handle*
      {$ENDIF}
       if GetStdHandle(STD_OUTPUT_HANDLE)hWritePipe then
       begin
         safeputs('Looks like SetStdHandle failed in Windows 7, will try
AllocConsole workaround'#13#10,hWritePipe);
         if not fWinConsole then
         begin
           AllocConsole;
           FreeConsole;
           SetStdHandle(STD_OUTPUT_HANDLE, hWritePipe);
         end;
         if GetStdHandle(STD_OUTPUT_HANDLE)hWritePipe then
         begin
           safeputs('AllocConsole workaround failed, will use
GlobalWritePipe workaround'#13#10,hWritePipe);
           GlobalWritePipe:=hWritePipe;
           TTextRec(output).Handle:=hWritePipe;
         end;
       end;
*rewrite(output);*
     {$ENDIF}
     {$IFDEF LINUX}
       {$ifdef fpc}
       fpdup2(hWritePipe, StdOutputHandle);
       {$else}
       dup2(hWritePipe, STDOUT_FILENO);
       {$endif}
     {$ENDIF}
   end;
end;

finalizing capture made via

procedure TLogger.Stop;
var dummy:integer;
{$IFDEF LINUX}
   p:pointer;
{$ENDIF}
begin
     if fExitFlag then Exit;
     fExitFlag := True;
   // make empty write to wake up the thread sleeping on read operation
   dummy:=0;
   FileWrite(hWritePipe,dummy,1);
   if hThread0 then
   begin
     {$IFDEF MSWINDOWS}
     if WaitForSingleObject(hThread,2000)=WAIT_TIMEOUT
     then begin
       TerminateThread(hThread,0);
       MessageBeep(UINT(-1));
       writeln('Thread was terminated abnormally');
     end;
     Closehandle(hThread);
     {$ENDIF}
     {$IFDEF LINUX}
         {$ifndef FPC} // Kylix
       pthread_join(hThread,nil); //Unfortunately, will wait INFINITEly
       pthread_detach(hThread);
       {$else}
       WaitForThreadTerminate (hThread, 2000); // implies pthread_join
       KillThread(hThread); // implies pthread_detach and pthread_kill
       {$endif}
     {$ENDIF}
     hThread:=0;
   end;
   if fCaptureStdout then
   begin
     {$IFDEF MSWINDOWS}
       SetStdHandle(STD_OUTPUT_HANDLE, hConsole);
       if GlobalWritePipe = hWritePipe then
GlobalWritePipe:=INVALID_HANDLE_VALUE;
     {$ENDIF}
     {$IFDEF LINUX}
       {$ifdef fpc}
       fpdup2(hConsole, StdOutputHandle);
       {$else}
       dup2(hConsole, STDOUT_FILENO);
       {$endif}
     {$ENDIF}
     rewrite(output);
   end;
     if hReadPipe  0 then
     begin
     FileClose(hReadPipe);
         hReadPipe := 0;
     end;
   if hWritePipe  0 then
     begin
     FileClose(hWritePipe);
         hWritePipe := 0;
     end;
     if IsLogFileOpened then
     begin
     FileClose(hLogFile);
         hLogFile := 0;
     end;
end;

There is no way that I know of to avoid the rewrite(), since the file descriptor records for Input, Output and StdErr are threadvars. And the threadvars must be initialized on thread start.

How to revert stdout back (stop capturing)?

Close and reopen using the standard filedescriptors ?

Michael.

You can write a "classic" pascal textfile device driver and assign directly to the global TEXT variables Input, Output and ErrOutput. Don't know is this is threadsafe, though.
I have some code by Peter Below on archive for that, that also works on FPC win. The implementation part of that unit can be adapted for nixen. the file is called streamio.pas and can be easily found on the codegear website, but I will be happy to send it to you if you want by email.

No, it will not be thread-safe. You'll have to do it over and over again for each thread.

I have some code by Peter Below on archive for that, that also works on FPC win. The implementation part of that unit can be adapted for nixen. the file is called streamio.pas and can be easily found on the codegear website, but I will be happy to send it to you if you want by email.

The same unit and implementation exists in FPC since years. You can assign any classical pascal textfile to an arbitrary TStream descendent.

Michael.

In our previous episode, said:

Just curious:Why not? I thought it was safe if you flushed before you created threads?

Writing to the files is thread-safe, but in new threads, the file variables are initialized from their initial main thread values, i.e. the ones without the recapturing.

Unless I missed something ?

Michael.

In our previous episode, said:

Some stuff is dawning on me, never mind.

It is not the same - it is just called the same - but can render the same effect.
Which also means that it looks like it only takes some normal precautions for threads, otherwise the whole console IO system wouldn't be usable with threads.
I guess Marco is right: Just flush after thread operations. On its own it is not threadsafe, I agree, but it might be a simple and solid solution. *without* the rewrites :)

Sorry, Michael,

Ignore my last post.
I am testing it and "strange things happen" :0

In our previous episode, Thaddy said:

I guess Marco is right: Just flush after thread operations. On its own it is not threadsafe, I agree, but it might be a simple and solid solution. *without* the rewrites :)

IIRC there is a pointer in the filetype that points into the buffer. (bufptr). Just copying that means they still point into the same pointer.

Marco van de Voort wrote

solid solution. *without* the rewrites :) IIRC there is a pointer in the filetype that points into the buffer. (bufptr). Just copying that means they still point into the same pointer. _______________________________________________

Yes, They (input, output err) are vars, not threadvars in system. So whatever thread, they point to the same.

They are threadvars in trunk:

ThreadVar

   ThreadID    : TThreadID;
   { Standard In- and Output }
   ErrOutput,
   Output,
   Input,
   StdOut,
   StdErr      : Text;

Michael.

That's not Delphi compatible, btw: ;) : op.cit: system.pas var
...........
Input: Text; { Standard input }
Output: Text; { Standard output }
ErrOutput: Text; { Standard error output }

It may be a good thing, though.
It also explains my strange results in testing D vs FPC this morning.

?? 25.11.2010 10:14, ???????(??):

So the question is:
How to reinitialize RTL to have new threads started with working output file (i.e. without need of rewrite).

regards,
Anton

In Delphi it works - with a critical section - without re-initialization because it's var,
In FPC there's no easy work-around because it is threadvar.

It's a nice puzzle, though :)

?? 25.11.2010 11:39, Thaddy ???????(??):

That is!

Due to incompatibility of RTL between FPC and Delphi my capturing tricks not work.
BTW what the real reason to make these files as threadvar? Per-thread StdOut, StdIn, StdErr (ooooo! the application with multiple standard outputs)?

regards,
Anton

Anton Kavalenka wrote

Due to incompatibility of RTL between FPC and Delphi my capturing tricks not work. BTW what the real reason to make these files as threadvar?

To prevent multiple threads writing to stdin/stdout/sterr at the same time from corrupting each other's data (all of these text files use an internal buffer, even when the output is immediately flushed afterwards).

Jonas

Anton Kavalenka wrote

?? 25.11.2010 11:39, Thaddy ???????(??):

Yes, How "standard" is standard? ....when you make it a threadvar. Should I submit a bug report? It seems like I should: Because the file drivers allow for redirection anyway and with good reason.

They are threadvars in trunk:

Of course if stdin, stdout, and stderr are not thredvars, their use is not thread safe.

But if they are threadvars, how are they initialized ? Supposedly the main thread instances are initialized as everybody expects and thus connected to the appropriate pipes (or whatever this is in Windows). But what are the instances for threads connected to ?

-Michael

That's not a task for a compiler or frame work, but a task for the programmer.
Std is short for standard and should be standard. Someone screwed up his semantics and thereby the code when this anomaly was invented for FPC. Although the rationale about why this happened is obvious, I have to agree.

?? 25.11.2010 12:01, Jonas Maebe ???????(??):

Is not the better way to use something like this:

lock_output();
try
   // do something nasty with the buffer
finally
    flush(output);
   unlock_output();
end;

There is NO other runtimes on this planet instead FPC RTL which take care of per-thread buffer of output.
It is programmers duty to lock/flush buffer in multithreaded environments.

If FPC wants to be so nice - let's introduce critical section in output structure.
Any outputs would be suspended until a thread acquires it by write operation.

regards,
Anton

To quote some fiction: "There can be only one."
In the context of stdxxx.

On 25-11-2010 11:08, Michael Schnell wrote:
But what are the instances for threads connected to ?

This is - should be - opaque.

Regard,

Thaddy

In our previous episode, Anton Kavalenka said:

Such solutions are fine for slow apps, but in a fast pace, app, this means that the logging with its locks has a significant impact on the behaviour of the application.

And IMHO the RTL should be workable for all kinds of apps.

?? 25.11.2010 12:39, Marco van de Voort ???????(??):

OK, let's replace threadvar for those variables with var.

regards,
Anton

It's obviously better for Delphi compatibility and for what you want to do, but in general I'd say that's simply a different way. It's also worse in other ways: performance (not just locking/unlocking, which doesn't matter that much since stdio is never that fast, but mainly creating the mutexes when the threads are started), backwards compatibility with existing FPC code, resources required (mutexes require more than threadvars).

I can't imagine that any of that matters much to you though, since it does not help you solve your problem. I can't immediately think of a way to make this programmatically changeable at run time either.

There is NO other runtimes on this planet instead FPC RTL which take care of per-thread buffer of output.

That would very much surprise me.

It is programmers duty to lock/flush buffer in multithreaded environments.

I don't think it's a programmer's duty to define a global lock that they manually acquire/release whenever they use writeln() to print something to stdout in a multi-threaded program using writeln(). Just like the heap manager, ioresult, and the widestring manager are also thread-safe by default. In general, I think that all "global" system unit functionality is thread safe by default. The only part of the standard RTL that isn't, is afaik the crt unit.

Jonas

It is about semantics in this case, a "contract" if you want. This contract hasn't been defined, except for implementation. It seems that is not good enough in the context of what FPC is trying to achieve.

> There is NO other runtimes on this planet instead FPC RTL which take > care of per-thread buffer of output. That would very much surprise me.

Example needed... In very bended Dutch: go your gang ;-)

In my opinion it is. Furthermore it is a classic example of when and how the programmer should not rely on what's given to him beforehand. If "standard" is interpreted als globally valid (Unless...). Are we wrong to assume this?
But I am also happy sending a bug report through Quality Central to ask for FPC compliance.
AFAIK it should be a lock, and a lock written by the programmer. It should definitely not be a threadvar in my opinion.

Hey, It's Friday!

Regards,

Thaddy

They are initialised when the thread is started (before your own routine is called) to the same values that are used for StdIO on application startup.

Regards,
Sven

?? 25.11.2010 13:14, Sven Barth ???????(??):

So the new question:
What I have to do to properly initialize these defaults for new threads AFTER capturing StdOut?

regards,
Anton

Sven Barth wrote

They are initialised when the thread is started (before your own routine is called) to the same values that are used for StdIO on application startup.

That means the StdIn, StdOut, and StdErr are opened multiple times by the application ? While this might be possible for StdOut and StdErr (mixing the outputs), I don't see what this means for StdIn.

-Michael

Anton Kavalenka wrote

What I have to do to properly initialize these defaults for new threads AFTER capturing StdOut?

Store a copy of your stdout in a global variable, and after creating a new thread

   close(stdout);
   stdout:=myglobalstdout;

(and maybe the same for "output").

One possible problem: the output handles are automatically flushed when a thread exits (so that may cause synchronisation issues).

Jonas

Actually, that won't work because the different threads will then work on a common buffer but with distinct pointers into it. A better solution is probably to do this in the intialisation code of each thread instead:

{$ifdef unix}
  fpclose(ttextrec(stdout).handle);
{$elsif defined(MSWINDOWS)}
  { this is a copy of do_close() from the rtl, I don't know whether
    a new handle from a thread can actually have any of these values }
  if (handle  StdInputHandle) and
     (handle  StdOutputHandle) and
     (handle  StdErrorHandle) then
    CloseHandle(ttextrec(stdout).handle);
{$else}
  {$error Add support for this platform}
{$endif}
ttextrec(stdout).handle:=myglobalstdouthandle;

Jonas

That's unsuitable. I have lots of modules and lost of threads. Many modules built with C++ runtime, others with C.
Output goes from many places and many threads.
The task is to do inside process what the UNIX tee command does outside.

regards,
Anton

Anton Kavalenka wrote

At 25.11.2010 19:57, Jonas Maebe wrote: > On 25 Nov 2010, at 13:21, Jonas Maebe wrote:

the std streams should be on a per process basis (implementation as var), not on a per thread basis (Iimplementation as threadvar). Not only is this not Delphi compliant, it is also not Posix compliant. Although I can see some neat advantages, it seems to me FPC should better comply with what is expected by virtually every other language since FORTRAN and up until and including all .NET. Plz consider this behaveour to change.

This is the world upside down. Both for nixes and windows, you would expect the streams on a per process basis, not per thread, and handle possible contention by code.
i.e. it is - should be - expected that they work on a common buffer/pointer by definition.

The C++ and C threads won't have their input/output handles replaced by default, since they are not started via the FPC RTL. The only exception is in case they call back into Pascal code and they are subsequently hooked by the RTL because the Pascal code (indirectly) accesses a threadvar.

Coincidentally, just the other day we were discussing introducing the ability to install hooks that will be called automatically whenever an external thread is hooked by the RTL (since the RTL has to initialise a bunch of things at such a point, so may user code). You could do whatever initialisation you need to do at that point.

I think we could extend such a callback mechanism to all threads, possibly with a boolean parameter indicating whether the thread was started via the FPC RTL or not in case the difference is important in some use cases.

Jonas

I think this is not bad; the question is whether we can do it on unix. On Unix, we can currently only check when a threadvar is accessed.

Michael.

Michael Van Canneyt wrote

I think this is not bad; the question is whether we can do it on unix. On Unix, we can currently only check when a threadvar is accessed.

The semantics of those callbacks would be that they are called as soon as the RTL is initialised for a thread. On Unix, this is indeed only done on the first threadvar access.

Jonas

If we decide to add this, maybe we should give the users a chance to test for this themselves ?

Procedure CheckThreadInitialize;

This way they can call this at the entry point of some externally called routines. For windows, we'd need this 'check' procedure anyway, since there are 2 notification mechanisms. (TLS callback and DLL_THREAD_ATTACH)
If the RTL is already initialized for this thread, it would do nothing.

Michael.

I'm not sure if closing the handle using the platform specific API is actually a good idea here, because the RTL is probably not opening the file in this case and trying to close it multiple times (in different threads) may have unwanted effects (it would result in a failure return code probably).

BTW, other alternative solutions for the original problem (not necessarily fitting your needs, just for completeness sake):

- Perform the writes always in one and the same thread dedicated for that purpose (i.e. have the other threads communicate to that thread using some sort of inter-thread communication).

- Don't write to stdout but write to your own global (text) variable instead while ensuring that only one thread is accessing the variable at a time by your own means.

- Alternative of the above - override Write / WriteLn by your own routine doing one of the former two.

Tomas

Tomas Hajny wrote

I'm not sure if closing the handle using the platform specific API is actually a good idea here, because the RTL is probably not opening the file in this case

You're right, so you don't even need that special code. Just change the handle and you're fine.

Jonas