fpc 3.0.4
Win 10 / 32bit

Lazarus trunk, but that may not be relevant.
This is the first time I am seeing this, and I am therefore not sure I will be able to reproduce, since it seems a race condition.

This appears to happen during unit initialization.

Something must have started a thread, before widestring manager got initialized.
In other word, some unit (used in such order that it's initialization runs before unit system initialization) has started a thread.

This thread crashes, if it is fast enough, because in the threads init it will access widestringmanager.

Is this a bug in fpc?
Or is this wrong of the calling code (eg starting threads during unit initialization).

In the latter case, is it documented?

The trace from the thread:
#0 ?? at :0
#1 OPENSTDIO({}, 55217, 168) at ..\inc\text.inc:2670 #2 SYSINITSTDIO at ..\win\syswin.inc:520
#3 INITTHREAD(16777216) at ..\inc\thread.inc:59
#4 EXEC_TLS_CALLBACK(0x400000, 2, 0x0) at ..\win\systlsdir.inc:90 #5 ntdll!WinSqmSetDWORD64 at :0
#6 ?? at :0
#7 ntdll!RtlDeactivateActivationContextUnsafeFast at :0 #8 ntdll!RtlCaptureStackContext at :0
#9 ntdll!LdrShutdownThread at :0
#10 ntdll!LdrInitializeThunk at :0
#11 ntdll!LdrInitializeThunk at :0
#12 ntdll!LdrInitializeThunk at :0
#13 ?? at :0

The main thread sit right before widestring initialization. #0 INITUNICODESTRINGMANAGER at ..\inc\ustrings.inc:2311 #1 SYSTEM_$$_init$ at :674
#2 fpc_initializeunits at ..\inc\system.inc:886
#3 :87

There actually isnt a thread started.

But during initialization "TThread.Queue" is called (because the code in question can be called in threads aswell, and must ensure work is done in the main thread)

"TThread.Queue" creates a new external thread. This fails during unit initialization for the reason below.

This leads me to the following questionQ

1) out of interest.
If "TThread.Queue" is called in the main thread, then it does not queue anyway, but rather run immediately.
In this case, why not avoid creating a thread, and check "CurrentThreadVar = nil" ?

2) how to solve my problem. (including other OS, for which I do not know if the issue exists)
- Is there a way to test, if current code is in the main thread? (and doing so, without creating a TThread instance)?

- If not, then for windows (and fpc 3.0.4) the following *should* do (but is really rather a hack)
  {$IFDEF MsWindows}  if WideStringManager.GetStandardCodePageProc = nil then

Any better ideas?

Ok, I guess the "TThread.Queue"  is wrong.
Because that may create an instance, but there would not actually be a 2nd thread as observed. (Question 1 is still interesting)

Yet there is no other TThread.Create call made.

So (guessing) the original issue may be due to the debugger. The debugger interrupts the target early on. And that does create a thread in the target.
If such an external thread happens, would fcp execute the code in question?

Could/Should there be a nil check at the location of the crash?

I have the same problem. It is very annoying because programs crash quite often when run in a debugger (on Windows).

I solved it with the attached patch - I added these lines to procedure
OpenStdIO(var f:text;mode:longint;hdl:thandle);
   {$if defined(FPC_HAS_CPSTRING) and defined(FPC_HAS_FEATURE_WIDESTRINGS)}
     while not Assigned(WideStringManager.GetStandardCodePageProc) do
       ;
   {$endif}

The problem is that
procedure InitThread(stklen:SizeUInt);
is called before the WideStringManager properties are set in system initialization when debugger is present.

How to solve it properly? Can the InitThread() procedure somehow wait until the system initialization is done? E.g. with a global variable set with InterLockedIncrement()? Would such a patch be accepted in FPC?

I'd like to solve the problem on FPC level to avoid my own patches.

Thanks
Ondrej

This is likely to be the cause, cause the EXEC_TLS_CALLBACK is executed by Windows for every thread that is started for an application. And if the debugger triggers the start of a thread...

Could/Should there be a nil check at the location of the crash?

The RTL assumes that it does not have to deal with threads during the initialization of the System unit. So even if the symptom of the crash is fixed in that location it isn't said that something else might crash or that in the future code might be added that crashes...

Regards,
Sven

Sven Barth wrote

The RTL assumes that it does not have to deal with threads during the initialization of the System unit. So even if the symptom of the crash is fixed in that location it isn't said that something else might crash or that in the future code might be added that crashes...

Is it possible to acknowledge that the current RTLs assumption is wrong and it has to deal with threads during the System initialization? (Because obviously the assumption is wrong, otherwise there was no crash.)

That means not to fix the symptom with checking
"Assigned(WideStringManager.GetStandardCodePageProc)" but acknowledge that the OS can start a thread before System initialization is done and create a mechanism to postpone thread initialization until System is fully initialized.

Something like:

     procedure InitThread(stklen:SizeUInt);
       begin
// new code begin
         while not SystemInitialized do
           Sleep(10);
// new code end
{$ifndef FPUNONE}
         SysResetFPU;
{$endif}

Ondrej

I think the assumption is definitely correct:
some subsystems needs to be initialized.

Since there are multiple systems to initialize, there will always be moments in code where one system is initialized and the other is not All code should be prepared to deal with that.

This counts double for systems where initialization is delegated to other units: strings/threads on unix, for example

If you can prepare a simple sample program that shows the problem, I can try to look into it. Maybe a fresh pair of eyes sees more than one pair of eyes?

Michael.

Michael Van Canneyt wrote

If you can prepare a simple sample program that shows the problem, I can try to look into it. Maybe a fresh pair of eyes sees more than one pair of eyes?

You don't need a special sample program. Every console application on Windows will do. Just run it with debugging in Lazarus.

My observation is that the problem is worse on slow machines (virtual machines, shared VPS). I actually almost never observed it on my local PC, but on the VPS it made debugging nearly impossible.

I'll disable the "Assigned(WideStringManager.GetStandardCodePageProc)" hack on the VPS and report back.

Ondrej

Ah, Windows.
I have no working installation available for it.
That will take time to set up :/

The fact that it shows up when execution is slow is strange.

Michael.

Ondrej Pokorny via fpc-devel schrieb am
Mo., 19. Dez. 2022, 08:30:

The RTL needs to make some assumptions, because it is the one that sets up the synchronization primitives and under normal circumstances this assumption *is* true. As you mentioned in the other mail the crash depends upon execution speed and thus it could very well happen during initialization of the heap as well.

It will probably come to something like this.

Though we should first check what it means for the user if the debugger thread is hold in place.

Regards,
Sven

Ok, I don't know too much about the whole initialization.... But on the off chance of triggering some ideas, I throw in a couple of my thoughts....

Maybe, maybe not always? But, yes at least in the case that I documented in 2018:
https://lists.freepascal.org/pipermail/fpc-devel/2018-July/039374.html

Yet in the system unit:

initialization
.....
   if not Assigned(CurrentTM.BeginThread) then
   begin
     InitHeap;
     InitSystemThreads;
   end;
   SysInitExceptions;
   initunicodestringmanager;
   InitWin32Widestrings;

So the WS-Mgr is only assigned after InitSystemThreads => which sets up  RelocateThreadVar => which can call InitThread.

So order may need to be changed there (if that is possible). Of course, the case where EXEC_TLS_CALLBACK calls InitThread remains.

A lot of the WS-Mgr init is assigning pointers to the correct procedure. So if InitThread sees that this has not been done yet, InitThread could call those.
Even if 2 threads (main and the early thread) both assign values to the pointer, they assign the same value, and so that should not do harm. Of course the thread init must not change the value, once it was set by the main thread, in case usercode assigned a diff value (so maybe in ThreadInit, use Interlocked....)

This may mean to break "InitWin32Widestrings" into 2 parts, because it also prepares some tables.
But those are not needed for "GetStandardCodePageProc"

So if the setting of the codepage related proc pointers are moved to a separate init-method, then that can be called (if needed) by InitThread.

--------------
Or maybe if EXEC_TLS_CALLBACK calls InitThread, it can do the (partial) init of WidestringMgr before? (if the order in the initialization section can be changed to do the same?)

--------------
I am not sure if this code needs to worry about any of the other WS-Mgr functions.
Any thread that early is not started by Pascal code (neither user, nor rtl). It will be a thread working some kernel proc. And that shouldn't access the WS-Mgr?

While I just looked through that code, I came to think there may be yet another issue.

If a thread is started, that can call EXEC_TLS_CALLBACK. And EXEC_TLS_CALLBACK  can call InitHeap (in DLL_PROCESS_ATTACH) before the main thread has run InitHeap.

But InitHeap does not seem to be threadsafe.

InitHeap itself does probably work if it runs overlapping. But, if one thread already goes into using the heap (e.g. allocating) while the other still is in InitHeap... That likely does not go well.

Please stop focusing on the WideString manager. That is simply a symptom. The real cause is that a thread is created where there shouldn't be one and under normal circumstance there indeed will be none:
- if the initialization is executed as part of an application then there won't be any other thread, because the system initialization doesn't create one
- if the initialization is executed as part of a library then there won't be any other thread, because the initialization is run during the DLL_PROCESS_ATTACH call of the entry point which Windows executes with the global loader lock held, thus no thread can enter the entry point (which is also why there are some things you shouldn't do in a unit initialization)

The only cause for a thread to exist at that point is if another process calls CreateRemoteThread(Ex)() to create a thread in the process that is just initializing (Note: DebugBreakProcess() internally uses CreateRemoteThread(Ex)()). This means that the thread can interfere at *any* point of the initialization simply depending on the behavior of the Windows scheduler (as you hinted at in your other mail from today).

So the only logical solution is to stop the offending thread from executing or not to have it call InitThread() while the initialization section of the System unit is still running. The former might have unintended consequences (e.g. not being able to debug the unit initialization) while the later might work in most cases as long as that remote thread doesn't need to execute code that relies on InitThread().

Regards,
Sven

Sven, thank you very much for the insights!

I would much appreciate a solution without my hacks.

Ondrej

The latter may be the best solution. Skip any rtl call when a thread attaches during initialization. In normal cases this will only be triggered when a debugger attaches. It usually only forces a breakpoint, so no fully initialized rtl is needed.

Marc

Ondrey

When you got the issue, did you:
- Use FpDebug?
- Lazarus 2.3
- Run/Debug - But *not* press the "pause" button?

Because if so, the debugger should not have sent any DebugBreak. Unless Windows itself injects threads, if an exe is started with "poDebugProcess".

So maybe something else is injecting threads... And the reason it happens (to be observed) only in the debugger is that the timing changes.

Ondrey, if the above (FpDebug, not pressed "pause") applies, can you test the following:
Modify the system unit, and in the "initialization" section, insert some "sleep(250);" between statements (downto after InitSystemThreads) Then run without debugger.