One requirement in a transformation into OOP is the replacement of all affected procedures by methods, and of many (currently global) reference variables by class members. When e.g. later multiple parser instances are created, one for every unit to compile, the current compiler directive handling should be redesigned.

Does somebody have experience with (lists of) procedural variables?

I can see these possible solutions for the replacement of procedure references, in e.g. the list of compiler directives:

1) The currently global list is duplicated for every scanner/parser instance, so that it can contain references to the concrete method instances. Fast execution, but more data and setup.

2) The procedure references are modified when actually used, i.e. the instance reference of the method pointer is replaced by the current instance, before the method is called.

3) The procedure references are replaced by directive IDs (enum), with an according case selection in the parser instance.

4) The procedures will not become methods, but will receive an additional instance reference as a parameter. This approach may result in the best SVN history, but may not be usable with threads, later.

Any ideas?

DoDi

While I in fact don't understand the meaning of your post:

Regarding OOP, would it not be better to do lists of objects with class procedures instead of lists of procedural variables. So these classes would be children of a parent class and you can do a clean list (TList derivate, array or whatever) of same.

-Michael

I also have problems to understand your answer :-(

The directive list contains records with the name and other attributes of every directive, and with a reference to the directive handler procedure.

These handler procedures need further information, that currently resides in global variables. In an OO approach these variables become class members, so that an additional reference to a concrete object is required in the directive handlers.

In so far "class procedure" doesn't help, if you meant that, because Self in a class procedure is the class type, not a class instance.

It's also useless to create new classes, because the handlers need access to the elements in a concrete parser/scanner class (instance).

DoDi

Why don't you separate refactoring in several steps:

1) First, you replace all global variables with class members, and as much as you can (global) procedures with class methods. You also add one global Parser (or whatever you call it) variable, and replace those "problematic" procedures with wrappers. Something like:

function f(...)
begin
  Result := Parser._f(...)
end;

This will result is nicer commit. You can even put wrappers just in front of wrapped methods (in the above example, function TParser._f would go just after function f). You'll probably need to add underscore or some other character at the beginning of wrapped names so you can still reference f from TParser methods.

2) After this, you should have parser which is still fully functional, but uses only one global variable - Parser - in wrapping functions. At that moment, you can analyze further refactoring (promoting Parser to parameter of wrapping functions, or some other solution), and even test them without having to deal with huge commits.

Just an idea...

Zitat von Hans-Peter Diettrich :

About converting global variables for threads:
What are the advantages/disadvantages of using ThreadVar versus member variables?

Did someone test the speed differences?

Mattias

Hans-Peter Diettrich wrote

One requirement in a transformation into OOP is the replacement of all affected procedures by methods, and of many (currently global) reference variables by class members. When e.g. later multiple parser instances are created, one for every unit to compile, the current compiler directive handling should be redesigned. Does somebody have experience with (lists of) procedural variables?

See ncnv.pas, ttypeconvnode.first_call_helper. In fact, for the directives it's even simpler because those methods don't have to be overridable (so you can call them directly instead of having to use wrappers).

Jonas

I don't see what you are trying to compare here.

Threadvars are addressed by a dedicated register (I suppose a segment register with on X86) that is different for each thread ant points to a memory area that is dedicated to each thread.

Member variables are addressed using the self pointer of the object and thus they are located on the heap and different for each instance of the Object but common for all threads.

I suppose the speed difference is very small but the function is very different.

If you want an instances of a class for each thread you need to have each thread create it's own instance (on the heap) and and hold the self-Pointer (the "instance"-variable ) in a threadvar.

-Michael

Zitat von Michael Schnell :

If I understood the mail thread correct, the goal was to change some global variables, so that each thread has its own. This can be achieved with Threadvar or one object per instance. I don't know how ThreadVar is implemented by the compiler on the various systems.

Threadvars are addressed by a dedicated register (I suppose a segment register with on X86) that is different for each thread ant points to a memory area that is dedicated to each thread.

Can some compiler guru confirm this?
After a quick glance on the rtl sources it seems ThreadVar works under some platforms internally pretty much the same as an object instance: it allocates some memory on the heap.

Mattias

Mattias Gärtner wrote

After a quick glance on the rtl sources it seems ThreadVar works under some platforms internally pretty much the same as an object instance: it allocates some memory on the heap.

That's correct. FPC has largely platform-independent support for threadvars, which is much more heavy-weight than simply using a segment register (indirect procedure call depending on the thread manager, get thread local storage pointer, index local storage with offset specific to the threadvar).

There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version.

Jonas

Jonas Maebe wrote

That's correct. FPC has largely platform-independent support for threadvars, which is much more heavy-weight than simply using a segment register (indirect procedure call depending on the thread manager, get thread local storage pointer, index local storage with offset specific to the threadvar).

Nonetheless in the end a register is needed to be used that is preloaded with different values for different threads. Otherwise a thread can't know which one it is. Registers is the only per-Thread information that is granted to persist by the OS during preemption.

There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version.

YAK

I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption.

-Michael

Op Mon, 19 Jul 2010, schreef Michael Schnell:

You could also work with the current thread id as a key to a pointer to the threadvars. However, segment registers are indeed the way libpthread resolves the TLS keys.

Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers.

Daniël

There have been attempts to get the segment register approach to work on Windows, but afaik it always crashed on at least one Windows version.

The segment register approach is not officially documented and requires anyways an additional code so the gain isn't that big.

Hi!

Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers.

As mentioned here http://en.wikipedia.org/wiki/X86_64#Windows in the 10th point Windows uses the GS segment register to store pointers to the TEB (see http://en.wikipedia.org/wiki/Thread_Environment_Block ) of the current running thread (x86 uses FS). So they should still be writable as the TEB is written to by the Win32 subsystem (of which a part runs in usermode) as part of its bookkeeping and maybe also by user applications to implement the 64 bit equivalent of SEH.

Regards,
Sven

Op Mon, 19 Jul 2010, schreef Sven Barth:

Well, the following works on i386 but doesn't on x86_64, even though modify_ldt succeeds:

program ldt_test;
{$asmmode intel}
uses baseunix,linux;
var p:pointer;
     ud:user_desc;
begin
   p:=pointer(fpmmap(nil,4096,3,MAP_PRIVATE+MAP_ANONYMOUS,-1,0));
   ud.entry_number:=1;
   pointer(ud.base_addr):=p;
   ud.limit:=4096;
   ud.flags:=UD_SEG_32BIT or UD_CONTENTS_DATA or UD_USEABLE;
   modify_ldt(1,@ud,sizeof(ud));
   asm
     mov ax,15
     mov fs,ax
     mov fs:[0].dword,29
   end;
   writeln(Pcardinal(p)^);
end.

If you know how to make it work, I keep myself recommended :)

Daniël

Hi!

Well... according to this thread
http://stackoverflow.com/questions/1180796/how-to-use-gs-in-64-bit-windows-assembly-eg-porting-tls-code it's only possible in kernel mode using a special instruction. The kernel might set up the necessary data and the usermode part of the Win32 subsystem fills in the data later.

Regards,
Sven

See ncnv.pas, ttypeconvnode.first_call_helper. In fact, for the directives it's even simpler because those methods don't have to be overridable (so you can call them directly instead of having to use wrappers).

Thanks, this was my solution #2 :-)

DoDi

I do know that gcc in Linux on X86 uses a segment register. I don't know how gcc works in Windows, but I suppose even Windows is assumed to restore all register values of a thread after a preemption.

Windows uses segment registers in exception handling, i.e. it should be possible. But the segment descriptor/selector must point to the correct thread-memory address in the address space, what can not be achieved in application code.

DoDi

Op Mon, 19 Jul 2010, schreef Hans-Peter Diettrich:

Okay, so the Windows kernel sets up the segment registers then by default, that is interresting to know. But Linux does not, yet libpthread is using the segment registers. What I find curious is that you can call modify_ldt sucessfully to setup a segment, but then you cannot use it.

Daniël

Daniël Mantione wrote

You could also work with the current thread id as a key to a pointer to the threadvars.

Theoretically you can, but in praxis this is not doable for each access to a threadvar, as you would need to request the thread ID from the OS with each access to the threadvar which would be horribly slow. You can't store the thread ID in a global variable nor on the heap (as those obviously are not a threadvars) so you can store it on the stack or in register, which both are dedicated to each thread. As the stackpointer value is not known at runtime within any function, using the stack is not possible.So AFAI can see, without alternate OS-support, dedicating a register as a pointer to the threadvars, is the only viable option a compiler has.

AFAIK, in Linux the archs specs define which register is supposed to be used as a theradvar pointer by compilers. There might be some archs that don't define a register but a dedicated memory location, managed by the OS when scheduling threads and processes, to be the threadvar pointer.

However, segment registers are indeed the way libpthread resolves the TLS keys.

Obviously.

Yes, though I am still puzzled how it works on x86_64; it seems regvars there are also accessed using fs, but x86_64 prevents you from writing to segment registers.

In fact it is not critical that the threadvar register (with X86/32 Linux) is a segment register, With NIOS it's just one of it's 32 general purpose registers. (I'm not very deep into ARM yet, but I'm quite sure that its similar here). So it might be the same with X86/64. The compiler just needs to make sure that this register is not modified ever and that it's used as a threadvar pointer.

-Michael

Where can be a detailed specification of this found? How does it handle dyn. loaded libs using thread vars?

If Windows sets the segment registers and the application is not supposed to modify them (and assume them to be restored after preemption), maybe one of them is a pointer to some thread-dedicated memory area, that by can be used (or is used by the OS) to hold a pointer to the threadvars or to hold the threadvars themselves (if the size of the threadvar area is denoted in the exe file and thus known to the OS, which might kindly allocate it when a thread is created).

-Michael

I found these:
http://stackoverflow.com/questions/1460483/on-which-platforms-is-thread-local-storage-limited-and-how-much-is-available

http://www.linux-mips.org/wiki/NPTL

How does it handle dyn. loaded libs using thread vars?

The just compiler addresses the threadvars relative to a register instead of the pointer to the global area with normal globals. (Or with X86/32 using another segment register instead of the default DS). So regarding the code it's just another Global area, in realtime the base of same is switching automatically when the thread is scheduled.

-Michael

Hi!

According to
http://en.wikipedia.org/wiki/Thread-local_storage#Windows_implementation you use a dedicated API to modify the TLS. And according to http://en.wikipedia.org/wiki/Thread_Environment_Block the address of the TLS is indeed accessible/saved in the segment registers.

Regards,
Sven

Florian Klaempfl wrote

The segment register approach is not officially documented

Are we discussing Win32 or Win 64 here ?

So, what is officially documented regarding restoring registers after preemption.

Obviously all "normal" registers and all FPU registers are restored.

CS, DS and SS obviously need to be handled appropriately and are not available.

Also ES needs to be restored as same is dedicated to be used for local stuff

What more do we have ? FS, GS.

Here some documentation would be necessary.

-Michael

On 07/19/2010 05:22 PM, Florian Klaempfl wrote: > The segment register approach is not officially documented Are we discussing Win32 or Win 64 here ?

At least Win32: accessing the tlb via fs does not work for all windows with one approach.

OK. Great. A lot more complicated than with Linux, but seemingly decently documented.

What about Win 64 ?

-Michael

Florian Klaempfl wrote

At least Win32: accessing the tlb via fs does not work for all windows with one approach.

According to Stevens Post using the "Thread Information block" which is addressed by FS is documented and thus needs to be working for 256 TLS-Slots (whatever that means) in NT (and supposedly higher). Who cares about Win9x :) .

No idea about Win64.

BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area.

-Michael

Where is it *officially" (e.g. MSDN) documented?

and thus needs to be working for 256 TLS-Slots

No.

Florian Klaempfl wrote

At least Win32: accessing the tlb via fs does not work for all windows with one approach.

The documentation Sven mentions says:

FS:[0x2C] 4 Win9x and NT Linear address of the thread-local storage array

Do you suppose Vista and Win 7 will not apply ?

-Michael

It is probably reverse engineered guesswork but no official documentation.

Michael Schnell wrote

BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area.

Only if you are guaranteed that no externally called code will modify it. In that case, you FPC itself also save/restore that register before and after calling any potentially non-FPC code, since other applications may make the same assumption. If you don't make that assumption, at the very least you still have to reload it after any call to potentially non-FPC code.

Jonas

Jonas Maebe wrote

Only if you are guaranteed that no externally called code will modify it.

This should be documented in the ABI description and of course could be different for the different calling conventions (Register, STDCALL, ...). If the ABI defines that GS can be modified by the callee FP should of course save it on the stack before calling and restore it afterwards. Seems like "Standard" to me.

-Michael

Florian Klaempfl wrote

It is probably reverse engineered guesswork but no official documentation.

Hmm. I think M$ wants us to do decent Applications and this includes multithreaded applications that use threadvars ("__thread" in C).

Thus I can't believe they did not document how a compiler is supposed to handle threadvars.

Also, I suppose there is a documentation for M$'s own C compiler and for GNU C how they do it on the appropriate arch, as same should be able to call ASM functions that define their own threadvars in a way compatible to the main program written in C. Maybe this is even true for Delphi.

-Michael

Michael Schnell wrote

On 07/20/2010 02:08 PM, Jonas Maebe wrote: > Only if you are guaranteed that no externally called code will modify it. This should be documented in the ABI description

I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other).

Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. Sending us off on wild goose chases does not help.

Jonas

Jonas Maebe wrote

I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other).

At least the "Register" ABI is FPC specific and needs to be documented with the compiler. I hope the others are available near that place.

Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents.

I tried to find it in the FPC docs, but was not successful.

-Michael

Michael Schnell wrote

On 07/20/2010 02:48 PM, Jonas Maebe wrote: > > I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). At least the "Register" ABI is FPC specific and needs to be documented with the compiler. I hope the others are available near that place.

The "register" calling convention is Borland-specific and does not specify anything regarding the segment registers: http://docwiki.embarcadero.com/RADStudio/en/Program_Control

> Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. I tried to find it in the FPC docs, but was not successful.

I know of no single calling convention that says anything about the segment registers. It's an OS ABI issue at best, and at worst undefined behaviour.

Jonas

BTW what about GS ? If same is just restored after preemption, this could be used by the compiler to point to the Threadvar area.

Since it turned out that loading segment registers is very time consuming on newer (i486+) processors, the memory management was changed from segmented (16 bit) into paged (32 bit flat). Segment registers never should be changed by an application, it should be left to the OS process/thread scheduler and thread-API calls.

DoDi

Hi!

The Linux and Windows APIs are more or less similar. The fact that the TLS address is stored inside a segment register shouldn't bother us / an app developer anyhow, because that is defined as "implementation detail" (the TEB is only documented by Wine devs not by Microsoft themselves (only the subsystem independant part of the TEB is)). That said Win 64 uses the same API as Win 32 and the segment register used for the TEB is GS instead of FS.

Regards,
Sven

Hi!

For this purpose the TLS-API is to be used. The segment registers that point to the TLS array are undocumented and part of the internal bookkeeping of a NT system (and yes, Windows Vista and 7 are NT systems as well).
But because many applications rely on this undocumented TEB Microsoft doesn't dare to change anything regarding this. But please no fight about this topic; I've read the blog and the book of a Microsoft developer who reports to what lengths Microsoft went to ensure application compatibility (it's called "The old new thing" and I'm suggesting it to everyone who wants to know something about Windows internals in a humorous way).

Btw: if you are really curious you might take a look at the TLS implementation of ReactOS, which is an NT OS after all: http://svn.reactos.org/svn/reactos/trunk/reactos/dll/win32/kernel32/thread/tls.c?revision=45018&view=markup

Regards,
Sven

Jonas Maebe wrote

The "register" calling convention is Borland-specific and does not specify anything regarding the segment registers: http://docwiki.embarcadero.com/RADStudio/en/Program_Control

Sorry. I did not expect that the ABI documentation is still that bad after all these years. It in fact does not even say what to do with the stack pointer (which is handled differently with STDCALL and CDECL and thus it's not really obvious that it needs to be preserved by the callee).

I know of no single calling convention that says anything about the segment registers. It's an OS ABI issue at best, and at worst undefined behaviour.

IMHO the ABI documentation should define how all registers are handled :(

I think sometimes it might be useful for an ASM programmer to use the segment registers, and so he should be informed whether he needs to save and restore them.

-Michael

Not really, The OS needs to set the segment registers, anyway, when returning to a preempted thread or process, otherwise it needs to be clearly documented that the user program is not supposed to use them at all. Assigning some value - e.g.the pointer to the threadvar area - to e.g. GS would not harm performance-wise.

it should be left to the OS process/thread scheduler and thread-API calls.

If a register is not predefined by the OS (such as GS on WIN32), I think the ABI should inform the ASM programmer about how to use it. Simply stating "all other registers need to be preserved by the callee" would suffice and answer the original question about if the compiler would need to safe GS in case an "external" function might trash it.

Moreover of course the ABI should clearly state how an ASM function is supposed to create threadvars, anyway.

-Michael

Sven Barth wrote

For this purpose the TLS-API is to be used.

I understand that the TLS-API is too slow to be really useful for doing global threadvars. Correct ?

The segment registers that point to the TLS array are undocumented and part of the internal bookkeeping of a NT system (and yes, Windows Vista and 7 are NT systems as well). But because many applications rely on this undocumented TEB...

I suppose they do so because of what is asked above :)

Microsoft doesn't dare to change anything regarding this.

If their C compiler uses this, many programs done with same would fail if they would change the Segment register based API. So I do bet they will never change it.

And thus I think it's viable for FP to do what the M$ C compiler does.

-Michael

Of course the *entire* CPU state is kept in an stored machine state record. My concern was about the ban of using more than one segment selector (DS=CS=ES=SS), related to one set of segment descriptor tables. The thread specific FS/GS selector most probably selects a tiny segment, with small related tables, that do not affect performance a lot. And since the segment registers are only changed on process/thread switches, the resulting overhead is kept to the absolute minimum.

otherwise it needs to be clearly documented that the user program is not supposed to use them at all.

Apart from that, I've heard that segment registers have no special effect in 64 bit mode at all? How can these registers ever be used, in 64 bit code? Safe bet: don't touch them ever, use only as documented (for FS/GS).

Assigning some value - e.g.the pointer to the threadvar area - to e.g. GS would not harm performance-wise.

When an OS does not provide an API to create segments (in a 32 bit flat model), there exist no values (segment selectors) that can be loaded into a segment register at all, by application code.

Otherwise it *would* harm performance-wise, to load another segment selector into a segment register, due to the associated overhead in the MMU (address translation logic, from virtual into physical addresses).

In a non-segmented environment (64 bit mode?) the segment registers have no special meaning, and consequently no such related overhead.

Of course this would be nice, but till then it's simply wise to not use such undocumented registers at all, neither in compiler created code, nor in ASM.

Moreover of course the ABI should clearly state how an ASM function is supposed to create threadvars, anyway.

IMO the OS provides means to allocate an thread-local memory block (Thread Local Storage), whose content *can* be used for threadvars. It's not up to the API or ABI to bother with the "record" layout of that TLS memory block, that's all application/compiler specific. A threadvar simply is a member of that record, and the API specified register contains the (direct or indirect) address of the TLS.

IMO also no physical/logical separation exists, between application (main thread) and other thread memory, everything exists in the overall process memory. It's up to the compiler/coder, to prevent unsynchronized access to memory locations that could be updated concurrently by multiple threads.

BTW, the definition of "threadvar" is not a very practical one. In practice it should be possible to assign threadvars to *specific* threads, otherwise *every* thread has to allocate an TLS containing *all* the threadvars of the entire application. Furthermore it would be sufficient to have all "threadvars" in the specific TThread-derived classes, eliminating the need for any additional TLS.

DoDi

Hans-Peter Diettrich wrote

. And since the segment registers are only changed on process/thread switches, the resulting overhead is kept to the absolute minimum.

Same with using a segment registers the base for the threadvar area. This only needs to be initiated when a thread is created. This is a quite rare occasion, as you will use a thread-pool if a threaded application needs to switch between multiple active/passive threads, anyway.

I did not do any research yet regarding X86/64 :(.

When an OS does not provide an API to create segments (in a 32 bit flat model), there exist no values (segment selectors) that can be loaded into a segment register at all, by application code.

If the OS does not provide an API that defines the use of the registers and the OS manufacturer does not provide a documentation that tells us otherwise, I think we are allowed to presume that these registers are "just registers" that work as documented in the way the hardware documentation describes and thus a user application is free to use them anywhere and the OS guaranteed that they are save regarding preemption. AFAIK, this is commonly done with ES, and so I don't see why it's not true with GS. It seems that on thread start in Windows, FS is predefined with a pointer to some Windows specific thread information, but that does not mean that it's not allowed to assign some other value to FS and that it will not be safe.

Of course this would be nice, but till then it's simply wise to not use such undocumented registers at all, neither in compiler created code, nor in ASM.

Hmm, is BX explicitly documented ?

IMO the OS provides means to allocate an thread-local memory block (Thread Local Storage), whose content *can* be used for threadvars.

AFAI understand, in Linux (any architecture) the OS just preserves a count of registers and the (e.g. C) compiler just uses one of them as a pointer to the threadvars. When creating a thread the thread library (e.g. libc) creates the threadvar block and assigns the pointer to the said register.

It's not up to the API or ABI to bother with the "record" layout of that TLS memory block, that's all application/compiler specific.

The way how threadvars are don _is_ compiler specific. Of course it does make sense if all compilers working for a certain OS use the same paradigm here, so the OS _should_ provide a documentation how they should do it,. In Linux of course the gnu C compiler team takes care of all this.

A threadvar simply is a member of that record, and the API specified register contains the (direct or indirect) address of the TLS.

I suppose this is the Windows specific way....

Of course this is correct. But the compiler only has two options to do this. Either do an OS API call foe each access to a global threadvar (very slow) or dedicate a register a a pointer to the block of global threadvars. I know that e.g. NIOS (that is similar to ARM or MIPS) just uses one of it's 32 general purpose registers for this. X86/32 does not have many registers, so it's a good idea to use a segment register. This is what Linux GNUC is officially documented to do and what the "unsafe" documentation mentioned in this thread states for Windows as well (here in fact maybe as a pointer to a pointer to the threadvar block).

The programmer is free to do this by using just another indirection. You may use a single global threadvar that holds the address of an Object (or record) that a thread allocates on the heap when it starts. The Object or Record now can hold any thread specific information.

Many global threadvars only make sense if you start the same code in many threads.

-Michael

Do you mean segment or general-purpose register(s)? How is that different from WinAPI?

When we consider threads created in a DLL/.so, too, then the platform has to establish such conventions, for the entire app<->library ABI.

> A threadvar simply is a member of that record, and the API specified > register contains the (direct or indirect) address of the TLS. I suppose this is the Windows specific way....

How else should it be done, in a practical way? Registers are not a solution, since:

Assume you have 10 units, each of which declares only one threadvar - then you would have to reserve 10 registers for these threadvars, application-wide!

When we agree that threadvars (etc.) are part of the application/process address space, then they can be accessed like any other variables - by their unique address. In a flat memory model a strict separation would require to shrink the application address space, by the amount required for thread address space. IMO not very likely...

IMO this duplicates the OS overhead (for the thread memory (TLS) implementation), in the application. A single threadvar then would be sufficient for *all* threads, i.e. the one (register) provided for thread-storage by the platform.

Many global threadvars only make sense if you start the same code in many threads.

Can you give a more concrete example?

I'm not familiar with actual thread programming, so I have no idea about the *usage* of threadvars in concrete code.

DoDi

Hans-Peter Diettrich wrote

Do you mean segment or general-purpose register(s)?

This depends on the arch. X86 does not really have general-purpose registers, while NIOS, ARM MIPS (and supposedly X86/64) has almost only general-purpose registers.

How is that different from WinAPI?

The whole discussion here is that X86/32-WinAPI in fact uses a segment register for this purpose but it's nowhere decently documented. Moreover (bus not a problem) I understand that gnu C on X86/32 Linux uses a segment register as a pointer to the threadvar area, while the non-documented WinAPI uses a segment register as a pointer to a thread-dedicated structure that again contains a pointer to the threadvar area.

When we consider threads created in a DLL/.so, too, then the platform has to establish such conventions, for the entire app<->library ABI.

Not really. The main executable can create a thread and a function called in the thread can internally declare and use a static threadvar. So the ABI needs to document how the (ASM) function is supposed to access the threadvar it internally defines. As it needs to use a register for this, the ABI needs to define that the alee is to use exactly this register and (supposedly) that the callee is not supposed to modify it when it returns.

Sorry but here you are decently wrong.

With most archs all static variables (such as globals and threadvars) always are accessed relative to a register. With X86/32 globals and statics are allocated relative to DS, with X86/32-Linux, threadvars are allocated relative to FS. The archs i know, that don't have segment registers use "address"- or "general purpose" registers for this. So we have two "basic data pointers": one for the normal static vars and one for the threadvars. Both are safe regarding preemption. The basic pointer for normal static vars is initialized at the program start and never changed within the application, the pointer to the threadvars is initialized when a thread starts and - because it is preemption save just like any other register - its never changed when this thread is running.

Moreover the point with threadvars is that threads that use the same code see different values in the same threadvar. These threads of course can't use different registers to access them as the run the same code.

See "the point with threadvars...".

Can you give a more concrete example?

See "the point with threadvars...".

I'm not familiar with actual thread programming, so I have no idea about the *usage* of threadvars in concrete code.

Using threadvars only make sens for multiple threads running the same code.

-Michael

> Do you mean segment or general-purpose register(s)? This depends on the arch. X86 does not really have general-purpose registers, while NIOS, ARM MIPS (and supposedly X86/64) has almost only general-purpose registers.

M68K has distinct address and data registers, x86 has segment and data registers, not to mention other special registers (flags...) in every architecture.

I see no difference here. IMO the segment register is used implicitly in thread API calls, with no further use by application code. Every thread has one "slot" reserved in the OS-maintained data structure, where it can store one private value. In the "Using Thread Local Storage" entry in the MSDN library this value is the handle of a memory block, created with LocalAlloc. It could be the handle of an local heap as well, created with HeapCreate, when special handling of threadvars shall be implemented in the RTL. No special segment register seems to be involved in the subsequent use of that allocated memory (flat memory model).

> When we consider threads created in a DLL/.so, too, then the platform > has to establish such conventions, for the entire app<->library ABI. Not really.

I found the Windows conventions in "Register Usage" in the MSDN Library, telling which registers are considered (non-)volatile.

Application code can use any non-volatile register for (thread...) communication purposes. When an application and a shared library shall work together, they must use the same convention about the use of such registers.

Here the segment register is somewhat irrelevant, since each maps to the entire 4 GB process address space. All 32 bit "pointers" simply are offsets into that address space, like in any other unsegmented architecture. The only difference may be access rights (CS: executable, read-only).

with X86/32-Linux, threadvars are allocated relative to FS.

FS and GS may be used for special (OS, SEH and TLS) management, they are useless in application code.

On Windows such base-pointers should reside in non-volatile registers, otherwise no difference in their use. As long as the base values are immutable, the equivalent absolute addresses can be used instead (in contrast to PIC).

Moreover the point with threadvars is that threads that use the same code see different values in the same threadvar. These threads of course can't use different registers to access them as the run the same code.

Consider a threadvar as a field of a related thread object (instance).

DoDi

24.07.2010 6:55, Hans-Peter Diettrich:

IMO the segment register is used implicitly in thread API calls, with no further use by application code.

Exactly the opposite (at least delphi on windows). See delphi's RTL.

fs and gs whould indeed hardly ever be touched by application programmer directly (by hand), however (as it was already metioned several times earlier) delphi (on windows) and gcc (supposedly both windows and linux) use them to implement this "special management" so as exceptions and threadvars can be actually used without explicitely using OS APIs.

In such case of course fs/gs will not map to the same flat address segment (as cs, ds, es, ss) common for the whole process and all of its threads. Instead, it will map so a special segment created by OS for a specific thread. This segment might also be accessible as some other offset in the flat address space, or might not, but it isn't the point. Point is that a thread's code basically just needs to do a segment override (add 1-byte opcode modifier on x86 AFAIK) each time it wishes to get into its instance-unique data block, and this is supposed to be very inexpensive compared to OS syscalls. Also note, that fs/gs are not much usefull for anything else in an application (can not do any arithmetics, can not index flat offsets directly, etc etc), so they would otherwise be most probably totally unused (wasted) which is kind of pity.

As long as the base values are immutable, the equivalent absolute addresses can be used instead (in contrast to PIC).

Sure. But, as discussed before, all threads share the same code (i.e. the offset can not be hardcoded). Therefore, you'll need some register to access the instance-specific value (unless you are going to make OS calls every time). Probably on any arcitecture some general-purpose register capable of holding address offsets can be used for that. Such register would then have to be mostly excluded from normal use in the application (in other words - mostly wasted). Now, on x86/x86_64 there is yet additional tricky option: imploy a segment register, which would otherwise be unused. This way no general-purpose register is wasted and no OS syscalls are necessary either.

Consider a threadvar as a field of a related thread object (instance).

This might be fine most of the time (for end user in particular), however there still might be some non-OO code which it is impossible/impractical to convert into object-oriented model. See FPC RTL.

HTH.
Nikolai

Nikolai Zhubr wrote

use them to implement this "special management" so as exceptions and threadvars can be actually used without explicitely using OS APIs.

Delphi does not use a segment register for threadvar handling but OS calls.

24.07.2010 17:16, Florian Klämpfl:

For a library - yes, you are right (My memory doesn't serve me good) However, for an ordinary application - no:

var
   a: longint;
threadvar
   b: longint;
begin
[...]
   a := 1;
         mov    ebx,00000001
   b := a;
         call   @GetTls
         mov    [eax+00000004],ebx
[...]
end.

where GetTls in case of an ordinary application (not dll) is just this:

MOV EAX,TlsIndex
MOV EDX,FS:tlsArray
MOV EAX,[EDX+EAX*4]

_______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

> IMO the segment register is used implicitly in thread API calls, with > no further use by application code. Exactly the opposite (at least delphi on windows). See delphi's RTL.

Well, once it's set by the OS, it can be used in hackery ;-)

In such case of course fs/gs will not map to the same flat address segment (as cs, ds, es, ss) common for the whole process and all of its threads.

I doubt that this address range really is excluded from the 4GB app address space, accessible through the other segment registers.

Where exactly should I look?

In the meantime I found one possible use for threadvars: when some subroutine can be called from different threads, it may want to retrieve the thread context, e.g. the thread object itself. Right?

DoDi

I wonder what TlsIndex here is?

Also FS:tlsArray seems to contain an address in the general address space, so that only the tlsArray seems to vary across threads? I had thought that TlsIndex also should vary accross threads.

DoDi

24.07.2010 19:12, Hans-Peter Diettrich пишет: [...]

I doubt that this address range really is excluded from the 4GB app address space, accessible through the other segment registers.

IMHO the question is not how to avoid using FS trick altogether (which is of course possible, right), but on the contrary, how to reliably employ it in order to avoid wasting some more usefull registers and excessive OS calls.

Nikolai

24.07.2010 19:46, Hans-Peter Diettrich:

tlsArray = $2C;
IIRC for an application (not a dll) TlsIndex is always 0 (But I might be wrong here though)

Also FS:tlsArray seems to contain an address in the general address space, so that only the tlsArray seems to vary across threads? I had thought that TlsIndex also should vary accross threads.

I think only FS selector (and/or descriptor) varies across threads.

The main thread may have an index of zero, but every other thread can have an different index. This can be tested or debugged only with real threads.

DoDi

Delphi (Win32) reads the word at location <FS> +4 and addresses the threadvars relative to same. I intend to take a look at FPC Win32 and Linux (32 Bit), gnuc and ;Microsoft C if I find the time to do so,

-Michael

X86/32 always uses a segment register to access memory, If none is given explicitly DS or SS is used appropriately.

-Michael

It is very clear what a threadvar is: the same variable automatically is instantiated dedicated for each thread while a normal (static or global) variable is instantiated once for all threads. Multiple threads that each use different code don't need threadvars as they can be programmed to use individual static/global variables. Inly if the same code is used for multiple threads, threadvars are necessary to allow these threads to identify themselves (e.g a threadvar can store an ID number for the thread so that it can know who it is without doing an OS API call.

-Michael

Florian Klämpfl wrote

Delphi does not use a segment register for threadvar handling but OS calls.

sorry but this is not true

Turbo Delphi:

code:

var test integer;
threadvar threadtest: integer;
   test := 1;
   threadtest := 2;

compiles to

mov [$...], $1

call @GetTls
mov [eax+$34], $2

with

@GetTLS resulting in (this is not exactly the original code, which in fact reads a zero form some location to EAX and uses it)

mov edx, fs:[4]
mov eax, [edx]
ret

-Michael

Hans-Peter Diettrich wrote

I wonder what TlsIndex here is?

In a simple test I did, it's 0.

BTW: FS is always the same value ($3B), Obviously Windows changes the table entry value, the selector $3B points to.

-Michael

Nikolai Zhubr wrote

I think only FS selector (and/or descriptor) varies across threads.

Seemingly not the selector value (the FS content seems to stay constant among the threads), but the table entry it selects.

-Michael

Hans-Peter Diettrich wrote

The main thread may have an index of zero, but every other thread can have an different index. This can be tested or debugged only with real threads.

Done (with Turbo Delphi): see above.

Trying to do tests with FP, gnuC and M$ C.

-Michael

Michael Schnell wrote

mov edx, fs:[4]

Typo: $2C, instead of $4

-Michael

Hans-Peter Diettrich wrote

In the meantime I found one possible use for threadvars: when some subroutine can be called from different threads, it may want to retrieve the thread context, e.g. the thread object itself. Right?

Yep. But of course you can do multiple threads that completely execute the same code (e.g. a Web Server that handles multiple externals requests at the same time.

-Michael

Here a simple test program with threadvars;
unit Unit61;
interface

uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,

   Dialogs, StdCtrls;

type
   TForm61 = class(TForm)
     Button1: TButton;
     Button2: TButton;
     procedure FormCreate(Sender: TObject);
     procedure Button1Click(Sender: TObject);
   private
     { Private declarations }
   public
     { Public declarations }
   end;
   TMyThread = class(TThread)
     procedure execute; override;
   end;
var
   Form61: TForm61;
   test: Integer;
   teststr: String;
   MyThread1, MyThread2 : TMythread;
threadvar threadtest: Integer;
implementation
{$R *.dfm}
function dotest: String;
begin
   inc(test);
   inc(threadtest);
   Result := IntToStr(test) + '/' + IntToStr(threadtest);
end;
procedure TForm61.Button1Click(Sender: TObject);
var s: String;
begin
   s := dotest;
   teststr := teststr + ' ' + s;
   caption := teststr;
end;
procedure TForm61.FormCreate(Sender: TObject);
begin
   test := 10;
   threadtest := 20;
   MyThread1 := TMythread.Create(false);
   MyThread2 := TMythread.Create(false);
end;

{ TMyThread }

procedure TMyThread.execute;
var s: String;
begin
   s:= dotest;
   teststr := teststr + ' .. ' + s;
   while (true) do
     sleep(10000);
end;
end.

Hi!

Michael Schnell wrote

On 07/26/2010 10:28 AM, Michael Schnell wrote: > > mov edx, fs:[4] Typo: $2C, instead of $4

I already wondered what "crap" Delphi is doing here :P

Btw: would you mind to start a new thread about threadvar implementation, because this discussion currently get's a bit lost in this big "OO rewrite" thread.

Regards,
Sven

It's only possible to dedicate another register to that purpose.

With X86/32 it does not seem to make much sense, as this arch has so few registers and dedicating one would slow down all programs - and the Win32 FS-"trick" documented in the unofficial aper mentioned above and used in Delphi (and supposedly gnuC and M$ C) is believed to work fine (AFAIK, in Linux GS is different with different threads and directly points to the TLS, but I'm not sure, yet).

With X86/64 and other archs (such as ARM) that provide many general purpose registers, I suppose it's a usual way to dedicate one of them as a Thread Local Storage area pointer.

Maybe Win64 uses some "Selector" mechanism (similar as ) so that there are no different Selector-Register values in different threads.

-Michael

Hi!

Michael Schnell wrote

Maybe Win64 uses some "Selector" mechanism (similar as ) so that there are no different Selector-Register values in different threads.

Win64 uses GS instead of FS for the TEB and everything else stays the same. In fact FS and GS (and to some extend CS) are the only selectors actively used by the x86_64 platform, but - if I understand that correctly - they can only be set up from ring 0 code (thus: kernel code), which normally happens during context switches so that the data of the current thread's TEB is available again.

The usage of the FS and GS selectors in 64bit is also described in AMD's manual: e.g http://support.amd.com/us/Processor_TechDocs/24592.pdf 2.1.2 Segment Registers

Regards,
Sven

OK. Supposedly, Win64 - like Win32 - use the same GS value for all threads and different tables this selector points to. But this is not relevant to the compiler or ASM programmer as it yields the same effect. Supposedly, the compiler does not ever need to write to FS/GS not the the table they select from, so no problem here.

But the question remains why - as stated above by the FPC team - trying to use GS in this way, did not seem to work reliably on Win64.

-Michael

As recommended starting a new message thread

Reverse engineering:

threadvar ithread: integer;
ithread := 1;

or in "C":

__thread ithread int:
ithread = 1;

Delphi, Win32 on X86/32 (stripping off additional offsets that seemingly are always 0 in normal applications):

mov edx, FS: $2c
mov eax, (edx)
mov ithread.offset (eax), 1

gnuC Linux on X86/32

mov eax, GS:0
mov ithread.offset (eax), 1

more to come....
I don't have Win 64 or Linux X86/64. Maybe someone can help...

-Michael

M$ C on Win32 (X86/32), using Visual Studio.

declspec(thread) int ithread;
   ithread = 1;

results in

mov eax, dword ptr [__tls_index (417328h) ]
mov ecx, dword ptr fs:[2Ch]
mov edx, dword ptr [ecx,+eax*4]
mov dword pth[edx+104h],1

This is essentially the same code as with Delphi. Supposedly as with Delphi [__tls_index (417328h) ] is always 0 with normal applications. (no idea what this is useful for)

-Michael

The TLS index is allocated by a call to TlsAlloc at application startup. It will be zero for the first call and increasing for subsequent calls. The subsequent calls are typically done by libraries that use threadvars. This approach ensures that threadvar blocks of each module in process do not clash.

Regards,
Sergei

I see. So in a DLL that implements it's own thread system, the TLS index will be non-zero. (The purpose being, that the Thread variables of each library can have it's own address space, which is necessary for non-statical linking.)

In Linux .so's this seems to be done in some other ways.

-Michael

FPC uses the same code with Win32 and with Linux X86/32. It does an indirect call to some external function. So this supposedly is less efficient than Delphi, GnuC or M$ C.

-Michael

26.07.2010 13:04, Michael Schnell:

You are right. Just verified here too. Seems a bit illogical, especially as 3B is a GDT (not LDT) selector, therefore it's not subject to automatic substitution upon task switch. The scheduler patching GDT entry on every task switch is IMHO too crazy. It might be that FS's (descriptor) base points to some non-shared region of linear address space, so that different threads have different paging tables (It might be alpha's relict, whatever)

Anyway. MS designed it in those ancient times when they still cared that their NT OS was technically good. Both delphi and gcc use this mechanism. Therefore, I'd suppose it probably makes some sense. (I don't have time to get deep into it right now myself, and not that I'm going to push anyone, especially developers, but I still don't see a reason why it couldn't be employed in FPC eventually)

Nikolai

Ranting on....

I did a small testing program using FPC on X86/32 Linux and found that GS stays the same with multiple threads (always $0033). So seemingly - just like with Windows (Linux using GS instead of FS) - it's not (as I supposed) that the GS value is a per-Thread constant, but the selector value is constant and the table it points to is different for each thread.

The address of the threadvar is the same in all threads, seemingly the address calculation does not acknowledge the threadvar type by modifying it's address to make relative to the DS selector (maybe this is not even possible at all). Don't use pointers to threadvars !!!! (Maybe this caused the problem mentioned in the other thread ???)

In Delphi (Win32) the address of the threadvar is different for the threads, as there is one more indirection in Windows, and the the FS-depending thread-specific area yields the DS-depending address of the final threadvar area.

-Michael

FWIW:

I checked gcc for the NIOS processor.

Same has 32 general purpose registers (similar to ARM) Here <R26>+offset is used to access any global (and static) variables, while <R23>+offset is used to access any threadvars. Obviously the OS just sees that these registers (like all others) are preemption-safe while the program is running. I don't know how the initial values of there registers are set (by the OS or by the program/thread initialization RTL function).

I suppose with this paradigm, pointers to threadvars should work (other than with X86/32-Linux), as the threadvars - other than with X86 - simply have have different effective user space addresses with the different threads they are used in. Thus even cross-thread usage of pointers to threadvars would be consistent (other then with X86/32 Linux and Windows).

-Michael

I suppose with this paradigm, pointers to threadvars should work (other than with X86/32-Linux), as the threadvars - other than with X86 - simply have have different effective user space addresses

What makes you think that this is different from x86?

DoDi

Hans-Peter Diettrich wrote

What makes you think that this is different from x86?

With X86/32 Linux, the "application"address of a threadvar in the ASM instruction is the same for all threads. The ASM instruction is done with a "GS:" prefix, thus instead of the DS sector, (that is used without the prefix), GS is used as a selector. Now the 16 bit selector reads an entry in an OS-provided table and same is added to the address given in the instruction. The OS provides a different table for each thread and thus the resulting memory address is different, even though the GS selector value itself is the same for each thread.

With X86/32 Windows, the "application"address of a thread var in the ASM instruction is the same for all threads, as well. The access is done reading a structure that exists for each thread, using the "FS:" prefix in the ASM instruction (this results in using the appropriate thread's structure, as - like with Linux - the OS provides a different selecting table for each thread and thus the resulting memory address is different, even though the FS selector value itself is the same for each thread. Now the threadvar is accessed using this pointer and the standard DS selector. One indirection more than is Linux, slightly less efficient.

With NIOS (and supposedly similar with ARM), there is no "selector" hardware. Instead, one of the 32 registers (R26, called GP ("General Pointer") ) is used to address normal static and global variables. Now another register (R23) is used to address threadvars. Within a running application the value of R26 is always the same, while the value of R23 is different for each thread (of course all registers are preemption-safe).

Regarding pointers this yields:

With X86/32 Linux, with C you can define a pointer type "__thread int i*". So this pointer knows that is handles a threadvar and thus the compiler can generate code accessing the threadvar via GS even when a pointer is used (I did not test yet whether this really works). With Pascal this is not possible (AFAIK). So using a pointer, a value is always accessed via DS. AFAIK, when calculating the pointer value it's technically not possible to "rebase" the address from GS to DS, as the selector tables can't be read from user space. Catch 22.

With X86/32 Windows, The threadvar is finally accessed via DS anyway. So no problem here. If you save the address of a threadvar in a global variable, a different thread will be able to access the threadvar of the thread that saved the pointer. I should test if FP really gives thread specific address-values for pointers to threadvars in Windows but not in Linux. (IMHO different behavior would be erroneous.)

With NIOS (and supposedly similar with ARM), the pointer value for a threadvar will be calculated adding the offset to R23 creating a normal memory address. So no problem here. If you save the address of a threadvar in a global variable, a different thread will 'be able to access the threadvar of the thread that saved the pointer.

- Michael

PS.: maybe some might be willing to help enhancing this list with real ARM, X86/64 Linux and X86/64 Windows knowledge,

> What makes you think that this is different from x86? With X86/32 Linux, the "application"address of a threadvar

Please distinguish: Segmented or Flat address?

Only FS and GS can add an non-zero offset, to the *offset* given in the instruction. The sum of both is the flat "address" (effectively another offset), that can be stored in a pointer variable. In 32/64 bit flat memory models all pointers are "near", i.e. they don't include an segment selector, and a zero-based segment is implied (DS...).

Thus GS provides an thread specific segment offset, to an block (record) of threadvars.

Here FS provides the thread specific segment offset.

Now the threadvar is accessed using this pointer and the standard DS selector. One indirection more than is Linux, slightly less efficient.

I.e. the intermediate result is a pointer to an block (record) of threadvars.

Here a register provides the offset to the block of threadvars.

Kind of a near pointer, with the thread segement (offset) implied.

You only have to determine the flat address, before storing it in the pointer variable - @MyThreadvar?

The only question is, whether you want the (absolute, flat) address of the threadvar of an *specific* thread, or the address of the threadvar in the *current* thread. Since the current thread can change, the offset of the threadvar record will change accordingly, and must be determined by the appropriate means (via segment or other address register).

DoDi

The segment ("Selector") content is not an offset (any more), but it uses a table of offsets that is managed by the OS. The user land software never sees this, so the pointer can't hold a "flat address", but when using a pointer some selector register needs to be used. if none is given, this is DS. As Pascal does not provide means to define a pointer type as being a pointer to a threadvar, it _always_ uses DS and thus a pointer to a threadvar is not possible with X86/32-Linux. Of course this is not true with a flat memory model. But is it possible to do "flat" software with FP on X86/32 Linux (or Windows) ?

-Michael

Hans-Peter Diettrich wrote

You only have to determine the flat address, before storing it in the pointer variable - @MyThreadvar?

How do you think this is possible ? How to you think it's possible in FPC to define a pointer that does not use DS to access the memory location ?

-Michael

This is what threadvars are all about,. Of course the effective address needs to be different for different threads (otherwise this would be just static/global variables, as the virtual address mapping is defined to be identical for all threads of an application).

The difference between the systems I checked seems to be that the non-flat address (disregarding the Selector to be used) (this seems to be what is stored in a pointer, assuming DS: in FPC) of the threadvars in different threads is identical with X86/Linux and different with all other systems. C can handle this by defining the pointer type appropriately (with "__thread") while FPC does not seem to have this ability.

-Michael

The segment selector refers to an segment descriptor, to which above segment-base restrictions apply in the flat memory model.

We are doing "flat" software all the time. The last use of segmented pointers dates back to Win3.1, with near and far pointers. The IA32 architecture restricted the total address space to 4GB (32 bit), with rare exceptions for server-type CPUs. Otherwise we could have used 2^16 different segments of 4GB each, in 32 bit applications. Since that time segment registers and selectors can serve only memory protection purposes, but can not extend the address space beyond the hard 4GB limit.

DoDi

No need to do so. Every accessible byte must reside in the 32/64 bit address space, so it is addressable by DS and an offset.

DoDi

Hans-Peter Diettrich wrote

We are doing "flat" software all the time. The last use of segmented pointers dates back to Win3.1, with near and far pointers.

It was you who introduced the flat vs segmented paradigm in this discussion. :)

Nonetheless, we _always_ do use segment registers (sometimes explicitly, mostly implicitly CS: DS, SS as appropriate, as the hardware and the OS forces this. For data access DS: and SS; are used, which contain the same value ($0023 with Windows, $007B with Linux, no idea what the these values mean). Only using threadvars asks for using another segment register (FS: with Windows 32 Bit, GS with Linux, details see above) - even if the user software does not ever modify the segment registers.

Of course pointers (doing something like "mov EAX,[EAX]") implicitly use DS:, as well. This does work for variables allocated on the stack, because SS and DS always contain the same value. The user software just sees the address relative to the value in the selector table denoted by as well DS: as SS:. So it simply does not need to care about segments (only not true with threadvars in X86/32-Linux, see above).

I don't care if this is called "segmented" or "flat software" :).

-Michael

Sorry but wrong (IMHO) with pointers to threadvars in X86/32 Linux (see above).

While indeed (supposedly) any threadvar is allocated withing the address space of the process (addressable by DS:), the user software has no chance to calculate this offset, as (AFAIK) it can't read the selector table entries that are pointed to by the (different) values in the DS and GS registers, which would be necessary to "rebase" the threadvar address that is relative to the entry table given by the GS content and a target address relative to the table entry given by the DS content.

Regarding the hardware and without rewriting or switching the selector table, a user land program could access 64K different (potentially non-overlapping) address spaces (each 4 GB size) using a selector register such as ES.

Without writing to a selector register and explicitly using a selector register in any ASM instruction, a user land program can theoretically use three different (potentially non-overlapping) address spaces: CS for instruction reads, SS for data access relative to the stack pointer and DS for all other data access. Of course the OS sees that DS and SS content is identical, so that pointers to stack based variables are possible.

-Michael

I'm always happy to learn and be helped to clearly phrase what I want to express, if I fail to do so. What exactly do you think is wrong with my statement above ?

-Michael

Hans-Peter Diettrich wrote

Getting the linear address of a threadvar depends on the OS/FPC implementation.

Of course the compilker needs to adhere to what the OS provides.

As already said multiple times, Win32 for each thread provides a block of user space memory that is addressed by FS and here the base address (to be used via DS) of the threadvar area thus the threadvr itself is rto be accessed using DS. OTOH, Linux X86 provides the address of the threadvar area directly in GS, so that the threadvar needs to be accessed using GS.

As Pascal (other than C) does not provide a means to define that a pointer type is a pointer to a threadvar rather than to a normal var, Pascal alway uses DS to access any variable.

As the user program only sees linear addresses (which have 32 Bit offsets are different regarding to which segment register (or segment register content is used) there are no means for the user land program to calculate a DS-denoted offset (to be used by a pointer) from the GS-denoted offset it knows for the threadvar.

IMO FPC *should* allow to get the address of a threadvar by AddressOf(threadvar)

Usually It does - using DS (the default, thus not explicitly noting a segment register in the ASM instruction).
Only with X86/32-Linux when not using a pointer it can't do this (in a high-performance way), as the DS-based offset of the threadvar is not known. In fact it does it "the slow way", doing a function call for any access to a threadvar. As Lazarus does not provide a means to do single setpping in ASM code, I was not able to find out what actually is done here.

- test yourself. It may be possible to get that address also with an LEA instruction (untested).

I did the test.

lea eax, FS:[$34] resulst in EAX = $34, even though FS is $3B and

mov eax, FS:[$34] results in correctly loading ax from memory and

mov eax,[eax] with eax = $34 correctly results in an access violation

in fact I did not suppose that
lea eax, FS:[$34] would compile at all but it does

the code of lea eax, FS:[$34] is 64 8D 05 34 00 00 00, using the DS prefix "64" while
the code of lea eax, $34] is 8D 05 34 00 00 00

I don't see what lea eax, FS:[$34] in fact is supposed to do.

AFAIR Delphi obtains the address of the threadvar memory block via FS/GS,

Right via FS it gets the "thread management block" that contains the (DS-based) offset of the threadvar block, (as stated before. and exactly as M$ C does).

so that it's possible to use the same instruction sequence,

This is what I tried to express in the beginning.

except that finally not the value of the threadvar is retrieved,

Yep, this is as Win32 handles it (but other than Linux does )

but its address (LEA instead of MOV).

Nope.

to read a normal var it needs to do
mov eax, normvar; (this is implicitely DS based)

to get the pointer to a normal var it needs to do
lea eax, normalvar; (this is implicitely DS based)

to read a threadvar, it needs to do
mov eax, FS:(threadmanagementblock+thrfeafvarblockoffset); (this results in the DS-based address of the threadvar block) mov eax, [eax+threadvar]

to get the pointer to a thrfeadvar it can do
mov eax, FS:(threadmanagementblock+thrfeafvarblockoffset); (this results in the DS-based address of the threadvar block) lea eax, [eax+threadvar]; (this results in the DS-based address of the threadvar)
or
mov eax, FS:(threadmanagementblock+thrfeafvarblockoffset); (this results in the DS-based address of the threadvar block) add eax, offset (threadvar)]; (this results in the DS-based address of the threadvar)

this can't be done in a single ASM instruction

-Michael

Hans-Peter Diettrich wrote

Well, different thread implementations exist for Linux. And the threadvar implementation is fully up to the compiler - so that FPC could use a Linux threadvar that holds the address of the FPC threadvar block - just like implemented for Windows.

OK, the FPC RTL could create a one word sized "thread management block", just to hold the DS-notation of the offset of the threadvar block. This is other than GNU C in Linux works, but maybe this in fact is a good idea as it allows for pointers to threadvars and is faster than the current implementation (with any access to a threadvar calling a function that I failed to analyze).

Since all pointers are linear addresses, this is not a problem.

Look at the document I mentioned: a "linear address" is 48 bits, while a pointer is only 32 bits (the "Offset") . So with FPC and Delphi always DS is used to provide the additional 16 bits, while with C (supposedly) GS is used to provide the additional 16 bits in case of a pointer type that points to a variable of a type denoted with "__thread".

Too bad, I had thought that LEA takes into account the segment base :-(

And I assumed it would not compile if a segment is denoted (also :-( :).

If lea would take the segment base into account (a nice idea) it would need to know about both the segment to be used as a source as the segment that the result is to be used with. Giving multiple segment prefixes is not possible. Moreover LEA would need to result in an error if the source and/or target segment does not provide the requested address.

Thus LEA without a segment notification assumes source and target segment register to be the same, i.e. just handles the 32 Bit "Offset". I fail to understand what LEA is supposed to do if a (single) segment register is given :(.

Funny stuff..

-Michael

Von: Michael Schnell :

As Pascal (other than C) does not provide a means to define that a pointer type is a pointer to a threadvar rather than to a normal var, Pascal alway uses DS to access any variable.

JFTR, plain vanilla C (i.e. ISO C99) does not even have a notion of a "thread variable", because it doesn't have threads. So I don't know what C you're referring to here.

> - test yourself. It may be possible to get that address also with an > LEA instruction (untested). I did the test.

No need to. The segment value is not used by LEA. It calculates the "effective address" which basically means the offset.

I'm sure you meant "FS prefix". DS doesn't have one AFAIR.

the code of lea eax, $34] is 8D 05 34 00 00 00 I don't see what lea eax, FS:[$34] in fact is supposed to do.

Loading the effective address. Any decent assembler could "optimize" that code into a "mov eax, 34h".

Or do that beforehand and "simply" store the pointer to the threadvar. This must be done anyway, because via pointer is the only way to reference a threadvar.

That means that:

threadvar x : Integer;
var       y : ^Integer;

should result in a proper initialization of "x" by the compiler when a thread is started and references would implicitly change like this:

x := 1;  => x^ := 1;
y := @x; => y  := x;

In that way, there is not even the notion of a "pointer to a threadvar" needed, because the compiler can handle that transparently like another reference type.

Actually, I was under the impression that the compiler does exactly such a transformation already.

Vinzent.

Michael Schnell :

I fail to understand what LEA is supposed to do if a (single) segment register is given :(.

Ignoring it, because it is not part of the "effective address" which it is supposed to calculate?

<http://www.intel.com/software/products/documentation/vlin/mergedprojects/analyzer_ec/mergedprojects/reference_olh/mergedprojects/instructions/instruct32_hh/vc150.htm>:

|"The source operand is a memory address (offset part)" ^^^^^^^^^^^

Says that only the offset is used.

|"specified with one of the processors addressing modes" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Obviously this includes "segmented addressing modes" with segment (or descriptor) prefixes.

So I don't know why you seem so confused. After all, it's x86. See signature. :)

Vinzent.

OK, it says the offset part is _specified_. But the text describes the LEA instruction ("8D") itself. regarding the hardware, the segment prefix is another self-contained instruction, that does nothing but set a flag to be used (and always cleared) by the next instruction. As normally it's obvious what they do, the description of the instructions usually does not mention these flags. even though, regarding the ASM syntax it's allowed to include the "FS:" in the ASM instruction itself (with the address specification).

IMHO, if the compiler should issue an error message if yo9u give a segment register with LEA, as the result obviously is erroneous. Maybe other compilers in fact do this. I only tested Turbo Delphi.

So I don't know why you seem so confused. After all, it's x86. See signature. :)

To male the LEA instruction work decently with segment registers you would need to give two of them, one for the "source" address specification and another one for the "target" (the segment register the calculated offset is to be used with). This very obviously is not possible with the normal ways of the instruction coding ant that is why I never assumed it would be possible to specify even only one. The current implementation of "LEA" just assumes the same same segment both for "source" as for "target", and thus it's clear what it is supposed to do (it ignores the flag set by any segment prefix instruction).

-Michael

OK I was wrong by two bits.

See http://flint.cs.yale.edu/cs422/doc/24547212.pdf page 3-7

The "logical address" is 24 bits: 16 bits "Segment Selector" and 32 bits "Offset". Obviously in an instruction, usually the "Segment Selector" is taken from one of the six Segment registers while the "Offset" is calculated in the Instruction from a constant and/or some register(s). Which Segment register is to be used is determined by the instruction (either DS or SS) and can be changed by the use of a prefix (*).

The Selector is used to take one of up to 2^13 "Segment Descriptor"s out of a set of two Descriptor tables. So there are up to 2^14 = 16,384 (and not 64 K as I assumed) such "Segment Descriptors". (Obviously much less are actually used practically.)

The (64 bit sized) Segment Descriptor provides a 32 bit "Base Address".

The final "Linear Address" is calculated by adding the Offset to this "Base Address"

This in fact allows for using a max of 16,384 Segments, each 4 GByte sized at max, but limited by a size given in the Segment Descriptor.

These Segments can overlap, but theoretically 16,384 non overlapping segments can be constructed. In this case of course they can't be 4 GByte each,

Of course all this is hardware, disregarding what limits any OS might impose.

-Michael

(*) "For virtually any kind of program execution to take place, at least the code-segment (CS), data-segment (DS), and stack-segment
(SS) registers must be loaded with valid segment selectors. The processor also provides three
additional data-segment registers (ES, FS, and GS), which can be used to make additional data
segments available to the currently executing program (or task)."

Figure 3-5 in the document (
http://flint.cs.yale.edu/cs422/doc/24547212.pdf ) clearly states that the "Logical Address" is 48 Bits. This is what the "Linear address" is calculated from.

During each instruction execution first a 48 Bit "Logical Address" is calculated and the hardware constructs a 32 bit logical address from same.

So to understand (or define) what an ASM instruction actually does, it's necessary to look at the "Logical Address" it deals with. Same is always constructed using segment registers and therefor influenced by the flags that might be set by a preceding "segment prefix" instruction.

An exception seems to be the the "LEA" instruction that just ignores the "Seg. Selector" part (upper 16 bits) of the Logical Address and just works with the "Offset" part (lower 32 bits) of the Logical Address, thus assuming the same "Seg. Selector" (whichever !) as well for the target as for the source.

The "Physical Address" is independent of segment registers and comes into play later in the process, introduced by the MMU, which in fact is "outside" of the CPU core.

-Michael

"Vinzent Höfler" wrote

JFTR, plain vanilla C (i.e. ISO C99) does not even have a notion of a "thread variable", because it doesn't have threads. So I don't know what C you're referring to here.

GNU C provides the "__thread" keyword to specify threadvars. M$ C provides "declspec(thread)" for the same purpose. I did check that this work (with M$C on WIN32 and with GNU C on NIOS/Linux) and took a look at the code constructed when accessing a threadvar.

I do know that there are GNU C implementations that don't support the __thread keyword, so it seems to be "optional".

In fact I did not reverse-engineer how one of these compiler deals with pointers to threadvars, but as __thread is a specification to a type and C can construct a pointer to any definable type, not doing it the correct way would be erroneous.

No need to. The segment value is not used by LEA. It calculates the "effective address" which basically means the offset.

Yep "basically". :) But with other instructions the same notation and description as used for "effective address" with LEA is used to access a value in memory and thus the 48 bit "Logical address" (see Figure 3-5 in http://flint.cs.yale.edu/cs422/doc/24547212.pdf ). I in fact did expect that LEA just works works with "Offsets", but I feel that it is not really unambiguously stated in the docs.

I'm sure you meant "FS prefix". DS doesn't have one AFAIR.

I did not check this
DS: PUSH AX

might make some kind of sense :)

Loading the effective address. Any decent assembler could "optimize" that code into a "mov eax, 34h".

the optimization would be just do kill the obviously unused prefix instruction.

Or do that beforehand and "simply" store the pointer to the threadvar.

There is no "the threadvar", as the memory location of same is different with each thread. So even the wording "the pointer to the threadvar" is ambiguous. If the pointer variable itself is a threadvar or a local var in a function, it might hold the correct thread specific address of the threadvar.

If the pointer variable is static or global, it can be used in one thread and hold the address of the thread var specific for _another_ thread.

So what do we want to accomplish in a certain program ? This obviously is not a problem of the compiler or the implementation, but a more "general" issue.

-Michael

IMHO, if the compiler should issue an error message if yo9u give a segment register with LEA, as the result obviously is erroneous. Maybe other compilers in fact do this. I only tested Turbo Delphi.

I had thought that this has been fixed on newer architectures, that use the flat model only. There you don't have to specify a target segment, because the effective address (32 bit offset) is sufficient.

IMO you still think too much about segment registers, as used in 16 bit code...

DoDi

Please be a bit more precise with your bitcounts.

The Selector is used to take one of up to 2^13 "Segment Descriptor"s out of a set of two Descriptor tables.

Please accept that no more than 3 different segments are in use nowadays. FS/GS for non-zerobased segments, all others are zero-based. That's all you have to know about the segment stuff.

DoDi

Hans-Peter Diettrich wrote

I had thought that this has been fixed on newer architectures, that use the flat model only.

This can't easily be "fixed" by hardware, how to express that LEA should convert e.g. a DS-based offset into a FS based one or the other way round, or even a GS based address into an FS based one ?

All other instructions work with at most a single "Logical Address" so a single prefix instruction (or set of flags denoting DS:, SS:, CS:, FS: or GS: ) is enough to denote how the 32 bit Offset is to be extended to the 48 Bit Linear Address.

LEA - if working on Logical addresses - in fact would need two sets of segment prefix flags. These are not provided by the basic paradigm of the instruction set and thus LEA works on 32 Offsets rather than on 48 Bit Linear Addresses.

-Michael

Sorry for the typo of course the Graphic shows an 48 Bit "Logical address" consisting of a 16 bit "Segment Selector" and a 32 bit "Offset" ;)

-Michael

Hans-Peter Diettrich wrote

Please accept that no more than 3 different segments are in use nowadays.

This is not true.

1) FS: as discussed in this thread, Windows uses FS to address the thread management block
2) ES: in my tests I modified ES in ASM code in a Delphi program and caused it to crash by this, So the Delphi compiler seems to use it (for whatever purpose)
3a) GS: is used to address threadvars with Linux
3b) in my tests I modified GS in ASM code ion a Delphi program and caused it to crash the debugger when single-stepping

User software is free to use ES, FS and GS in a decent way

Anyway, we are talking about the processor hardware here to explore what the compiler might be allowed to do and how to do it. Of course the OS and compiler internals imposes additional restrictions.

FS/GS for non-zerobased segments, all others are zero-based.

I have no idea what you mean by "zerobased".

Regarding the processor hardware, address calculations is done with the same algorithm, independent of what segment register is used in the instruction.

The only different that _as_a_default_ for certain address calculations certain segment registers are used. For calculating the address of an memory argument in an instruction the normal use (of DS) can be overridden by a flag that can be set by a prefix instruction, so that instead of DS one of CS, SS, GS or FS is to be used.

The defaults are
- CS for instruction address
- DS for memory operator address unless SP-relative
- SS for memory operator address when SP-relative

-Michael

Michael Schnell :

Well, LEA does not access the memory in question, otherwise more than just #UD could occur. So there's no MMU involved, thus: No descriptors. That's quite clear from the docs, I suppose. If it would access the memory, those nice multiplication tricks and efficient, different length NOPs used for aligning loops wouldn't be possible (because then you would have to make sure that the "address" part in things like "lea esi, [eax + eax*4]" denotes a valid address).

> Loading the effective address. Any decent assembler could "optimize" that code into a "mov eax, 34h". the optimization would be just do kill the obviously unused prefix instruction.

Hmm. An instruction encoding "FS: mov eax, 34h" would probably work the same. What's the processor supposed to do with the prefix anyway?

> Or do that beforehand and "simply" store the pointer to the threadvar. There is no "the threadvar", as the memory location of same is different with each thread.

Yes, I missed that. Still that's not the problem. The problem is where to store the pointer. :D So, in fact, the transformation has to be done each time you access a thread variable.

So even the wording "the pointer to the threadvar" is ambiguous.

Not after you resolved it to the "DS based location". Then it does in fact point to the very same instance of a thread variable, no matter what thread the access would happen in.

Yes.

So what do we want to accomplish in a certain program ? This obviously is not a problem of the compiler or the implementation, but a more "general" issue.

And - assuming a correct implementation by the compiler - is quite easily resolved by the programmer depending on what he wants. If the pointer is not a thread variable itself, it does point to the very same location (i.e. the thread variable of a specific thread). If it *is* a thread variable, each thread has its own pointer and thus each one points to a different instance:

threadvar
   x  : Integer;
   tp : ^Integer;
var
   gp : ^Integer;
tp := @x; // Points to a thread's specific instance of x.
gp := @x; // Points to a specific thread's instance of x.

Apart from the fact, the it gets really complex when such variables are exchanged between different threads (but then: why do you used thread variables at all?), am I missing something here?

Vinzent.

Michael Schnell :

2) ES: in my tests I modified ES in ASM code in a Delphi program and caused it to crash by this, So the Delphi compiler seems to use it (for whatever purpose)

No surprise, in any sane environment DS should be equal to ES. Most notorious instruction requiring that would be "movsb/w/d" which is using ES: for the target address. AFAIR a segment override is possible there, but only for the source address.

Vinzent.

> FS/GS for non-zerobased segments, all others are zero-based. I have no idea what you mean by "zerobased".

Then you missed fundamental 32/64 bit hardware and software principles :-(

Please read and understand the available documentation, my preceding posts, and ask further questions by PM.

DoDi

Such a deriding tone is completely uncalled for (and moreover you yourself are constantly asking very basic questions about "make" and debugging the compiler on this same list, which makes it doubly inappropriate to start publicly chastising other people simply because they're not familiar with one term you used).

This thread has also strayed way off topic with lots of back and forth emails about details of the x86 instruction set. In the end, a summary about how everything applies to a realistic threadvar implementation for FPC could be useful (but keep in mind that the only reason that intialising/finalising threads not started via the FPC rtl now works under Unix is exactly because every threadvar access requires a call into a helper routine).

However, most of what's currently under discussion has been said several times already in multiple ways and what's left seems to be mostly miscommunications and clarifications.

This thread is now moderated on this list.

Jonas
FPC mailing lists admin

Jonas Maebe wrote

(but keep in mind that the only reason that intialising/finalising threads not started via the FPC rtl now works under Unix is exactly because every threadvar access requires a call into a helper routine).

Of course this is why I started the thread. The C compilers seem to work more efficient on that behalf and thus it might be viable to improve FPC. To do that the details of how segment registers are used with the X86/32 architecture in Linux and Windows needs to be thoroughly understood. (In fact I still don't know how the compiler should construct a pointer to a threadvar in Linux).

-Michael

How can a C compiler be more efficient, if the threadvar concept does not exist in C ? On posix platforms, you must use a series of thread calls to get to your 'thread-local' variable. When accessing a thread variable, FPC uses the same series of calls as one would in C using pthreads.

Michael.

I said that the (recently introduced) support for externally started threads under Unix *depends* on the fact that you cannot access a threadvar without calling that helper routine at least once. You did not start the thread with the purpose of breaking that support again, but as I said such breakage would be a direct consequence of moving to direct selector-based accesses on Unix platforms.

The C compilers seem to work more efficient on that behalf and thus it might be viable to improve FPC.

If you really need fast access to a thread variable, take its address and use that afterwards. That's what e.g. the heap manager does. In general, I doubt that the speed of accessing threavars matters much to most applications.

Nevertheless, if you want to add support for accessing threadvars under Windows via a segment override, feel free to debug it. The basic code is already in the compiler from previous (failed) experiments. Uncomment tf_section_threadvars in compiler/systems/i_win.pas, and grep for its occurrences in the compiler source code to see what it does.

To do that the details of how segment registers are used with the X86/32 architecture in Linux and Windows needs to be thoroughly understood.

This thread has gone on long enough about that subject, and most of the details that have been discussed are completely irrelevant regarding anything FPC would have to do when using them to access threadvars. If you feel that anything further needs discussing about that, then do it on a list specific to low level programming on Linux, Windows or for the x86 architecture (or on fpc-other).

(In fact I still don't know how the compiler should construct a pointer to a threadvar in Linux).

It (indirectly) calls CRelocateThreadvar() in rtl/unix/cthreads.pp

Jonas

Michael Van Canneyt wrote

How can a C compiler be more efficient, if the threadvar concept does not exist in C ?

Most C compilers nowadays have extensions to support threadvars using special keywords.

Jonas

OK all points taken.

The pthread-library is used to manage the pointers to the base of the threadvar block. (Same of course does use the GS: stuff on Linux-X86/32 to manage it's own thread specific storage.)

Besides the performance argument, the original argument that triggered the discussion was that somebody suggested that it would be a good idea to get rid of the libc (and thus pthreadlib) dependency.

-Michael

IMO there exists no requirement, that POSIX threadvars are accessible in Pascal. So FPC could use the Windows scheme for its own threadvars, where the OS only provides the address of an thread-specific memory block - equivalent to 1 POSIX "thread-local" variable. The referenced memory blocks, one per thread and containing all the Pascal threadvars, can be allocated on the heap.

DoDi

Michael Schnell :

In fact I still don't know how the compiler should construct a pointer to a threadvar in Linux.

pthread_getspecific()

It returns a pointer to it (well, to the block of associated memory, so maybe adding an offset would be required if more than just one variable is stored there).

Vinzent.

Jonas Maebe wrote

I don't see it there (to the extent that there is an official win32 abi document; it seems that information is spread over several msdn pages that don't even link to each other). Please stop speculating that certain things "should be documented" or "should work" and post links to the official documents. Sending us off on wild goose chases does not help.

This document

http://www.akkadia.org/drepper/tls.pdf

explicitly states that the implementation of the thrdeadvar needs to be part of the ABI of the compiler. (Thanks to Vinzent Höfler to point me to this file).

-Michael

This is an extension to C which is specific to GCC; it is in no way an official spec except maybe for GCC (and probably linux-only, at that). Microsoft for example doesn't use ELF; as I understand his remark, Jonas wanted an official Microsoft spec.

Other than that, it's of course useful to know.

Michael.

Finally I think that there is a decent chance to modify the compiler and the RTL in a way that threadvars can be used fast and without needing a libc binding, as well reading and writing them and constructing a pointer to them (which seemed impossible to me recently):

Linux X86/64: using this document
Linux X86/32: using this document (*),
Windows X86/32: using the document mentioned in another contribution, based on reverse-engineering of the Microsoft C compiler (*) Linux ARM: tbd, but seemingly quite obvious
WinCE ARM. tbd, but I suppose this is not really "on the list".

(*) as both archs provide a TCB, the implementation can be done very similar, first constructing the DS-relative pointer to a threadvar and then doing the access:

Windows: the pointer to the threadvar is constructed by MOV EAX, FS:Windows_TCB-element_offset
MOV EAX, threadvaroffset[EAX]

Linux: the pointer to the threadvar is constructed by MOV EAX, GS:Linux_TCB-element_offset
MOV EAX, threadvaroffset[EAX]

-Michael

Op Tue, 10 Aug 2010, schreef Michael Schnell:

> Finally I think that there is a decent chance to modify the compiler and the RTL in a way that threadvars can be used fast and without needing a libc binding

Yes there is. I worked on this a long time ago, but I didn't finish it. The code for that is still in the compiler, work on that can be continued.

However, it won't be usefull without libc independend thread mananger.

Daniël

Daniël Mantione wrote

However, it won't be usefull without libc independend thread mananger.

Of course you are very right here !

Is there any question that I might be able to help with (provided that I did a lot of research regarding porting a heavily multi-threaded / multi-language / multi-arch embedded project).

-Michael