Hello,

As written in other mails, I'm currently working on the Unicode RTL of FPC.

Debugging programs using the unicode RTL is possible, but not comfortable:

I have currently identified 2 problems in Lazarus which make debugging such programs difficult:

- The IDE does now know the -tSUBTARGET option.
This means that it does not correctly identify the IFDEF paths in source code, as it didn't correctly parse all config files.

- Debugging programs works in general quite OK, except for displaying of some strings. Most notable, the exception message is affected: only the first character of the exception message is shown.

It seems to me obvious that the IDE needs to be aware of the -t option. But for debugging this is not enough: I think the IDE debugger needs another mechanism to determine using whether a program uses a unicode RTL or not. The way this happens in code is
{$IF SIZEOF(CHAR)=2}
// unicode RTL code.
{$ELSE}
// Ansi rtl code.
{$ENDIF}
Obviously, this is of no use in the debugger. The subtarget name is also not usable, since any subtarget definition can specify that the RTL should be unicode.

Questions:

1/ Should the compiler provide some symbol in - say - the system unit to deterine whether the RTL is compiled in unicode mode, or is another mechanism preferred to do this ?

2/ Where can I find the code that extracts the message from an exception object ? So I can have a shot at trying to fix the display.

Michael.

Michael Van Canneyt via lazarus schrieb am
Do., 12. Jan. 2023, 10:42:

"Char" already is such a symbol that the debugger can and must use to determine this. And if no debug information for aliases is created then *that* needs to be fixed instead.
But the debugger also needs to be able to determine Unicode vs. ANSI if no debug information for the System unit is available thus it needs to rely on CodeTools then and what *that* provides for Char.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20230112/937be290/attachment.htm>

In earlier discussions I was told

Type
   Char = UnicodeChar;
Var
   C : Char;

means 'replace char with unicodechar'.

So in the final binary only the effect of the declaration

var
   C : unicodechar ;

will be seen. Which is IMO the only correct behaviour.

If you want a different type you need to do

Type
   Char = Type Unicodechar;

which creates 2 distinct types which are assignment compatible. This will lead to other problems.

I also don't think the debugger needs to be able to determine the definition of "char".

It needs to determine the definition of "string" which will be an array of unicodechar or ansichar. Once it knows that, the rest will follow, since the definition of unicodechar or ansichar will have the correct size.

But the debugger also needs to be able to determine Unicode vs. ANSI if no debug information for the System unit is available thus it needs to rely on CodeTools then and what *that* provides for Char.

I don't think this is a very reliable method. In moderately difficult code, half of the time the codetools don't work properly, so relying on them to provide this info is a recipe for disaster. Hence my question...

Michael.

Michael Van Canneyt wrote

- Debugging programs works in general quite OK, except for displaying of ? some strings. Most notable, the exception message is affected: ? only the first character of the exception message is shown.

This is hardcoded, and may be change-able, if debug info is avail.....

With a look at the next paragraph below....
If
? ? Exception = class .... end
?is available (ideally within the correct unit, so the debugger can distinguish it from some user declared type), then the debugger could use this.

Ideally it will by in unit system/sysutils.

NOTE:
On Linux, such symbols are referenced cross units. But on Windows they are repeated in every unit using them. That again makes it harder to know if it is the correct type (since the user may have its own type by the name).

But for debugging this is not enough: I think the IDE debugger needs another mechanism to determine using whether a program uses a unicode RTL or not. The way this happens in code is {$IF SIZEOF(CHAR)=2}

I don't think the debugger should have or need such a "global" flag.

Currently if it looks at a string (and there is dwarf debug info), then the dwarf contains the size-of-char: 1 or 2.
And the debugger does utf8 or utf16 (the latter may be buggy).

Joost even added code to read the hidden encoding field of strings.

Though this relies on being able to detect
?? PChar? ? String? ? array of char
which is a PITA.

I don't think that is useful.
Better find the individual places that go wrong, and fix them.

As I said, Exceptions are hardcoded (and IIRC not even in FpDebug, but in LazDebuggerFp).
They could/should first search for the type "Exception"

2/ Where can I find the code that extracts the message from an exception object ? So I can have a shot at trying to fix the display.

unit FpDebugDebugger;
procedure TFpDebugDebugger.HandleSoftwareException
// Get address of object

? if not
FDbgController.DefaultContext.ReadUnsignedInt(FDbgController.CurrentProcess.CallParamDefaultLocation(0), ??? SizeVal(SizeOf(AnExceptionObjectLocation)), AnExceptionObjectLocation) ? then

....
 ??? ExceptionMessage :=
ReadAnsiString(AnExceptionObjectLocation+DBGPTRSIZE[FDbgController.CurrentProcess.Mode]);

That currently does not yet have the read of the "encoding" field.

-------------------------------------------------------------------------------------------------------- What about:
?? TSomeClass.Classname

is that utf16 too?
Because the debugger reads it for automatic class casting

Any other changes in the RTTI where things switched to 16 bit?

The big issues is still, that dwarf does not make any (official/documented) diff between PChar, string and array of char. Except for internal encoding details ....

I don't think this is a very reliable method. In moderately difficult code, half of the time the codetools don't work properly, so relying on them to provide this info is a recipe for disaster. Hence my question...

The debugger does and should not relay on codetools.

Well, there is a? case where codetool helps preparing.... If you add a watch (or hover for a hint), codetool will determine the boundaries of the expression.

So?? "foo.bar"? will not just add bar, but the correct term. Or? "foo.bar[x.y]"?? if you hover on y, will get x.y, but if you hover on "]" will get the full term.

And codetool will resolve "with" at that time.

But that is it.
And it should be it.

Source can have changed in the mean time.

....

Important:

If you make any change is the FpDebug package itself, and code that is FPC specific must go into
? unit FpDbgDwarfFreePascal;

All other units should be kept free from such knowledge (not 100% achieved yet, but...).
After all, FpDebug is meant to be a generic debugger for any app that has dwarf.

Exception may be semi-generic stuff like prologue detection based on fpc, but potentially working for other compilers too.

---------------------------
Of course
?? LazDebuggerFp? (containing FpDebugDebugger)
is in a diff package.
And it can have as much FPC knowledge as it wants.

Sorry, I don't understand at all what you are saying here.

?

I have meanwhile established that it has some problems: I have the impression that PChar is hardcoded in some places to mean UTF8.

See below.

Important remark: Make it a habit not to use Char any more.

It no longer exists as a type. It is an alias to AnsiChar or Wide/UnicodeChar.

So be specific: use AnsiChar or UnicodeChar (and their pointer variants).

For code that does not care whether it is 1 or 2-byte, you can still use char.

But for such low-level things as a debugger, the use of "char" and "pchar" is IMO absolutely forbidden.

That seems prudent, yes :-)

code will need some changes.

No. It remains shortstring.

Because the debugger reads it for automatic class casting Any other changes in the RTTI where things switched to 16 bit?

None spring to mind.

Michael.

Since the PChar/Char types are now an alias, logically the debug info should only have references to AnsiChar/UnicodeChar.

But I have no idea how to test this assumption.

We are 100% agreed on that, hence my questions.

So, if I understand you correctly:

The problem is now to see if the debugger can determine from the available debug info/rtti whether the RTL uses AnsiChar or UnicodeChar.

Which still leaves the question:
What to do if there is no debug info for the system unit ? By default, FPC distributes units without debug info.

As far as I know, TObject does not have any RTTI associated with it.

Michael.

I may have to double check some details, just going from memory....

1)
If you look at the current code getting the message, it just hardcoded. Get pointer to instance, add hardcoded offset to FMessage field. Assume it is an AnsiString. Read it.

But if it could find (the correct) type "Exception" describing the class, then it could evaluate:
???? Exception( instance_addr_from_reg ).FMessage

And thus get the dwarf info for FMessage which includes the size-of-char.

2)
However a user can define his own class name "Exception" that has nothing in common with the above.
So if LazDebuggerFp could relay on finding this class in unit system or sysutils => then that would make it more fool proof.

If the class is not found, the current behaviour can be used as fallback.

3)
Looking up "char" is possible. But there can be a "char" definition in each unit.... (On Windows there almost certainly is)

If I use TFooList from unit Foo, then a dwarf definition of TFooList is expected in unit Foo.

IIRC, on linux that is the case.
And on linux any other unit has a reference to that unit.

On Windows: There may be, if the type is actually used in unit Foo. On Windows, every other unit in which it is used, will repeat the entire dwarf.

This may depend on the linker..... ??

Example? I did not find that " below"

Anyway, what you might have seen, is not that the name "PChar" maps to utf8, but that the dwarf description of:
? ? ? pointer > basic-dwarf-type(char with size =1) Is interpreted as such.

For such a type it is not known what the encoding is. (Eventually the user should be able to chose in the watch-properties). But in LCL apps, the above is most likely a PAnsiChar to a (sub-)string that is utf8 encoded.

....

But for such low-level things as a debugger, the use of "char" and "pchar" is IMO absolutely forbidden.

For the debugger the name of the type is irrelevant. It looks up the dwarf definition.

If the user's code contains a type name FooChar, then the debugger has no influence on that.
Equally if the user enters a watch: ^char(xy)^

This code IIRC only acts, if it finds:
??? basic-dwarf-type(char with size =1)

So that should be ok.

It should be in the
 ? unit FpDbgDwarfFreePascal;

if adding a watch containing? "^char(x)" fails, then that will upset plenty of people. (including me)

That alias should always be encoded as type, if it is used in the user's code.

So, if I understand you correctly: The problem is now to see if the debugger can determine from the available debug info/rtti whether the RTL uses AnsiChar or UnicodeChar.

Not really.

Whenever the debugger accesses "char" or "sequence of char" data, it should have debug info for this.
If it has, this debug info will contain the size-of for that/those chars. And that is enough to decide if that particular instance is utf8 or utf16.

If the RTL (or user code) would be partly AnsiChar, and partly UnicodeChar... => no problem.

The problem is, that the Exception.FMessage is not using dwarf info. (Because it is not always included)

Which still leaves the question: What to do if there is no debug info for the system unit ? By default, FPC distributes units without debug info.

Well if it is not possible to include
??? type sysutils.Exception
in the dwarf info,
then some other kind of flag needs to be agreed that is on top of the normal dwarf.

The question is where to store this.
- It could be some $$$char? type declaration (to make sure it can't clash with user names). But then should be in a specific unit, so the debugger knows where to look.
- It could be an addition to the "producer" in the Compilation-unit (but that needs to be repeated in every CU)
- It could be a "linker symbol" like FPC_RAISEEXCEPTION", ? where the compiler either includes FPC_CHAR_IS_ANSI or FPC_CHAR_IS_UNI

Or, depending on how the whole stuff works with the linker.... Even if sysutils.o does not have debug info, the compiler could write and include a dummy .o file that has debug info and gets included. It may have to use a diff unit name for that: sysutils$$fpc.o

Those are just ideas, I don't know the compiler/linker well enough....

Any "global" flag, will cause extra work when it comes to dll/libraries. Because then all FpDebug code needs to handle this per dll.

Michael Van Canneyt wrote

Since the PChar/Char types are now an alias, logically the debug info should only have references to AnsiChar/UnicodeChar.

Well technically neither of them is (always) an actual character.

They should be called

AnsiCodepoint
UniCodeCodePoint

So people will be reminded that they do not contain characters (they may by coincidence, but software should not be build on coincidence)

But that is just my personal 2cent.

Martin Frb wrote

Though this relies on being able to detect ?? PChar? ? String? ? array of char which is a PITA.

Example:
? FPC 3.3.1 (but IIRC also 3.2.x)
? tested on Win, but similar on other OS
? dwarf-3

var
 ? s1: AnsiString;
 ? s2: array of char;

Both are encoded as
? ? DW_TAG_array_type

The name is not relevant, since the user could have done "type Foo = type AnsiString"

The debugger can only make the difference by looking at "current implementation in fpc"

For an array, FPC adds
- a stride
- a type for the subrange.

Both are not guaranteed, and could change any day.

If they do, a string may suddenly be shown as
? s1:?? ('H','e','l','l','o',' ','W','o','r','l','d')

----------------
With DWARF-2? string are using the same dwarf as "PChar" (independent of its new name).

So for the watch "s1[2]" you get the value
PChar 'l'; String 'e'

Because the debugger simply can't tell if the index is 1 or 0 based.

No, since in the case of a surrogate it is not a codepoint either. It is the unit of encoding granularity, nothing more.

But anyway, that ship sailed in (D)2009.? It is probably the ever lasting tension between easier transition and long term best nomenclature.

So people will be reminded that they do not contain characters (they may by coincidence, but software should not be build on coincidence)

I have some doubts about such simple nomenclature really explaining something as complex as unicode. It will never be intuitive and always need study.

Ouch, yes got me there. And in utf8 it is also not a codepoint. I should have use "CodeUnit".

Agreed.
But I couldn't resist.