Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2013-03

Debugging fixed strings in UTF8 encoding

17 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Ernest V Miller

Hello!

As we know, Lazarus is UTF8 friendly. When I want to see a value of a string var in cyrillic, I should convert it to UTF8, else I only get "?????????" in Watches window.
For this code

 function TesterClass.quickTest : Boolean;
    var
      utf8, ansi : string;
      stf : string[50];
    begin
       ansi := Utf8ToAnsi('?????');
       utf8 := AnsiToUtf8(ansi);
       <....>
    end;

I see value of utf8 in russian. And I am absolutely happy. But as soon as I want to use fixed strings, the only way for me to see my native language when debugging is to declare an additional string var as you can see below:

function TesterClass.quickTest : Boolean;
    var
      utf8, ansi : string;
      stf : string[50];
    begin
       ansi := Utf8ToAnsi('?????');
       utf8 := AnsiToUtf8(ansi);
       stf := AnsiToUtf8(ansi);   // here I get
"#208#154#209#131#208#186#208#187#208#176" in Watches for stf
       utf8 := stf;                    //but here I see my grandfather's
language again
       <....>
    end;

I consider having a special string var for debugging russian is completely stupid. Maybe I don't know some GDB tweak to force it to recognize the characters of my ancestry in fixed strings?

--------------------
Regards,
Ernest Miller

Reply by email

Paul Ishenin
to Ernest V Miller
Ernest V Miller wrote
I consider having a special string var for debugging russian is completely stupid. Maybe I don't know some GDB tweak to force it to recognize the characters of my ancestry in fixed strings?

At the moment (FPC 2.6.x) ansistring variables does not contain encoding field which can help debugger to recognize encoding. When FPC 2.7.1 will out this will be possible.

The only solution I see is to have a combobox with possible ansi encodings to convert your string data.

Best regards,
Paul Ishenin

Reply by email

Martin
to Ernest V Miller: Hello! As we know, Lazarus is UTF8 friendly. When I want to see a value of a string var…
Ernest V Miller wrote 21 lines
Hello! As we know, Lazarus is UTF8 friendly. When I want to see a value of a string var in cyrillic, I should convert it to UTF8, else I only get "?????????" in Watches window. For this code function TesterClass.quickTest : Boolean; var utf8, ansi : string; stf : string[50]; begin ansi := Utf8ToAnsi('?????'); utf8 := AnsiToUtf8(ansi); <....> end; I see value of utf8 in russian. And I am absolutely happy. But as soon as I want to use fixed strings, the only way for me to see my native language when debugging is to declare an additional string var as you can see below:

Currently the only way would be to allow the user to specify how to interpret a watch, that is the user would set in the watches properties, if he wants utf8 or whatever encoding.

You can add a feature request, but currently that would have very low priority. (or "patches welcome")

Reply by email

Ernest V Miller
to Martin

As you could see in my previous examples, Lazarus shows the content of variable string correctly, no matter how the symbols are encoded. So there is no need in special encoding options in Watches. The point is that when IDE evaluates the string content, there is a difference for it whether the string has variable of fixed length. After all, strings are ordered sequences of characters and should be recognized very very similar.
I think it is a bug, and it seems to me that it can be fixed with minimal changes.

Martin wrote 28.03.2013 03:13:01:

Show quoted text 12 lines
Currently the only way would be to allow the user to specify how to interpret a watch, that is the user would set in the watches properties, if he wants utf8 or whatever encoding. You can add a feature request, but currently that would have very low priority. (or "patches welcome") -- _______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Martin
to Ernest V Miller

Sorry, I looked at the wrong part of your mail. I thought this was about the #208#154 representation. Probably because I had never debugged anything that caused the "???".
Maybe I did reproduce it now....

Also in your first example

ansi := Utf8ToAnsi('?????');

is that cyrillic? Can that be translated into ansi on your system? Have you tried to output the string

for i := 1 to length(ansi) do s := s + inttostr(ord(ansi[i]))+', ';

There are 2 issues (in what I reproduced):

1) *** "???"

Actually the "?" are really present in the string. They are put there by Utf8ToAnsi for characters that could not be translated to ansi. So in that the debugger shows the correct content.

At least that happens with the chars I used for testing:
var
s1, s2: String;
s3, s4: String[55];
begin
s1 := 'abc?a ? 1 ??';
s2 := Utf8ToAnsi(s1);
s3 := 'abc?a ? 1 ??';
s4 := Utf8ToAnsi(s1);

? = ansi 253
?? does not exist in my Ansi code page

Leads to http://imagebin.org/251942

2) *** #208#154

The IDE currently does not correct any of the shown values. They are all exactly as GDB returns them.

Internally AnsiString is a pointer. String[5] is not. Apparently that makes a diff for gdb.
Maybe newer (or even older) gdb will behave different. I have not tested. But gdb some bug in 7.3 (still present in 7.5) for windows. So I do not recommend it.

However this is something the IDE might be able to catch and fix. But as I saif, I have plenty other on my list.

If you want to do it yourself: One place where it could be done is gdbmidepugger.pp line 12604 in "procedure FixUpResult"

In order to go in the official code: It needs to check, that it only translates, if "#" sequences are actually utf8. Users may be looking at none utf8, and then translation will be unwelcome. And also it needs good test cases. And maybe it needs to be limited to string[].

Ernest V Miller wrote 31 lines
As you could see in my previous examples, Lazarus shows the content of variable string correctly, no matter how the symbols are encoded. So there is no need in special encoding options in Watches. The point is that when IDE evaluates the string content, there is a difference for it whether the string has variable of fixed length. After all, strings are ordered sequences of characters and should be recognized very very similar. I think it is a bug, and it seems to me that it can be fixed with minimal changes. Martin wrote 28.03.2013 03:13:01: > Currently the only way would be to allow the user to specify how to > interpret a watch, that is the user would set in the watches properties, > if he wants utf8 or whatever encoding. > > You can add a feature request, but currently that would have very low > priority. (or "patches welcome") > > -- > _______________________________________________ > Lazarus mailing list > > http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus -- _______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Ernest V Miller
to Martin

Martin ???????? 28.03.2013 19:58:17:

Show quoted text 6 lines
Also in your first example > ansi := Utf8ToAnsi('?????'); is that cyrillic? Can that be translated into ansi on your system? Have you tried to output the string for i := 1 to length(ansi) do s := s + inttostr(ord(ansi[i]))+', ';

Yes, it is cyrillic and it can be translated into ansi. My value of s after your test loop for '?????' is '202, 243, 234, 235, 224, '

Show quoted text 11 lines
There are 2 issues (in what I reproduced): 1) *** "???" Actually the "?" are really present in the string. They are put there by Utf8ToAnsi for characters that could not be translated to ansi. So in that the debugger shows the correct content. At least that happens with the chars I used for testing:

Despite the fact that each character of '?????' is represented in ansi, Utf8ToAnsi gives ? in exchange for every normal ansi symbol. Interesting thing is that further call of

            AnsiToUtf8(ansi);  //ansi contains '?????' as a result of

previous Utf8ToAnsi('?????') conversion
returns '?????' again, not '?????'.

Reply by email

Martin
to Ernest V Miller
Ernest V Miller wrote 19 lines
> > There are 2 issues (in what I reproduced): > > 1) *** "???" > > Actually the "?" are really present in the string. They are put there by > Utf8ToAnsi for characters that could not be translated to ansi. So in > that the debugger shows the correct content. > > At least that happens with the chars I used for testing: Despite the fact that each character of '?????' is represented in ansi, Utf8ToAnsi gives ? in exchange for every normal ansi symbol. Interesting thing is that further call of AnsiToUtf8(ansi); //ansi contains '?????' as a result of previous Utf8ToAnsi('?????') conversion returns '?????' again, not '?????'.

Ok, than you have an error that I can not reproduce on my system. I made my observation from reproducing the effects on my system as close as I can get to it.
For me GDB never did the ??? thing.

Can you generate a log please
http://wiki.lazarus.freepascal.org/GDB_Debugger_Tips#Log_info_for_debug_session

The above sting in 4 versions: utf8 and ansi both for unlimited strings and string[x]
And the source, so I can see what is what.

Is your ansi a multibyte ansi? Maybe GDB does not understand that? (just a wild guess)

Also:
Are you on Windows? Which one?
32 or 64 bit?
which version of GDB? (If you do the log, I can find it from that)

Can you try (including another log) using a newer GDB
* win 32 bit:
- IDE comes with 7.2-1
- go to http://sourceforge.net/projects/mingw/files/MinGW/Extension/gdb/
* win 64 bit
- IDE (not sure) IIRC 7.3.5
- http://www.drangon.org/mingw/

The IDE supports all versions (6.3 to 7.5 / upcoming 7.6 has not yet been tested).
But at least the 32 bit versions 7.3 and up have issues that can lead to gdb loosing the debug session)

Reply by email

Ernest V Miller
to Martin

Martin ???????? 29.03.2013 10:21:16:

Show quoted text 12 lines
Ok, than you have an error that I can not reproduce on my system. I made my observation from reproducing the effects on my system as close as I can get to it. For me GDB never did the ??? thing. Can you generate a log please http://wiki.lazarus.freepascal. org/GDB_Debugger_Tips#Log_info_for_debug_session The above sting in 4 versions: utf8 and ansi both for unlimited strings and string[x] And the source, so I can see what is what.

Sent log & source to .

Is your ansi a multibyte ansi? Maybe GDB does not understand that? (just a wild guess)

It is 1-byte Win-1251 charset, here you can see the details: http://commons.wikimedia.org/wiki/File:Ascii2.gif

_______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Martin
to Ernest V Miller
Ernest V Miller wrote
Sent log & source to .

Ok, I can see what happens.

Your GDB returns slightly different from mine (but correct). And it triggers the Code for utf8 correction.
That is why you actually see the proper utf8 string, while I never do: I always get the #123#124 representation.

*** ???

The IDE does not know, if the text is not utf8, so if it translates ansi #234 as if it was utf8, then it ends up with invalid utf8, and that fails to be displayed.

It is possible to always suppress that translation. But then you never see utf8, you always see #123#124.

Other then making it a user set-able decision in the properties, there is little that can be done.
It could do a heuristic, checking if the result has such invalid chars, and if there is one then do all as #123. But an ascii sting may be a valid utf8 string sometimes, yet the utf8 would map to entirely different chars. In this case the heuristic would show a utf8 without warning that the content is wrong (well it already does/would do)

*** #123#124
That is actually a missing call to the clean-up, that makes the utf8 read-able. But if I add it, then it will garble the ansi in string[5] to ???

------------------
It may be a while until I get to it.

I have not tested any of then, but if you want to look at it, for some kind of temporary fix, you can look at either of the following (all in debugger\gdbmidebugger.pp ):

Doing 1 or 2 may interfere with displaying the classname of an exception, maybe other class name stuff (detection of real class for "Sender: TObject").

Doing the test for invalid utf8, will avoid this, except where the class name is longer than 127 chars. Because a classname sometime is returned as first byte= length, then the name. And a length of 128 or more, will be invalid utf8. Bun needs to be returned as the caller will fix it.

1) line 11000
function TGDBMIDebuggerCommand.GetText(const AExpression: String;

   const AValues: array of const): String;

....
if not ExecuteCommand('x/s ' + AExpression, AValues, R, [], DebuggerProperties.TimeoutForEval)
...

   Result := ProcessGDBResultText(StripLN(R.Values));

end;

last line, remove "ProcessGDBResultText", so the line will be
   Result := StripLN(R.Values);

That will always show #123, and no longer do utf8

You can call it first, then test for faulty utf8 (there is a function, but not sure of the name), and if so assign the none-fixed value

2) line 10800

    function TGDBMIDebuggerCommand.ProcessGDBResultText(S: String): String;

Do the same fix in here. Note this may also be called when getting float values

3) line 12609

   procedure FixUpResult(AnExpression: string; ResultInfo: TGDBType = nil);

...

     case ResultInfo.Kind of

...
0, 1, 2: begin // 'char', 'character', 'ansistring' ...
then

               FTextValue := copy(FTextValue, i+2, length(FTextValue) -

i - 1)
else

here you can add utf8 translation (and again add the utf8 test, if you like)
             then
begin
               FTextValue := copy(FTextValue, i+2, length(FTextValue) -
i - 1)
   FTextValue := MakePrintable(ProcessGDBResultText(   '\t'  + FTextValue));
end
             else

Reply by email

Mattias Gaertner
to Martin: Ok, I can see what happens. Your GDB returns slightly different from mine (but correct).…
Show quoted text 6 lines
[...] It could do a heuristic, checking if the result has such invalid chars, and if there is one then do all as #123. But an ascii sting may be a valid utf8 string sometimes, yet the utf8 would map to entirely different chars. In this case the heuristic would show a utf8 without warning that the content is wrong (well it already does/would do)

How likely is this case?

Mattias

Reply by email

Martin
to Mattias Gaertner
Mattias Gaertner wrote 11 lines
On Mon, 01 Apr 2013 10:13:22 +0100 Martin wrote: > [...] > It could do a heuristic, checking if the result has such invalid chars, > and if there is one then do all as #123. But an ascii sting may be a > valid utf8 string sometimes, yet the utf8 would map to entirely > different chars. In this case the heuristic would show a utf8 without > warning that the content is wrong (well it already does/would do) How likely is this case?

This wasn't a case against it, as more a general observation... And it is already happening.

Reply by email

Mattias Gaertner
to Martin
Show quoted text 15 lines
On 01/04/2013 10:53, Mattias Gaertner wrote: > On Mon, 01 Apr 2013 10:13:22 +0100 > Martin wrote: > >> [...] >> It could do a heuristic, checking if the result has such invalid chars, >> and if there is one then do all as #123. But an ascii sting may be a >> valid utf8 string sometimes, yet the utf8 would map to entirely >> different chars. In this case the heuristic would show a utf8 without >> warning that the content is wrong (well it already does/would do) > How likely is this case? > This wasn't a case against it, as more a general observation... And it is already happening.

Do you mean, there were already real cases where a string had bytes #191..#254 and is valid UTF-8, but was windows codepage?

Mattias

Reply by email

Martin
to Mattias Gaertner
Mattias Gaertner wrote 20 lines
On Mon, 01 Apr 2013 11:03:31 +0100 Martin wrote: > On 01/04/2013 10:53, Mattias Gaertner wrote: >> On Mon, 01 Apr 2013 10:13:22 +0100 >> Martin wrote: >> >>> [...] >>> It could do a heuristic, checking if the result has such invalid chars, >>> and if there is one then do all as #123. But an ascii sting may be a >>> valid utf8 string sometimes, yet the utf8 would map to entirely >>> different chars. In this case the heuristic would show a utf8 without >>> warning that the content is wrong (well it already does/would do) >> How likely is this case? >> > This wasn't a case against it, as more a general observation... And it > is already happening. Do you mean, there were already real cases where a string had bytes #191..#254 and is valid UTF-8, but was windows codepage?

How can I know? I don't know every string anyone has debugged until now?

All I say is, it is possible, that a string in ANSI, (with bytes > 128) may at the same time be a utf8 string. Probably not read able text in both ansi and utf8. But maybe random pass generator. Anyway it does not matter.

What matters more:
- verify what else may be affected, and ensure it does not break
- decide what to do with none utf8:
* assume current codepage, and convert (probably not a good idea)
* display all chars >= 128 as #nnn
* assume still utf8, and only display broken chars as #nnn
- add tests (so It can be verified for a wide range of gdb versions)

Reply by email

Mattias Gaertner
to Martin
Show quoted text 23 lines
On 01/04/2013 11:18, Mattias Gaertner wrote: > On Mon, 01 Apr 2013 11:03:31 +0100 > Martin wrote: > >> On 01/04/2013 10:53, Mattias Gaertner wrote: >>> On Mon, 01 Apr 2013 10:13:22 +0100 >>> Martin wrote: >>> >>>> [...] >>>> It could do a heuristic, checking if the result has such invalid chars, >>>> and if there is one then do all as #123. But an ascii sting may be a >>>> valid utf8 string sometimes, yet the utf8 would map to entirely >>>> different chars. In this case the heuristic would show a utf8 without >>>> warning that the content is wrong (well it already does/would do) >>> How likely is this case? >>> >> This wasn't a case against it, as more a general observation... And it >> is already happening. > Do you mean, there were already real cases where a string had bytes > #191..#254 and is valid UTF-8, but was windows codepage? > How can I know? I don't know every string anyone has debugged until now?

With real cases I meant cases you know of.

Show quoted text 13 lines
All I say is, it is possible, that a string in ANSI, (with bytes > 128) may at the same time be a utf8 string. Probably not read able text in both ansi and utf8. But maybe random pass generator. Anyway it does not matter. What matters more: - verify what else may be affected, and ensure it does not break - decide what to do with none utf8: * assume current codepage, and convert (probably not a good idea) * display all chars >= 128 as #nnn * assume still utf8, and only display broken chars as #nnn - add tests (so It can be verified for a wide range of gdb versions)

I'm sure you are already thinking about making this optional. :)

Mattias

Reply by email

Martin
to Ernest V Miller: Hello! As we know, Lazarus is UTF8 friendly. When I want to see a value of a string var…
Ernest V Miller wrote 5 lines
I see value of utf8 in russian. And I am absolutely happy. But as soon as I want to use fixed strings, the only way for me to see my native language when debugging is to declare an additional string var as you can see below:

I committed a partial fix for this.
http://bugs.freepascal.org/view.php?id=24380#c67731

It will later be merged to Lazarus fixes too.

Reply by email

Ernest V Miller
to Martin

I'll bend my knees to fair gods for your health!

Martin ???????? 17.05.2013 18:51:53:

On 27/03/2013 04:08, Ernest V Miller wrote: > I see value of utf8 in russian. And I am absolutely happy. But as soon

as I

> want to use fixed strings, the only way for me to see my native

language

> when debugging is to declare an additional string var as you can see

below:

Show quoted text 12 lines
> > I committed a partial fix for this. http://bugs.freepascal.org/view.php?id=24380#c67731 It will later be merged to Lazarus fixes too. -- _______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email