Hi,

hopefully someone can help me with my first steps at migrating my Delphi programs to Lazarus.

I have a TListBox which I modify in the following way:
-------------------------------------------
with Form1.ListBoxAusgaben do
    begin
    Items.Add('aaaaaaaaaaa');
    Items.Strings[Count-1] := 'xxxxxxxxxxx';
    Items.Add('bbbbbbbbbbb');
    Items.Strings[Count-1] := 'yyyyyyyyyyy';
    ItemIndex := Count-1;
    Repaint;
    end;
-------------------------------------------
The result is that I get 3 lines with the first line being 'xxxxxxxxxxx'
plus 2 empty lines. It seems that only the first ADD and the first
aasignment to Strings[Count-1] is done but all following changes do not
appear. Is this a bug in FP/Lazarus?

Jürgen Hestermann.

Maybe. Please create a bug report with a complete example.

Mattias

Hi,

I have analyzed the bug in the TListBox further and found that adding or modifying a string from to a TListBox fails if the strings contains foreign characters like german umlauts as in

Form1.ListBox1.Items.Add('ööööööööööööö');
Form1.ListBox1.Items.Strings[Count-1] := ('ööööööööööööö');

but it works if changed to UTF8 as in

Form1.ListBox1.Items.Add(AnsiToUTF8('ööööööööööööö'));
Form1.ListBox1.Items.Strings[Count-1] := (AnsiToUTF8('ööööööööööööö'));

How is this supposed to work? Even if the wrong coding is used I don't understand why the whole string is dropped when not using AnsiToUTF8. Without the conversion I would expect to see incorrect characters instead of the umlauts.

Jürgen Hestermann.

Hello Jürgen,

Saturday, December 20, 2008, 6:46:06 PM, you wrote:

JH> Form1.ListBox1.Items.Add(AnsiToUTF8('ööööööööööööö')); JH> Form1.ListBox1.Items.Strings[Count-1] := (AnsiToUTF8('ööööööööööööö')); JH> How is this supposed to work? Even if the wrong coding is used I don't JH> understand why the whole string is dropped when not using AnsiToUTF8. JH> Without the conversion I would expect to see incorrect characters JH> instead of the umlauts.

The UTF8Decode routine kills any string with malformed UTF8 sequences.

http://bugs.freepascal.org/view.php?id=11791

The UTF8Decode routine kills any string with malformed UTF8 sequences. http://bugs.freepascal.org/view.php?id=11791

Thanks for that hint.

I am a bit astonished that this bug has survived 2 updates (was reported for 2.2.0 and still exists in 2.2.2).

I also didn't find any information that the strings are forced to be UTF8. And how do I insert constant strings in UTF8? Do I realy have to look up all sequences when using extented characters?

It took me many hours to find out that the stings were missing only when having german umlauts in them. Not a very reliable programming environment for serious programming if I invest more time in debugging the LCL than in debugging my own programs.

Jürgen Hestermann.

The UTF8Decode routine kills any string with malformed UTF8 sequences. http://bugs.freepascal.org/view.php?id=11791

In the bug report it says that Delphi 7 behaves the same but this is not true. I ported my program from Delphi 7 (where it ran without problem) to FPC. So I am not sure whether we are talking about the same bug here.

Jürgen Hestermann.

You don't need the AnsiToUTF8 if you encode your source file as UTF-8 without a BOM.

The difference from Delphi is caused by Lazarus supporting unicode and also some limitations of Free Pascal, but nothing which prevents the correct development of unicode apps if you follow the documentation:

http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

Jürgen Hestermann wrote

In the bug report it says that Delphi 7 behaves the same but this is not true. I ported my program from Delphi 7 (where it ran without problem) to FPC. So I am not sure whether we are talking about the same bug here.

Delphi 7 doesn't support unicode, so I doubt you were using unicode routines there. Isn't that correct?

The bug is about a unicode routine.

You don't need the AnsiToUTF8 if you encode your source file as UTF-8 without a BOM.

Well, I installed and started Lazarus in the standard way so why isn't it using UTF-8 by default if that's the only way to store strings? Why let the user step into this trap? And how do I change my current code to UTF-8?

The difference from Delphi is caused by Lazarus supporting unicode and also some limitations of Free Pascal, but nothing which prevents the correct development of unicode apps if you follow the documentation: http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

Thanks for the link. But why isn't it documented in the Help of TListBox? No word there that the strings may be UTF-8 only (and that you somehow have to store your code files in that coding if you use stings constants).

Delphi 7 doesn't support unicode, so I doubt you were using unicode routines there. Isn't that correct?

Yes, of course that's correct. That's the reason I didn't use Unicode for Lazarus either. But that failed. So I had a hard time to find out myself that I have to use AnsiToUTF8 for each and every constant string I use. A quite distracting first contact with Lazarus/FPC.

Jürgen Hestermann.

Well, I installed and started Lazarus in the standard way so why isn't it using UTF-8 by default if that's the only way to store strings? Why let the user step into this trap? And how do I change my current code to UTF-8?

In the menu Tools --> Convert encoding of file/project

But I never used it. I would just open in a text editor which can handle multiple encodings, like notepad++

Thanks for the link. But why isn't it documented in the Help of TListBox? No word there that the strings may be UTF-8 only (and that you somehow have to store your code files in that coding if you use stings constants).

Because it applies to all of LCL. We would need to repeat that for all classes from the LCL.

Yes, of course that's correct. That's the reason I didn't use Unicode for Lazarus either.

You cannot not use Unicode with Lazarus. Only UTF-8 is supported, which convers all languages, so it's enougth =)

A quite distracting first contact with Lazarus/FPC.

There is obviously some work to be done when migrating to Unicode, but you will be very happy if you ever need to localize your application to Russian or chinese, because the application is already unicode capable and can easely support those languages.

> And how do I change my current code to UTF-8? In the menu Tools --> Convert encoding of file/project

I used the Delphi-to-Lazarus convertion for my (Delphi) program. Why isn't it also converting to UTF-8 if that's necessary?

But I never used it. I would just open in a text editor which can handle multiple encodings, like notepad++

You mean: Don't use Lazarus but use an alternative standard editor? What purpose is the IDE then?

In a documentation I would expect to find such important information (at least as a link). Otherwise I don't need it at all and can find out how it works myself by trial and error.

You cannot not use Unicode with Lazarus. Only UTF-8 is supported,

Isn't UTF-8 unicode in the same way as UTF-16?

There is obviously some work to be done when migrating to Unicode, but you will be very happy if you ever need to localize your application to Russian or chinese, because the application is already unicode capable and can easely support those languages.

Yes. I don't have a problem with Unicode at all, just the opposite. I only miss a clear help that tells me that TStrings in the LCL have to be UTF-8 per definition. And it would have saved me lots of debugging hours if the strange bug would not have omitted the whole string whenever they contain extended characters. It took me a very long time to realize that it's a bug in the LCL and that it's related to extended characters and that it can be avoided by using AnsiToUTF8.

Jürgen Hestermann.

Maybe we should consider removing the convert from the IDE, if it misleads more than helps.

Vincent

In a documentation I would expect to find such important information (at least as a link). Otherwise I don't need it at all and can find out how it works myself by trial and error.

The wiki is the most important part of the documentation. The other part being the reference of classes/units/etc

If you coming from Delphi you should have read:
http://wiki.lazarus.freepascal.org/Lazarus_For_Delphi_Users

And also anything else necessary in:
http://wiki.lazarus.freepascal.org/Lazarus_Documentation

If you think this document needs improvements, then you can edit it.

And your tone isn't very helpful. I responded only in a intention to help, and yet the response doesn't seam to have been apreciated.

And your tone isn't very helpful. I responded only in a intention to help, and yet the response doesn't seam to have been apreciated.

Forgive me if I am sounding a bit rude. Of course that's not meant against any person on the list. Every help is realy appreciated. I only was already frustrated about Delphi (7) which has lots of bugs and a poor help too. Then I read many good things about FPC/Lazarus so I was expecting it to be at least better than Delphi but found that it's even worse. I am not sure anymore whether I should migrate to FPC/Lazarus. Transfering my Delphi code may be more work than I thought and I am not sure whether it's worth it if I am having even more problems afterwards.

Jürgen Hestermann.

> I used the Delphi-to-Lazarus convertion for my (Delphi) program. Why > isn't it also converting to UTF-8 if that's necessary? Maybe we should consider removing the convert from the IDE, if it misleads more than helps.

I don't think so, it's better than nothing. The problem is, that there are more pitfalls than the conversion routine handles (can handle?).

I just converted to UTF-8 but now got problems with constant strings in MessageBox calls which require Ansi instead of UTF-8. It's quite awkward if different codings are used within one source code file. I would have expected that one single encoding (UTF-8?) is used throughout all source code and all routines that need a different encoding do the conversion themselve. Now each programmer has to learn for each and every routine what encoding is used (and that's not even documented within the help for these routines).

Why confront the programmer with more complexity than needed? Especially those who start learning programming from scratch realy have a hard time if such (IMO unnecessary) things are not avoided.

Jürgen Hestermann.

Yes, and until that is fixed, it may be better to remove it, so we don't get such long threads and to spare you debug time.

I think it should be simple: LCL UTF8, RTL Ansi.

Where is this messagebox call defined?

Vicnet

I just converted to UTF-8 but now got problems with constant strings in MessageBox calls which require Ansi instead of UTF-8.

If you are using MessageBox from the LCL, then it uses UTF-8.

Maybe your are using a Windows API.

But how would that avoid the confusion? Just the opposite. It would add even more because the conversion avoids at least some of the problems.

The main problem realy is, that (some!) stings are completely omitted when they are Ansi where UTF-8 is needed. If I do it the other way round (convert my code to UTF-8 and use these strings for RTL routines) I immediately see that something is wrong with the encoding ("funny" characters are displayed). But when nothing is displayed I have no idea why that happens.

I think it should be simple: LCL UTF8, RTL Ansi. Where is this messagebox call defined?

That's the correct question. How do I know which routine is from where? And why am I forced to think about such things when programming? In Delphi I am not.

Jürgen Hestermann.

If you are using MessageBox from the LCL, then it uses UTF-8. Maybe your are using a Windows API.

For example, I am just writing

MessageBox(0,'Programmfehler!','SPEED TEST',MB_ICONEXCLAMATION or
                    MB_TopMost or MB_SetForeGround);

the interface part has the following "uses" entries

   LCLIntf, Messages, SysUtils, Classes, Graphics, Controls, Forms,
   Dialogs, StdCtrls, LResources;

and the implementation part uses

Windows,FileCtrl,DateUtils,clipbrd,interfaces;

So what am I using and how do I know what I should use?

Jürgen Hestermann.

Well, they cannot put the blaim on the Lazarus converter, but only on their own ignorance with respect to porting to Lazarus. I have a feeling that the conversion tool is bad publicity for Lazarus that it gives people a false feeling of easy conversion. Lazarus should avoid giving this impression.

The main problem realy is, that (some!) stings are completely omitted when they are Ansi where UTF-8 is needed.

Yes, garbage in garbage out.

If I do it the other way round

Because in Delphi RTL and VCL are unified. But Lazarus programming may be too complicated ...

Do Find Declaration and see what the source directory is. Or press F1 a see what the package is (top left, the string after the #).

Vincent

Jürgen Hestermann wrote

MessageBox(0,'Programmfehler!','SPEED TEST',MB_ICONEXCLAMATION or MB_TopMost or MB_SetForeGround);

You are using a Windows API routine, from the Windows unit. I know that because I know the routines, but it is easy to find out. Just right click "MessageBox" and select "Find Declaration"

The "Find in Files" dialog is also useful.

Well, they cannot put the blaim on the Lazarus converter, but only on their own ignorance with respect to porting to Lazarus.

Why ignorance? I remember someone asking here how to learn programming with FPC/Lazarus and someone else answered that each Delphi learning book would do. How does that fit together when such huge differences between Lazarus and Delphi exist?

I have a feeling that the conversion tool is bad publicity for Lazarus that it gives people a false feeling of easy conversion. Lazarus should avoid giving this impression.

A much better way would be to improve the converter in that it gives hints about the different coding of strings. This would be enough.

> The main problem realy is, that (some!) stings are completely omitted > when they are Ansi where UTF-8 is needed. Yes, garbage in garbage out.

Why "garbage"? A correct and running Delphi program was converted to Lazarus with the converter, how can it be garbage?

Jürgen Hestermann.

About unicode:
The LCL uses one unicode format: UTF8.
The problem is that many other libraries (e.g. winapi) use other string formats.
Delphi Tiburon tried to solve this problem with compiler magic by implicit conversions, which helps to quickly port old Delphi code, but might kill your speed or have some unexpected and hidden side effects.
The FPC team is working on something similar, although because of the many platforms, the problem is a magnitude more difficult. There were very long mail threads about this topic.

About the converter:
The converter is a simple tool that already saves a lot of work, but I agree with Vincent that it is easily mistaken as a complete tool like converting a word document to openoffice.
Basically it handles the filenames, uses sections and compiler options. And it does not yet do any syntax/library conversion. The tool should warn newbies and open the wiki pages. So there is a lot of room for improvements.

Mattias

I am glad I did not give that advice.

Yes, that would be great. Can we expect some improvements (patches) from you soon. Until now, very few people have contributed to it. Improving it would be nice, but people who have Delphi code rather stay with Delphi, and that might not be unwise.

You feed ansi encoded strings where UTF-8 is needed. That is garbage. Don't trust the converter too much.

Vincent

Can we expect some improvements (patches) from you soon.

I would like to but as I am new with Lazarus my knowledge about it is not sufficient to give advices (as you can see ;-)). More of such pitfalls may lurk somewhere which I don't know about so I cannot give advices about it.

people who have Delphi code rather stay with Delphi, and that might not be unwise.

But people using Delphi are the most likely to be converted to FPC/Lazarus so it would not be wise to reject them. Otherwise this project may lack of contributors even more.

You feed ansi encoded strings where UTF-8 is needed. That is garbage. Don't trust the converter too much.

That's like saying: Don't trust FPC/Lazarus too much. How can a new user know which parts of Lazarus can be "trusted" (whatever this is) and which not.

Wouldn't it be easy to do the UTF-8 conversion within the converter too? Everybody knows that Delphi uses Ansi and everybody (except newbies) knows that Lazarus uses UTF-8. So staying with Ansi provokes such incompatibleness.

99% of my problems with this issue would have been avoided if not the whole string is dropped in case that a string is not conform to UFT-8. That doesn't make any sense. If it's such an error why not raise an exception instead of silently doing nothing? But it would be better to just drop malformed characters but not the whole string so a user would immediately see that something is wrong with the coding.

Jürgen Hestermann.

No, not advices. Updates (source code) to improve the current behavior.

I don't reject them. But people not using Delphi have very little to gain form a converter from Delphi. It is in their interest to focus on other things, like docking, better code tools/refactoring, code formatter, more documentation, off line help (local wiki and identifier help) etc. So you should not expect people not using Delphi to work on the convert.

Correctly. That is why I suggested to remove the converter altogether. You cannot know, it is not yet fool proof, and if you don't have it, you don't use it. OTOH, if you work on improving it then it is good to include it, so maybe it should just be an easily installed IDE package.

Vincent

Op maandag 22-12-2008 om 10:08 uur [tijdzone +0100], schreef Jürgen Hestermann:

iirc it's windows that does this. If the winapi encounters characters in a string for a messagebox it doesn't understand, it drops the whole string. There's nothing Lazarus can do about it.

Joost

iirc it's windows that does this. If the winapi encounters characters in a string for a messagebox it doesn't understand, it drops the whole string. There's nothing Lazarus can do about it.

It's not the MessageBox function that drops strings (they always appear). It's the LCL that does this (as far as I read from the bug report) for TStrings that are not UTF-8 conform.

Jürgen Hestermann.

Hello Joost,

Monday, December 22, 2008, 11:40:02 AM, you wrote:

JvdS> iirc it's windows that does this. If the winapi encounters characters in JvdS> a string for a messagebox it doesn't understand, it drops the whole JvdS> string. There's nothing Lazarus can do about it.

It's the UTF8ToUnicode provided by fpc which is more or less the same as the Delphi one, kills the output on any error, and allow other errors to be passed away to the result string which will bypass some security functions.