AFAIK, Delphi XE features a second parameter "encoding" with TStringList.LoadFromFile.

Lazarus / fpc misses same.

Bug or Feature ?

-Michael

Known missing feature, there exists a bug for it.

Main problem is that this necessitates a unicode stringlist, and by extension a unicode RTL. We're working on it.

Michael.

Why do you need a unicode stringlist? I wrote a class helper for myself that does it.

Ondrej

TStringList.LoadFromFile.

that does it.

I'm wondering that as well. After all we'll need that functionality for the ANSI RTLs as well...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20160709/5670c568/attachment.html>

If the characters in the file encoding cannot be preserved in the current encoding for 'ansistring', then what ?

Michael.

Michael Van Canneyt wrote

If the characters in the file encoding cannot be preserved in the current encoding for 'ansistring', then what ?

Is there even something like "current encoding for 'ansistring'" in FPC 3.0? AFAIK FPC 3.0 AnsiString can handle all encodings and convert between them internally. By default TEncoding output could be set to AnsiString(CP_UTF8). Conversion to other codepages of AnsiString is done by the compiler.

I see now that FPC's TEncoding uses UnicodeString and PUnicodeChar. For me it's a strange decision. I would expect it to be String ( http://wiki.freepascal.org/Character_and_string_types#String ) and use UTF8 since AnsiString(CP_UTF8) is compatible to String.

Ondrej

OK.

So what is the encoding argument supposed to do ?
- Convert to current codepage (as set in
http://www.freepascal.org/docs-html/rtl/system/defaultsystemcodepage.html ) with possible loss of characters during read ?
- Keep all strings in the codepage that was passed on ?

AFAIK It is Delphi compatible, and there it is unicodestring (well, widestring).

Michael.

Actually "UnicodeString". "WideString" is something other on Windows. I suggest that if we use the the word "widestring" in context of Free Pascal we explain that we mean the type "UnicodeString". One can often read "WideString" instead of "UnicodeString" in Free Pascal Unicode example code which is not optimal.
We always can write that "UnicodeString" is the wrong name for a reference counted utf-16 string because UTF8String or AnsiString with default code page set to utf-8 also is Unicode in order to express our anger about the bad marketing driven decision of the Delphi owners. ;-)

Martin

I see now. You need UnicodeString support for the delphiunicode mode. So now there are String/PChar overloads missing in TEncoding for FPC default ansi mode. Once you have String/PChar support in TEncoding, adding TEncoding support to ansi RTL is easy.

The best IMO is that the String-based TEncoding routines handle passed/returned strings in current DefaultSystemCodePage with possible character loss. If DefaultSystemCodePage is not UTF8 it will probably mean that 2 conversions have to be executed (SOURCE->UTFxxx->TARGET). If DefaultSystemCodePage is UTF8, one conversion must be enough. LazUtils have the LConvEncoding unit for UTF8CP conversions.

Ondrej

With Delphi, TStringList seems to be hard coded to use UTF16 with "Add" and StringList[i] (which is a really bad idea in a String system providing versatile encoding). TStringList.LoadFromFile seems do default to the system ANSI encoding (which seems like rather contra-intuitive in a system that defaults to coding strings as UTF16).

With Lazarus, TStringList seems to be (considered to be, as StrinCodePage(SomeString ) is 65001) hard coded to use UTF8 with "Add" and StringList[i] (which is a bad idea, as well, but not worse than Delphi forcing UTF16) TStringList.LoadFromFile uses (defaults to) "no conversion" which in fact means UTF8 encoding (which in fact is straight forward and obvious, and hence better than Delphi).

I don't see why a redesign of the "hard coded UF8" TStringList would be necessary just for setting the code of the file to be loaded by TStringList.LoadFromFile, just a settable conversion at that function (and the at SaveToFile) would be helpful. Of course a versatile encoding of TStrings would be a much better way to go, but I don't suppose this ever will happen (partly because Delphi does not feature his either).

The current makeup results in different behavior between Delphi and Lazarus.

I was hit by this when doing a (Windows) program that analysis text files that are generated in local (German) ANSI code.

To load a text file, the most convenient function to me seems to be TStringList.LoadFromFile and here (with the current official Windows distribution of Lazarus) the output of myString := StringList[i] obviously is wrong.

Test: Speed

There is no 'hard coded UTF8' stringlist.

This is an assumption made by Lazarus LCL, which - depending on the version of FPC - sets the default single-byte string codepage to UTF8.

TStringList is not part of Lazarus, and hence must cater for other codepages as well.

Michael.

Martin Schreiber wrote

We always can write that "UnicodeString" is the wrong name for a reference counted utf-16 string because UTF8String or AnsiString with default code page set to utf-8 also is Unicode in order to express our anger about the bad marketing driven decision of the Delphi owners.

G*d, I so agree with that too! I simply hate the name "UnicodeString" implicitly implying UTF-16 only. "Unicode" is an algorithm with 3 official encodings, not just UTF-16.

Then to boot, they introduced the AnsiString mess in FPC 3.0 - which now doesn't only mean ANSI encoding (contrary to what the name suggests), it now means Unicode encodings too. Other compiler projects must be laughing their asses off at Free Pascal now!

So what other types in Object Pascal can we f**k with next? Boolean, Double?

Regards,
Graeme

Of Course I do know all this and this is exactly what I tried to point out.

In fact TStringList (fpc) is hard coded to 8 Bit, and hence can hold any 8 bit encoding, including UTF-8 and the German ANSI Variant used for the file I try to load. and it does not do any Lazarus seems to set the compiler to by default handle strings as UTF-8 coded.

This obviously creates the said problem.

-Michael

Not the compiler, the RTL.

Michael.

They were not necessarily missing, they were perceived as 'not necessary'.

As far as I can see, they are still unnecessary (see below). One could add them for completeness or symmetry reasons, but I don't see the point.

The best IMO is that the String-based TEncoding routines handle passed/returned strings in current DefaultSystemCodePage with possible character loss.

I am not sure that this 'possible character loss' is a good idea.

But if the general consensus is that this is acceptable, then why not. If you already have a helper class, then this can probably be easily integrated in the TStrings class.

If DefaultSystemCodePage is not UTF8 it will probably mean that 2 conversions have to be executed (SOURCE->UTFxxx->TARGET). If DefaultSystemCodePage is UTF8, one conversion must be enough. LazUtils have the LConvEncoding unit for UTF8CP conversions.

This conversion should already be fully automatic if the widestring manager is used and the 'SetCodePage' function and friends are used.

One does not need TEncoding for that. TEncoding is just a wrapper around the widestring manager with some utility functions, implemented for Delphi compatibility.

Michael.

Yep.
Thanks for the clarification

-Michael

... by setting a system variable.

So simply avoiding to do this (and use the locale ANSI setting) would make Lazarus (user code) Unicode - unaware and hence avoid such problems with 99 % of the applications, while of course the 1 % of the applications that in fact do need Unicode handling would not work in that mode (but maybe could be configured to use a "Unicode mode", if they don't want to define their Unicode aware string variables explicitly ) ???

(Of course conversion to/from Unicode in the LCL-interface for GUI calls might cost some addition CPU cycles.)

-Michael

Michael Schnell wrote

So simply avoiding to do this (and use the locale ANSI setting) would make Lazarus (user code) Unicode - unaware and hence avoid such problems with 99 % of the applications, while of course the 1 % of the applications that in fact do need Unicode handling

100% of end-user Windows LCL applications need Unicode. You can't simply force your customers to set the German codepage in their OS settings.

Ondrej

You know well that the name UnicodeString came from Delphi, where it fits, because it is their only string supporting Unicode. No one forces you to use this name in your code. You can define your own alias type.

Then to boot, they introduced the AnsiString mess in FPC 3.0 - which now doesn't only mean ANSI encoding (contrary to what the name suggests), it now means Unicode encodings too.

1. AnsiString comes from Microsoft ANSI code pages, which was not an ANSI-standard at all, so the term "Ansi" was a misnomer from the beginning.
2. MS accepted that and nowadays calls them only "code pages". But many of their pages still use the term "ANSI code page".
3. The Unicode consortium added UTF-8 specially designed for legacy code using 8-bit strings.
4. Microsoft added the UTF-8 code page 65001 (and also code pages for UTF-16 and UTF-32), but no MS Windows used it as system code page.

FPC's AnsiString uses the MS code pages numbers, which includes UTF-8.

The new FPC 3.0 strings made it easier to use UTF-8 strings
- aka you need less conversions and more RTL functions support Unicode - while still keeping compatibility.

[...]

Mattias

Ondrej Pokorny wrote

100% of end-user Windows LCL applications need Unicode.

Can you explain why ?

For some 20 Years (before Delphi introduced Unicode) Pascal users were forced to use their locale ANSI instead of Unicode, and hence it was difficult to create programs that were useable in multiple regions. But AFAIR, the complaint about that fact have been rather limited.

You can't simply force your customers to set the German codepage in their OS settings.

1) not set "German codepage" but Lazarus using the "locale ANSI Code Page", as set in the OS (exactly as it had been before Lazarus moved to UTF-8 some years ago).

2 ) not force, but provide a setting if the user wants to use Unicode or not.

-Michael

Michael Schnell wrote

2 ) not force, but provide a setting if the user wants to use Unicode or not.

Define DisableUTF8RTL?
Save your sourcecode encoded as some Windows codepage (not UTF8). Good luck trying to open a file with e.g chinese characters in the filename then on your German machine.

Bart

Maybe from the FPC's POV they are not necessary but from LCL application's POV they definitely are. Because LCL applications use the String type, TEncoding has to be compatible with String as well. (Now it is only UnicodeString compatible.)

I am not sure that this 'possible character loss' is a good idea.

If you set DefaultSystemCodePage to UTF-8 there won't be any character loss. If you decide to write your FPC application for a national 8-bit codepage, it's your decision to have the character loss. In this case you can still use the UTF-16 based UnicodeString overloads. For me it makes perfect sense.

If you already have a helper class, then this can probably be easily integrated in the TStrings class.

It's based on my TEncoding class written years ago during FPC 2.x.x era: https://sourceforge.net/p/oxml/code/HEAD/tree/trunk/units/OEncoding.pas It uses only UTF-8 based strings and conversion functions from LConvEncoding. FPC needs its own solution because LConvEncoding is in LazUtils and it needs to support DefaultSystemCodePage and not UTF-8 only).

The main task is to have AnsiString support in TEncoding. The integration to TStrings is only a few lines of code (depending on how far you want to go with Delphi support).

Then it should be pretty easy to do it. Actually I made a first attempt. I attached a patch to http://mantis.freepascal.org/view.php?id=29848

It misses some functionality from the unicode version - particularly GetAnsiByteCount and GetAnsiCharCount. But IMO these are not really needed because actually you do
"GetAnsiByteCount():=Length(GetAnsiBytes())". BTW, the unicode versions do the same, which results in executing 2 conversions in TMBCSEncoding.GetString and .GetBytes, which I wanted to avoid in the ANSI versions.

Ondrej

Bart wrote

Good luck trying to open a file with e.g chinese characters in the filename then on your German machine.

I don't see any use for something like this with any of my current and planned applications.

Of course, if a programmer intends to allow for such, he should be free to activate Unicode and live with the complexity (or switch to Delphi and live with some different complexity).
-Michael

Lucky you.

Of course, if a programmer intends to allow for such, he should be free to activate Unicode and live with the complexity (or switch to Delphi and live with some different complexity).

Complexity is supporting Unicode, which can be switched off.

Mattias

Mattias Gaertner wrote

supporting Unicode, which can be switched off.

This is exactly what I asked but got no answer I could understand.

I understand that by an RTL variable I can set the code page new strings are constructed to use.

I understand that TStringList.LoadFromFile will work as expected (if the file to be loaded comes in the same encoding).

But this is FPC. Will Lazarus be able to cope with that ? Obviously the GUI interface in the LCL still will need to provide the correct encoding (supposedly UTF-8 with Linux and UTF16 with Windows) towards the external GUI infrastructure.

-Michael

On 07/12/2016 10:09 AM, Mattias Gaertner wrote: > supporting Unicode, which can be switched off. This is exactly what I asked but got no answer I could understand.

I meant that Lazarus supports both. UTF-8 by default and via -dDisableUTF8RTL the default.

Mattias