Hi,

I deal with in Japanese (and sometimes other languages) in a lot of my programs, and nothing I do seems to work consistently on Windows systems. (OS X is no problem).

I have followed steps in the Wiki, etc., but to little avail, so I have some questions for anyone who knows more than me:
1. What encoding "should" I be writing to the terminal? from experimenting with text files using the cat command in powershell, it seems that local ("ANSI") encoding should be used. This makes sense since older versions of windows only supported local encodings.
2. Is there any reason why writing out data in the local encoding (with write statements, etc.) should get corrupted? For example is some level of the RTL assuming something about the encoding? (I don't think so, but...)
3. Is there a way to set the output to UTF8 so I can just write out UTF8 and be done with it?

Just to give an example:
1. I read in an SJIS CSV file, and display it on the screen, and it's corrupted.
2. I convert it to UTF8 before displaying it, and it's still corrupted. 3.I cat the file to the screen and it's ok.
4. I write the output to a file instead of the console, and it's ok.

Something seems odd.

Does anyone else have these issues?

Thank you,
Noah Silva

p.s.: I know that the actual data in my programs isn't broken, because if I write it to a file or database, there is no problem with corruption.

Noah Silva wrote

1. What encoding "should" I be writing to the terminal?

The console code page. You can get it using the following function:
   function GetConsoleOutputCP : UINT; stdcall; external 'kernel32'
name 'GetConsoleOutputCP';

The ansi code page is (or at least can be) different, that's the the result of

   function GetACP:UINT; stdcall; external 'kernel32' name 'GetACP';

Not in 2.6.x. In 2.7.x, every ansistring is tagged with a code page (the ansi code page by default) and the RTL will convert it to the console code page before writing it.

3. Is there a way to set the output to UTF8 so I can just write out UTF8 and be done with it?

I don't know.

Jonas

Yes.

See:
http://bugs.freepascal.org/view.php?id=22461
for info on the situation in FPC trunk.

Jonas Maebe wrote

On 09 Jul 2013, at 11:02, Noah Silva wrote:

.
.

There is code page 65001 for UTF-8, however I don't know since which version of MS Windows it is supported (and to which extent - e.g. including the console output). You may change the output code page for console using the Windows API call SetConsoleOutputCP, but e.g. our unit Crt currently resets the code page used for output to console to the "main" process code page (as returned by GetACP) on each write and thus ignores the user selection - this will probably change in trunk in the future as part of the Unicode support activities, but the current state is like this.

Tomas

In our previous episode, Tomas Hajny said:

Afaik Windows XP, but there might be font problems on console.

Keep in mind he is talking powershell, not cmd.exe

Noah Silva wrote

I convert it to UTF8 before displaying it....

Not a good idea.

The FPC developers are right now busy implementing the new Delphi Strings. This _could_ mean that the application programmer can use any encoding (such as multiple different ANSI byte-codes, UTF-8, UTF-16, ...), but in fact to be 100% Delphi compatible ("nothing less, nothing more"), it seems that only UTF-16 will gain full decent support (e.g. class inheritance, in TStringList, the Lazarus user API etc.)

-Michael

In our previous episode, Michael Schnell said:

Well, the main reason is not FPC, but Windows. UTF8 in Windows on _API_ level is simply not a good idea.

I have tried some programs/batchfiles that do chcp 65001 in a cmd.exe, but that doesn't really work nice. E.g. the "dir" command might simply fail afterwards, and some foreign chars don't work properly either (some do though!?!?)

I mostly rewrote them to delphi console apps (the delphi RTL does everything over -W functions).

Please state the windows version you are using. XP or Windows 7? I deal with chinese in my programs so I know your problems. The same delphi 5 program works differently on XP and Windows 7. Looks like Windows 7 has removed support for non unicode (I am not sure whether the Unicode it uses is UTF8, UTF16 or UTF32).

Seems that all filenames in XP are treated as unicode code. If you type a non unicode file name in Explorer, it will be auto converted to unicode. Also, in XP, when text is copied to MS Office from other programs and vice versa, XP seems to do an automatic conversion to UTF-8 and vice versa. That is if you copy some text in your program which is encoding in SJIS and paste it to Word, the text seems to be auto converted to unicode.

As for file handling, it seems some programs will read the first 2 bytes of the text file to determine the the encoding of the file. Google about it.

Sorry, I don't have exact answers to your questions. Just share some of my experience.

Dennis

Marco van de Voort wrote

Well, the main reason is not FPC, but Windows. UTF8 in Windows on _API_ level is simply not a good idea

You are absolutely right.

Here, in theory the Lazarus LCL and the FPC RTL could help by by providing an application programmer API that does not force a dedicated encoding. But to do so without the necessity to provide multiple OS depending versions of the LCL and RTL interface definition, it would need the compiler to provide support this.

In a perfect world the application programmer could use the encoding he thinks is appropriate for the internals of his busyness logic and the programming system does the correct things do use the API of the OS the program is compiled for (while for optimum performance do code conversions only when necessary).

-Michael

In our previous episode, Dennis Poon said:

Please state the windows version you are using. XP or Windows 7?

In XP there were separate Far East versions of Windows. In Vista+ this was wholly integrated and there is only one Vista base system with various language packs.

It might be that some of the Far East apis were deprecated later, but I would expect them to run if you configure the backwards compatibility options on the shortcut.

I deal with chinese in my programs so I know your problems. The same delphi 5 program works differently on XP and Windows 7. Looks like Windows 7 has removed support for non unicode (I am not sure whether the Unicode it uses is UTF8, UTF16 or UTF32).

UTF16 and a bit of UTF8. (e.g. notepad groks utf8 now).

Seems that all filenames in XP are treated as unicode code.

All NT derived systems are Unicode (UCS2, later UTF16) in the heart. The whole ansi support is a win32 compatibility layer.

I also thought about mentioning True Type fonts (like Lucida) being a better option in this case, but then I assumed that with Japanese, this might be necessary anyway.

I have no experience with powershell yet, but as long as it uses a console window (albeit designed differently), the launched applications should behave the same way, shouldn't they?

Tomas

In our previous episode, Tomas Hajny said:

Dos compatibility has waned somewhat. I would assume a new console type wouldn't be OEM. And probably even the 8-bit api would be a compatibility layer over the underlaying 16-bit api.

True. However, this probably changes the default value of console output CP, but whopefully should not change the principle.

And probably even the 8-bit api would be a compatibility layer over the underlaying 16-bit api.

Isn't this already the case of the original console anyway?

Tomas

Tomas Hajny wrote

> Afaik Windows XP, but there might be font problems on console. I also thought about mentioning True Type fonts (like Lucida) being a

It's not so much about True Type fonts, but about enabling a Unicode font in the console. There are many True Type fonts that only support the ASCII character-set.

Regards,
G.

Sure, but you can't select just any True Type fonts for console windows (only Lucida is offered in WinXP; I just don't know if the same font is offered if the main locale is Japanese or if it isn't another font in later MS Windows versions).

Regarding True Type fonts only supporting ASCII (and disregarding the fact that those wouldn't support Japanese mentioned in the original post anyway) - I meant using an appropriate True Type font as a minimum condition (i.e. not necessarily sufficient per se).

Tomas

Tomas Hajny wrote

Sure, but you can't select just any True Type fonts for console windows (only Lucida is offered in WinXP;

And is so since Win95. Just amazing that Windows Console is so far behind other platforms - it's rather ridiculous if you think about it.

If I used Windows, I would install a different console application — there must be better after-market ones out there. But yes, this probably doesn't solve your or your clients problems.

Anyway, here is some background information on Microsoft's terrible Console Window.

Why are console windows limited to Lucida Console and raster fonts?

http://blogs.msdn.com/b/oldnewthing/archive/2007/05/16/2659903.aspx

Here is how you can add more fonts — but you are on your own (the words from Microsoft):

http://support.microsoft.com/kb/247815

Regards,
G.

I seriously doubt it...

I mean, in the olden days, the cmd program was actually a fully/partially [delete inappropriate] DOS compatible virtual machine, starting with : being able to run real x86 mode programs... It wasn't a shell as *nix has, never was meant to be...

You'd have more luck using DosBOX emulator these days...

-L;)

Hi,

First, I want thank everyone for taking the time to answer!

2013/7/9 Jonas Maebe

Japanese system, it has to either be some form of Unicode or SJIS. (And it almost has to be SJIS since older windows versions didn't support Unicode!).

Again I will check this today, but see above.

Hmm, so at any rate, I need to support two code paths:
1. Convert it myself if on 2.6
2. Don't convert it if on 2.7

I use UTF8String for everything - I know that it is still just an ANSIString on FPC - but is the codepage set to UTF8 automatically?

told me before. Unfortunately, it didn't work, and had bad side-effects (the output window flickers and gets very slow).

Jonas

Thank you,
Noah Silva

______________________________**_________________ fpc-pascal maillist - http://lists.freepascal.org/**mailman/listinfo/fpc-pascal<http://lists.freepascal.org/mailman/listinfo/fpc-pascal>

Hi,

Thank you very much for the link.

I think it will take some experimenting to get everything to work how it should on my systems.

I am not using the {$codepage utf8} in my files, but I don't usually have any non-ASCII text in the programs. (The source files are in UTF8 though). The data I am loading is loaded from database, text files, etc. I am sure it(s not corrupted in memory, because I can write it back out to text files, store it in the database, or even view it in the debugger.

The SetColsoleOutputCP is probably what I tried before that caused the Windows Console to act strange.

What's worse, adding UTF8ToANSI not only didn't fix the problem on Windows, it broke MacOS (because console on OS X is always UTF8, but ANSI encoding still exists, so it(s not a noop).

thank you,
Noah Silva

2013/7/9 Reinier Olislagers

Hi,

Some comments...

2013/7/9 Marco van de Voort

I have tried in both. I tend to use PowerShell more often because I have this habit that goes like this:

cmd.exe

cd SourceCode ls Err: Command Not Found

(Swear words)

dir

Using powershell saves me a little sanity since I am used to Unix Systems and I can't install Cygwin everywhere.

At any rate, I assume that since the console is a separate thing from the program running in it that powershell and cmd would be the same from the point of view of a program executed from them. Does powershell use a different console?

Thank you,
Noah Silva

_______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

In our previous episode, Noah Silva said:

I don't know. But I wouldn't assume it is the same without testing.

Hi,

2013/7/9 Marco van de Voort

variable length (multi byte) encoding that works similarly to UTF8 but only supports local characters. For the most part, none of the system APIs cared what was in a string, they just plastered it somewhere. Since 1 byte 1 char, but normal code didn't know that, you were safe with things like Concat, but couldn't safely do things like copy(string,3,4) and expect it to work. That's still a problem with FreePascal, but in general, things like Copy() can work with all character sets not only because of unicode. Anything fancy that had to be done before was almost always at the application layer.

Anyway, if Windows changed the encoding from "local" to UTF8 suddenly, every single Japanese program that outputs SJIS to the command-line would break. In the GUI it is a little different because the system "knows" if the programs are Unicode or non-Unicode, and there is a setting as to what encoding to use for non-unicode apps.

This is definitely not the case, plenty of non-unicode programs work on Windows 7. They would face a huge revolt.

UTF16 and a bit of UTF8. (e.g. notepad groks utf8 now). Finally, not if only Excel could save UTF8 or handle UTV8 CSV files...

user space.
You click on a CSV file and Excel needs it to be SJIS (or whatever), even though I am sure modern versions of excel are storing it as UTF16 internally anyway.

Sadly it doesn't really matter how good NT is if everything uses the Win32 layer. (I heard the NT kernel supported hard links and multiple mount points like Unix too...)

At any rate, it would be nice if we can have FreePascal libraries talk directly to the Unicode interfaces (even the win32 ones), instead of local interfaces where possible. Besides being easier, it would be more reliable.

Some how I ended up spending 14 hours at work today (a Sunday!), so I didn't have a chance to look at the hints given by others yet, but tomorrow... I may make something like a ConsoleWriteln(const s:UTF8String) function that "just Works" to the extent possible.

Thank you,
Noah Silva

_______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Hi,

2013/7/10 Graeme Geldenhuys

On 2013-07-09 13:08, Tomas Hajny wrote: >> Afaik Windows XP, but there might be font problems on console. > I also thought about mentioning True Type fonts (like Lucida) being a

The font is not the issue here...

If you see text that looks like lots of empty squares (looks like ❏❏❏❏), then you probably have a situation where the system is properly decoding your text, but the current font doesn't have glyphs for it. That's no big deal, you just change your font.

If the system thinks your text is in one encoding, but it's really in another encoding, etc., then you will have "mojibake", which basically means that your original text isn't restored properly even in memory, and you will see lots of semi-random characters.

It's not so much about True Type fonts, but about enabling a Unicode font in the console.

This is also not a problem. Most of the fonts use Unicode, so even if you "type" a SJIS file on the console, the system is internally converting it to Unicode and displaying it anyway on newer versions of Windows.

There are many True Type fonts that only support the ASCII character-set. Yes, but the system normally will automatically switch to one that can,

and f.e. the default fonts in Japanese versions of Windows can.

Thank you,
Noah Silva

_______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Hi,

This answer is a bit late, but...

2013/7/9 Dennis Poon

Windows 7, but there isn't any difference in this case.

They both use the NT kernel and so use UTF16 most everywhere internally, but the assume that most programs use local encodings.

Seems that all filenames in XP are treated as unicode code. If you type a non unicode file name in Explorer, it will be auto converted to unicode.

Filenames as accessed from where? Filenames in NTFS are always stored in Unicode, so far as I know. When you enter a file name in Explorer, how do you know if it is being entered in Unicode or not? Basically, explorer is probably a Unicode-aware program (so far as I know), so I would assume any text you enter starts life as Unicode via the IME.

Also, in XP, when text is copied to MS Office from other programs and vice versa, XP seems to do an automatic conversion to UTF-8 and vice versa.

MS Office is also of course Unicode Aware, and the newer XML file formats they use save everything as as Unicode to the best of my knowledge. When importing/exporting text files, clipboard, etc., they will use the local encoding for most things.

That is if you copy some text in your program which is encoding in SJIS and

paste it to Word, the text seems to be auto converted to unicode. Seems reasonable.

As for file handling, it seems some programs will read the first 2 bytes of the text file to determine the the encoding of the file. Google about it.

This is well known, though many programs use much more than 2 bytes in order to guess if there is no BOM. Though Microsoft's guessing algorithm is rather entertaining at times.

Sorry, I don't have exact answers to your questions. Just share some of my experience.

Well certainly the more the merrier, other people who search this list may benefit from your comments even if they aren't relevant to my case.

My problem is this:
I want to output text to the console and have it not be corrupted. I have the text in my program as UTF8, but when I write it to the console, it shows corrupted garbage. This happens whether I write it out as UTF8 or ANSI (SJIS). Since the console basically works with local encoding, it should work when I use UTF8ToAnsi to convert the text before writing it, but it doesn't.

Since it works with other programs and things like the "type" command, it occurred to be that it might be a FPC issue, or that I might be interfering with some built-in auto-conversion feature (which is why I tried to output as UTF8 as well).

I'll figure it out eventually, though.

Dennis

Thank you,
Noah Silva

Hello everyone,

OK I have checked this out:
2013/7/9 Tomas Hajny

According to MSDN, GetACP returns ANSI code page (i.e. never Unicode pages CP_UTF8)
On my system, the codepage it returns is 932 (SJIS), so writing SJIS to the console should be no issue, even is writeln resets the code page every time.

http://msdn.microsoft.com/ja-JP/goglobal/cc305152.aspx

Yet, doing the following yields garbage:
              Writeln(UTF8ToANSI(Object.Text));

(and I have verified that the Text field content is indeed in UTF8 format).

As for my FPC version:
Lazarus 1.1
2012-11-06
FPC 2.6.1
SVN 37432
x86_64-win64-win32/win64

(I have various other versions installed, but all on OS X machines...)

Tomas

thank you,

Noah Silva

Hi,

2013/7/9 Jonas Maebe

This is 932 on my system, as expected.

http://msdn.microsoft.com/ja-JP/goglobal/cc305152.aspx

(I assume they should always be the same though, I don't think most programs check the console output page before writing).

out that way. I think I will have to debut writeln...

Jonas Thank you,

Noah Silva

Hi,

2013/7/9 Michael Schnell

what the Windows API does and what f.e. SAP's ABAP does. On the other hand, UTF8 is very common in files and transferring string data via things like HTTP/XML, so it has to be fully supported any way around it. OS X uses UTF8 as the "local" encoding (so you never have to worry there, except in Java), and apparently so does GTK2. To make it simple, UTF8 can be used everywhere that ANSI encodings were used, because it is ASCII compatible when only ASCII is used. UTF16 and UTF32 can't easily be substituted because they contain "padding" bytes for normal ASCII.

For things like WideString, this is fine. The reason UTF16/UTF32 are popular for in-memory variables is that it's easy to achieve higher performance. For example, with UTF32, there is no need to "decode" the string to find out what character you are on, how many bytes a certain character takes up, etc. If you want the 4th character of a string, you simply go to the 4th 4 byte array element and retrieve the value. UTF16 works the same way if you are dealing with the 99% of characters in use that take only two bytes (but this leads to bugs because people usually don't handle the remaining 1% properly). So you gain processing speed and code simplicity by using UTF16 or UTF32. You lose out on memory if you are dealing with ASCII data only - which is no big deal in most cases. UTF8 saves memory and is more ASCII compatible, but requires more decoding/encoding. Since they represent the same character set, it doesn't really matter in the end - there is a trade-off either way. If you are doing mainly I/O, UTF8 is convenient, if you are doing heavy duty string processing, UTF32 is convenient. Either way, not supporting one or the other is simply not an option if you want to be able to write Unicode compliant programs. One can be the "main" way used by internal routines, and this is UTF16 in more operating systems than not.

If FPC used UTF8 for everything and automatically converted it, then calls to the Win32 API would be slowed down by this, so it makes sense to use UTF16 on Windows, but... then again if GTK2 requires UTF8 then you have the same (but opposite) problem there. Lazarus also doesn't support only Windows, so we have to think a little wider than Delphi.

Another interesting point is that I have heard no end of complaints about Delphi's Unicode strategy, so while we want to be compatible, perhaps we should consider how to do that while possibly avoiding some of the same pitfalls.

To address your specific points:
1.Lazarus User API already supports UTF8 so far as I know.
2. TStringList could easily support both, but as long as the conversion to/from other code pages (especially UTF8) is automatic, I wouldn't mind.
3. Not sure what class inheritance has to do with this...

-Michael

Thank you,
Noah Silva

p.s.: Unicode is an area that I know a lot about, so if anyone working on the RTL needs help testing, let me know...

______________________________**_________________ fpc-pascal maillist - http://lists.freepascal.org/**mailman/listinfo/fpc-pascal<http://lists.freepascal.org/mailman/listinfo/fpc-pascal>

Hi,

2013/7/9 Jonas Maebe

but but saving the file as UTF8 with BOM. (Until now, I had been using UTF8 w/o BOM).

http://wiki.freepascal.org/LCL_Unicode_Support/ja

And the garbage it shows is the same garbage I see if I convert from UTF8 to ANSI manually using UTF8toAnsi or UTF8ToSys.

Detecting the Unicode BOM or not seems to be a strange way to switch the behavior of the compiler, I wonder why a $define wasn't used instead. (Looking at the difference in handling, the BOM way seems much more convenient, so I will convert things over to that...)

Jonas

Thank you,
Noah Silva

______________________________**_________________ fpc-pascal maillist - http://lists.freepascal.org/**mailman/listinfo/fpc-pascal<http://lists.freepascal.org/mailman/listinfo/fpc-pascal>

Hello everyone!

Success! Writeln was indeed the main culprit.

The following works:
  WritelnUTF8(('あいうえお秋葉原'));

(Where:
1. WritelnUTF8 is a procedure that uses the Win32 API WriteConsole directly, bypassing whatever writeln is doing.
2. Where the source code is saves as UTF8 with BOM
3. Where I don't set the console output page to UTF8)

But I'm outputting UTF8 to an ANSI console and it works?! Preseumably, somewhere the RTL sees that my source code is UTF8 and decodes to convert it to ANSI and that gets output and it works since the codepage is ANSI.

A cleaner path would be to set the output codepage to ANSI and tell the compiler not to touch my UTF8, but that doesn't seem to work.

My guess is that Writeln is somehow/somewhere trying to convert the already converted text again, which would very certainly result in garbage!

Once I got the idea to use WriteConsole, I searched and found this: http://forum.lazarus.freepascal.org/index.php?topic=17548.0

(svn: 37432)

thank you,
Noah Silva

2013/7/9 Noah Silva

Noah Silva wrote

Oddly enough, I can get my (2.6.1) version to change the garbage it shows but but saving the file as UTF8 with BOM. (Until now, I had been using UTF8 w/o BOM).

If your source code is in UTF8 but you do not tell this to the compiler, it will directly pass the "garbage" in your source code to the Win32 APIs. After all, there is no way for it to know what else it should with it, and instead has to assume that the programmer knows what he is doing.

Detecting the Unicode BOM or not seems to be a strange way to switch the behavior of the compiler,

Why? A BOM unequivocally states the encoding of a file.

I wonder why a $define wasn't used instead.

A directive is also available: http://www.freepascal.org/docs-html/prog/progsu88.html

And a command line parameter also (-Fc).

Those programs would be wrong. A user can easily change the console output page with the "chcp" command.

Jonas

Noah Silva wrote

Using UTF16 for internal string handling is a sensible option.

It depends.
UTF-16 needs more memory used
Linux OS API in most cases is 8 Bit, while Windows OS API is 16 bit Conversions are very expensive.
If you need to import export much data but don't do much calculating of course using the the import/export format all over the place is sensible. If you do many calculations, the type of calculation might suggest a certain encoding.

To address your specific points: 1.Lazarus User API already supports UTF8 so far as I know.

I suppose this is bound to change once fpc has completed the move to "new Delphi Strings".

2. TStringList could easily support both, but as long as the conversion to/from other code pages (especially UTF8) is automatic, I wouldn't mind.

I already delved into this in another thread here: I do believe that it is easily possible to invent a string type that supports any encoding and that can be used to create such a flexible TStringList, but this needs additional compiler support in an way that is not anticipated by Delphi. IMHO this is possible without risking noticeable performance degradation in any of the thinkable application variants.

3. Not sure what class inheritance has to do with this...

If you do TSrtingList (in fact TStrings)that uses this new type in the user-programmer interface it needs to be possible to derive classes from those that use the fully Delphi compatible String types with predefined encoding. The compiler magic needs to be done appropriately to handle this cases, requesting automatic conversions (only) when necessary.

-Michael

[...] > To address your specific points: > 1.Lazarus User API already supports UTF8 so far as I know.

Yes, since many years.

I suppose this is bound to change once fpc has completed the move to "new Delphi Strings".

fpc is not "moving" to Delphi strings. That would break all compatibility. It will support both.

> 2. TStringList could easily support both, but as long as the > conversion to/from other code pages (especially UTF8) is automatic, I > wouldn't mind.

Note: TStringList can be used with UTF8 except for the LoadFromFile and SaveToFile functions.

[...]

Mattias

Mattias Gaertner wrote

fpc is not "moving" to Delphi strings. That would break all compatibility.

This is exactly what Delphi did :-( .

It will support both.

There might be an appropriate compiler switch. But what about the RTL-Interface like the type StringList offers?

Note: TStringList can be used with UTF8 except for the LoadFromFile and SaveToFile functions.

If the general option is set to "new Delphi Strings" and this does work in fully Delphi compatible way, TStringList will offer just 16 Bit strings as it's interface.

Using UTF8 with this would be easily possible, as this mode does auto-conversion.

But forcing a conversion each time when adding and when retrieving a String would result in sluggish behavior.

-Michael

Hi,

2013/7/29 Jonas Maebe

Yes, however in several places the Wikis recommend UTF8 w/o BOM, so I had tried that some of the time. I have never had any problems other than the Windows Console issue.

to make the encoding of the file clear, but not so much as a mode switch (which is what the Wiki makes it sound like).

In most other situations, adding/removing the BOM doesn't make much of a difference. Adding it may cause compatibility issues if programs don't expect it, and it may help if they didn't properly autodetect the encoding anyway.

This makes sense to me.

program and it works on every Japanese Windows system. One user changes their code page to Greek or some other code page that their system might not even have the fonts for and then complains that the program doesn't work. The answer will be "We only support Japanese systems." This makes sense for local encodings, since you can only usually support one, and things can't be cleanly cross-coded anyway. (f.e. if I changed my console to Korean with chcp and my program tried to properly convert the code page , it wouldn't work anyway, because the characters used in Japanese don't exist in the destination character set). With the possibility for Local or Unicode character sets of course it makes sense to check which one is in use and output to that one - I just doubt that most programs actually do.

Thank you,
Noah Silva

Jonas_______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Hi,

2013/7/29 Michael Schnell

No, UTF16 only needs more memory if most of the text is ASCII. It actually uses less than UTF8 in the average case for Japanese, for example.

Linux OS API in most cases is 8 Bit, while Windows OS API is 16 bit

I assume by 8bit, you mean variable byte encoding like UTF8.

Conversions are very expensive.

This is not as bad as some people make it out to be. You have to be converting a *lot* of data for it to be noticeable.

If you need to import export much data but don't do much calculating of

then either format is just fine.

I really don't think so, the reasons are even well detailed in the Wiki.

an attribute internally along with their length, and since the compiler/runtime library will know their codepage, it can convert as necessary. Either way, you can make your own StringList variants for each type easily enough.

For example, I normally use UTF8 for everything, but I have one linguistic analysis program I wrote that does heavy duty analysis of the string, so that stores everything in memory as UTF16. I use StringList and similar without any problems. (I don't use UTF8 and UTF16 in the same structures though...)

"just works" with UTF8 and/or UTF16. Then you would only need to occasionally worry about local encoding for legacy import/export use. It's easy enough just to make an overload for Windows API calls (ick) that can accept UTF8 or vice versa for UTF8 native calls you want to use with UTF16 if you really need to. The real issue to be is that until now, the compiler doesn't actually *know* that a string is UTF8 or SJIS, which means it doesn't give you an error when things aren't right, they just get garbled, and the programmer gets left with a mystery to sort out.

Noah Silva wrote

No, UTF16 only needs more memory if most of the text is ASCII. It actually uses less than UTF8 in the average case for Japanese, for example.

Of course you are right here.

Yep.

That is why I pointed out that the way to select an encoding depends on how much "calculations" are done on the strings.

But in fact I tend to agree, while the argument why - when converting to Unicode - the Lazarus team chose to do the LCL API in UTF-8 (while MSE chose UTF-16 for the same purpose) was exactly this (I never felt comfortable with that, BTW).

I always was told that Delphi compatibility is the primary driving forth for any modifications. This necessarily suggests this move (which is not possible before fpc does provides "new Delphi Strings"). But there might be multiple opinions.

In fact my primary intentions with Lazarus / fpc are not to do my own generic projects, but to help my colleagues to move their huge Delphi XE program system to Linux. This in fact needs complete support for "new Delphi Strings".

From what I understand, the plan is for strings to store their codepage as an attribute internally along with their length, and since the compiler/runtime library will know their codepage, it can convert as necessary.

That already is ready to use in the svn and is exactly the said "new Delphi Strings", and - when activated - completely compatible with Delphi XE. It's rather nice and fast, but Delphi lacks a _completely_dynamic_encoding_ type with auto-conversion only when necessary. (IMHO rather easy doable by compiler magic, but "forgotten" in Delphi XE)

Either way, you can make your own StringList variants for each type easily enough.

Not without compiler support (if you want auto-conversion when necessary).

In fact, I am fine with manual conversions, so long as 99% of everything "just works" with UTF8 and/or UTF16.

I'm not fine with TStringList and friends forcing any predefined encoding. This in fact does work rather nicely without the application programmer even noticing it. But IMHO a cross platform system like fpc can be expected to do better, doing away with windowish remains from Delphi whenever possible.

-Michael

As mentioned before, that causes the compiler to directly pass your UTF-8 data around with any conversion. That is how Lazarus works (it stores UTF8 data in plain ansistrings) and as long as you only use LCL routines it will work fine, but using such code directly with the OS API (via the FPC RTL or not) will obviously cause problems if that API does not expect data in UTF-8 format.

It causes the compiler to interpret the string constants in your program as UTF-8 rather than as unknown binary data, and hence convert them at run time to the current ansi code page when assigning them to an ansistring/shortstring. This is unrelated to mode switches.

Those are business decisions, which are irrelevant to the FPC RTL. Our code is written as much as possible to work correctly under all circumstances.

Jonas

Hi,

>> If your source code is in UTF8 but you do not tell this to the compiler,

...

I pretty much use only UTF8 in most programs, and make exclusive use of UTF8String. (I only use ANSIString if the string is known to be in a local encoding, such as having been read from an SJIS file) - but I was under the understanding that UTF8String is just an alias to ANSIString for now anyway. I convert if necessary when using the OS APIs. (The main problem is knowing when it is necessary...)

Ooh, this clears up a lot that the Wiki didn't explain very well!

So basically saving the file with BOM tells the compiler/RTL to take care of things for you, and saving as UTF8 without BOM is appropriate if you will take care of any conversions yourself.

If UTF8String is just an alias to ANSIString, then I assume it also means that right now the compiler would convert such constants to the local encoding even when assigning to a UTF8String? (If so, this explains why my finally working console code works only with No BOM).

Also, I assume that the treatment of ResourceString and any other constants is the same?

Thanks for your comments,
Noah Silva

You may want to adapt the wiki in that case. It is mostly written by users for users. We only maintain a few pages on the wiki ourselves (such as User_Changes_Trunk).

Yes.

If UTF8String is just an alias to ANSIString,

It is, in 2.6.x.

Indeed.

Also, I assume that the treatment of ResourceString and any other constants is the same?

Yes.

Jonas