Hello

I have opened issue #0018144 in the bugtracker and uploaded a new version of my codepages unit.

My description on this :

In September we had a discussion on the Lazarus-mailing list to rewrite LConvEncoding and move the functionality to the RTL (Thread: rewriting of LConvEncoding).

Since there I did a lot of coding to implement an effective algorithm, both for Singlebyte- as for Doublebyte-Codepages. A first release was on the mailing list mid-October, mainly as a base for further discussions. But there were no comments or suggestions on this.

So here is a nearly final release with many changes to the first version.

Major points:
- The unit supports Single- and Double-bytecodepages trough the same functions
- Widestringsupport (configurable)
- UTF8 and UTF16 support (UTF16 needs widestrings)
- Direct conversion from CP to CP without intermediate string
- Uppercase and Lowercase support
- Underlying Unicodes as of V 6.0.0 (October 11, 2010)
- A converter-application to convert Unicodedefinitions to a complete pascal unit. The cp_* units are entirely generated by this app.
- Conversion up to 80% faster for SBCS. For DBCS up to 100 times

As for now there are only units for ISO-8859-1, ISO-8859-2 and CP932 (SHIFT_JIS). More to be added for the final release. The converter-subdir has all the definition files that I could find. I will add them all.

The units:
codepages.pp : the main unit (highly configurable trough codepagesdef.inc) unicodemappings.pas : Some definitions from unicode.org, especially the tables for uppercase, lowercase and the unicodeblocks.
utf8.pas : mainly the UTF8 functions from LCLProc + some new utf16.pas: same for UTF16
acpinfo.pas: info for codepages supported by Windows, as published on MSDN

Some first test results as attachment.

Best regards

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed... Name: TestResults1.txt
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101203/8a41c476/attachment.txt>

Zitat von Guy Fink :

It does not compile under 2.4.2:
cp_ISO88591.pas(69,37) Error: Constant strings can't be longer than 255 chars

Great.

- Direct conversion from CP to CP without intermediate string

Nice.

Ehm, you made many functions inline. Even those that are more than a few lines of code. This will enlarge the executables and can cost performance in normal applications (e.g. Lazarus). You call for each character a conversion function. But most real world texts contain a big part of ASCII characters, where no conversion is needed for UTF-8. My guess is that for most texts this approach is slower. But I have to wait till it compiles before I can test.

- For DBCS up to 100 times

;)

Mattias

It does not compile under 2.4.2: cp_ISO88591.pas(69,37) Error: Constant strings can't be longer than 255 chars

Ohoh... Ok, that are the Teststrings, they contain ALL the unique definded characters for the codepage. I have developped under 2.5.1

Undefine cpCnvGenerateTeststrings in codepagesdef.inc. So it should compile. I only use the teststrings to verify the conversion process.

> - Widestringsupport (configurable) > - UTF8 and UTF16 support (UTF16 needs widestrings) Great.

And I add UTF-32 support when FPC implements a 4-byte character :-)

> - Direct conversion from CP to CP without intermediate string Nice.

. and fast

I will have a look at this.

You call for each character a conversion function. But most real world texts contain a big part of ASCII characters, where no conversion is needed for UTF-8. My guess is that for most texts this approach is slower. But I have to wait till it compiles before I can test.

My tests dont show this for now. The conversion function is used in DirectConversion only, or when the codepage has no translation tables (as for SHIFT_JIS resp. all the DBCS). The SingleByteToxxx use a table function, and also use the ASCII-optimisation if approriate for the Codepage. But as you see, I will also include EBCDIC-Codepages..

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Major problems that I see from a quick look:
- does not integrate with FPC's existing systems.
- FPC already has a CP_ generator and loader system (creumap and charset)

- Introduces an own enumeration for charsets. No possibilities to integrate this with system codepage enumeration. (or whatever substitute FPC will define for this). I know that it doesn't help that FPC hasn't made a decision about this yet wrt cp_newstr
- when using these routines for simple UTF8 operations, the large table with descriptions of unit unicodemappings is always linked in.
- no attempt to use system codepage routines and tables. (this could be done on widestr manager level for the platform though)

None of these faults prevent inclusion in say packages/ as a backup solution for systems without codepages, but for rtl/ it is not modular enough (too big, too much a standalone system that doesn't integrate with system)

Major problems that I see from a quick look: - does not integrate with FPC's existing systems. - FPC already has a CP_ generator and loader system (creumap and charset)

We had this discussion in September. I think there is no need to repeat this again. I will add a replacement for charset, using the codepages unit, so existing code will still work.$

At least it has an enumeration, and it is no problem to integrate whatver FPC will implement. ACP support is already integreated.

- when using these routines for simple UTF8 operations, the large table with descriptions of unit unicodemappings is always linked in.

UnicodeBlockMapping can be taken off. I have implemented it for further enhancements. But if there is no need, no problem. By the way, this was in LCLProc before, I have only updated to Unicode V 6.0.0 and added the Uppercase-tables.

- no attempt to use system codepage routines and tables. (this could be done on widestr manager level for the platform though)

Exactly, this was the goal, to get it completely independent of the system. There is a pseudo-codepage called ANSI .. it will call the system routines in the final release (if they exist).

It is so modular that you can choose codepage by codepage what you need. Furthermore you can tune size vs. speed by the compilerdirectives. And for the same functionality, the result is not bigger than with charset.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Yes, so I was curious to see your progress on this front in 3 months.

I think there is no need to repeat this again.

Well. I hope I made clear in September I'm not going to commit anything without this is being fixed, so submitting this report was IMHO a bit premature.

I will add a replacement for charset, using the codepages unit, so existing code will still work.$

I prefer a patch that extends the current unit, not a new unit. It's hard to evaluate of an entire 3rd party product is truly backward compatible. (and charset has been in FPC for ages, so must be supported fully backwards compatible)

> decision about this yet wrt cp_newstr At least it has an enumeration, and it is no problem to integrate whatver FPC will implement. ACP support is already integreated.

As said in September, it is wiser to work with a registration system, that will registrers codepages to an internal table on demand, with their system dependant registration.

We can then put this dynamic registration deep in the RTL, and allow higher level units to plug into it.

Keep in mind that Lazarus apps are always fairly large. FPC is also used for e.g. cgi apps, and even embedded arm code. So such systems must be optional, and default as light as reasonably possible.

This means that it must go in a packages/ then, since if it is not needed for the default RTL, it can be in packages/.

The default RTL widestring manager must be as lean as possible, and thus be able to work with OS tables.

There is a pseudo-codepage called ANSI .. it will call the system routines in the final release (if they exist).

(just fyi: Windows has _three_ different encodings. One is always utf-8 (the -W routines, the other is "system ansi" which can be any 1-byte codepage, and the third is OEM, a more hardware font oriented encoding used for the console)

- It has a compiletime defined enumeration of codepages, which I assume means that adding new encodings can't be added without recompile
- it doesn't manage any relation to system encodings. I don't know hw

Furthermore you can tune size vs. speed by the compilerdirectives. And for the same functionality, the result is not bigger than with charset.

Most people install FPC precompiled, and such and any other configuration must be possible without recompilation.

Some more points:
- is iconv actually used ?
- why is unit dos included? Risky, since using the wrong function might cause truncation due to ansistring<->shortstring functions

Marco van de Voort wrote

(just fyi: Windows has _three_ different encodings. One is always utf-8 (the

^^^^^^^^^^
As Vincent correctly pointed out, this should be utf-16.

-W routines, the other is "system ansi" which can be any 1-byte codepage, and the third is OEM, a more hardware font oriented encoding used for the console)

What do you mean with "fully backwards compatible"? That function header and syntax is the same, or that charset is a holy cow where no byte may be changed?

When you look at the code you see that it is exactly this, a registration system. But a static one, not a dynamic. The table is filled with default value, so the code does not need any search and "if then else"-orgies to find a codepage. if a codepage is not registered it raises an exception. As I wrote, I have not included all the codepages at this point, because I do not want to do the work twice if any changes come out of this discussion.

What shall be ligther than to only include what is needed?

just fyi:
There are 4 different: You forgot the default MAC-codepage and also just fyi:
Following Microsoft documentation Widechars are encoded UTF-16 not UTF-8

For the final release there will be at least 67 codepages to choose from, and only UTF-8 and UTF-16 will be compiled in. (perhaps ISO-8859-1 as a last ressort also.. will see). And you dont't have to recompile anything to use this codepages, just include the needed units in your uses-clause. If the RTL is compiled with switch cpCnvUseAutoregister they will register themselves.

Some more points: - is iconv actually used ?

no it is not.

- why is unit dos included? Risky, since using the wrong function might cause truncation due to ansistring<->shortstring functions

both points are a leftover from LConvEncoding ... They will be taken off.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

I believe he means the interface part of the unit and also the semantic one.

E.g.
"B := functionXYZ(A);"
in the old unit should calculate the same B for the same A in the new unit.

Regards,
Sven

Hello Lazarus-List,

Friday, December 3, 2010, 1:24:17 PM, you wrote:

GF> UnicodeBlockMapping can be taken off. I have implemented it GF> for further enhancements. But if there is no need, no problem. By GF> the way, this was in LCLProc before, I have only updated to GF> Unicode V 6.0.0 and added the Uppercase-tables.

Keep waning about upper/lower tables, they are country dependent and your tables seems to be the country agnostic ones. I know that adding the country exceptions are quite difficult as it happends to me.

The UTF8 decoders included in your code (the same as LConvenconding) are highly insecure and do not pass the stress test due remove failed sequences (character must be replaced by an error indicator to be secure) and output wrong sequences as good codepoints.

The upper/lower tables in Unicodemappings are pure Unicode (please take a look). They are in NO way country dependent. What make you think " tables seems to be the country agnostic ones"

As I said, I have taken this from LCLProc.

The Unicode > UTF8 should be ok... It does not generate a codepoint for a character outside the Unicoderange.

For UTF8 > Unicode, well this is a desiign question. There is a function that generates an exception on a wrong sequence. What else would you do?

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Hello Lazarus-List,

Friday, December 3, 2010, 7:27:51 PM, you wrote:

GF> The upper/lower tables in Unicodemappings are pure Unicode GF> (please take a look). They are in NO way country dependent. What GF> make you think " tables seems to be the country agnostic ones"

In some languages some unicode codepoints have different uppercase/lowercase pair. In example "i" in english (and most others) region is uppercased to "I" while in Turkish it is "I"+Upperdot (i can not write it here).

Take a look over: "Why Applications Fail With The Turkish Language" at http://www.i18nguy.com/unicode/turkish-i18n.html

GF> As I said, I have taken this from LCLProc.
GF> The Unicode > UTF8 should be ok... It does not generate a GF> codepoint for a character outside the Unicoderange.

Well, in fact yes, try converting Unicode $FFFF to UTF8 (not tested, but 99% of plain implementations just overlook exceptions).

GF> For UTF8 > Unicode, well this is a desiign question. There is GF> a function that generates an exception on a wrong sequence. What GF> else would you do?

There are 2 problems, invalid sequences and malformed sequences, both are different beasts.

Malformed sequences: They are unterminated UTF8 sequences. So they starts as an UTF8 sequece, but they ends abruptally with a non continuing mark. In example #128#1

Invalid sequences: This are writting most times intentionally to bypass protection systems, raise buffer overflows and other "funny" things. They exploit the ability of UTF8 to using one sequence obtain a totally different character. In example #$C0#$80 which will output NULL char which is in fact dangerous when playing with C-Style strings.

Recomendation is to replace the invalid/malformed codepoint by "?" or better by the unicode error question mark '?' (U+FFFD), or raise an exception, but never eat it.

In order to test you can use the stress test: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

Take a look over UTF8ToUnicode in fpc sources (there is one border case that it is still wrong).

There is no information on the language in a string, even not in a Unicodestring. So it is impossible to react on this point here.

The uppercase/lowercase tables have been generated purely on the official Unicode-Character-Description. Characters having a "SMALL" in their description are replaced by the one having "CAPITAL" on that place and vice-versa. (only if the counterpart exists) You can't do more on this level. Please feel free to implement the functionality you mention, I'll be sure it will be appreciated.

We are Pascal, not C. And in Pascal NULL is a valid character.

Once again, I have taken most of this from LCLProc, but I agree that improvements can be done here. But this was not the aim of this coding. The actual encoding is in most cases reversible (exceptions are double definitions of characters like in SHIFT_JIS. Converting those characters to unicode ends in loosing some information, so the back conversion may result in another SHIFT_JIS-character, but which represents the same graphical glyph)

On the other side there is a function called UTF8FixBroken to take off invalid sequences and codepoints. But it is also not perfect, because it is a C-style function.

Once again, improvements to this coding is welcome.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Hello Lazarus-List,

you wrote

> Take a look over: "Why Applications Fail With The Turkish > Language" at > http://www.i18nguy.com/unicode/turkish-i18n.htm

GF> There is no information on the language in a string, even not GF> in a Unicodestring. So it is impossible to react on this point GF> here.
GF> The uppercase/lowercase tables have been generated purely on GF> the official Unicode-Character-Description. Characters having a GF> "SMALL" in their description are replaced by the one having GF> "CAPITAL" on that place and vice-versa. (only if the counterpart GF> exists) You can't do more on this level. Please feel free to GF> implement the functionality you mention, I'll be sure it will be GF> appreciated.

I'm not trying to offed your work, just trying to ring a bell before somebody starts to complaint about different behavior in a system when using OS functions and when using native pascal ones.

GF> We are Pascal, not C. And in Pascal NULL is a valid character.

Once again, I'm not fighting against you.

GF> Once again, I have taken most of this from LCLProc, but I GF> agree that improvements can be done here. But this was not the aim

That's the reason I'm trying to let you note that there are some anomalies here and there in code that you are taking from other side. No more, no less.

GF> On the other side there is a function called UTF8FixBroken to GF> take off invalid sequences and codepoints. But it is also not GF> perfect, because it is a C-style function.

UTF8FixBroken is "broken" :) It fixes with spaces which is indeed wrong and it does not detect all broken strings, and also, yes, it is a NULL terminated string function :-? quite strange.

If you want I can send you my code to normalize canonical strings if you wish to add it, but again it is a quite big table and country agnostic.

IMO there is no need to have a language encoded in the string. Strings won't get autoconverted to upper/lowercase. It's always a user call to Upper/Lowercase(S)

To take the Language into account when converting, functions like Upper/Lowercase should have a 2nd optional parameter indicating for what language the conversion should be done.
THen the default conversion still can take place, but based on the specified language, the exceptions can be implemented (if there anrent many exceptions, only a simple case will do)

Marc

I agree completely.

I also agree, but as Jos? Mejuto wrote the tables for that seem to be very large. I'm sure optimisation can be done but you have also seen the comment from Marco van de Voort that the tables are by now too big for the RTL.

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

As Marco van de Voort requested me to reuse the large functionality of charset (see bugtracker comment) I have enlarged my test-application. Here are the results :

...

ISO-8859-1 >> UTF-8 using LConvEncoding ? Input string has 256 characters. --------------------------------------------------------------------------- Evaluating LConvEncoding.ConvertEncoding(string,iso88591,utf8):string 100000 times, Time: 0,312 [s] : Result is correct. Evaluating LConvEncoding.ISO_8859_1ToUTF8(string):string 100000 times, Time: 0,249 [s] : Result is correct.

ISO-8859-1 >> UTF-8 using Charset ? Input string has 256 characters. --------------------------------------------------------------------- Charset does not support conversions to UTF8, using utf8-unit for that Evaluating utf8.UnicodeToUTF8(Charset.getunicode(string,iso88591)):string 100000 times, Time: 2,480 [s] : Result is correct.

ISO-8859-1 >> UTF-8 using Codepages ? Input string has 256 characters. ----------------------------------------------------------------------- Evaluating DirectConversion(string,chEncISO-8859-1,chEncUTF-8):string 100000 times, Time: 0,187 [s] : Result is correct. Evaluating ConvertToUTF8(string,chEncISO-8859-1):string 100000 times, Time: 0,234 [s] : Result is correct.

...

ISO-8859-1 >> UTF-16 using Charset ? Input string has 256 characters. ---------------------------------------------------------------------- Charset does not support conversions to UTF16, using utf16-unit for that Evaluating utf8.UnicodeToUTF16(Charset.getunicode(string,iso88591)):widestring 100000 times, Time: 7,847 [s] : Result is correct.

ISO-8859-1 >> UTF-16 using Codepages ? Input string has 256 characters. ------------------------------------------------------------------------ Evaluating DirectConversion(string,chEncISO-8859-1,chEncUTF-16):widestring 100000 times, Time: 0,203 [s] : Result is correct.

ISO-8859-2 >> UTF-16 using Charset ? Input string has 256 characters. ---------------------------------------------------------------------- Charset does not support conversions to UTF16, using utf16-unit for that Evaluating utf8.UnicodeToUTF16(Charset.getunicode(string,iso88592)):widestring 100000 times, Time: 7,831 [s] : Result is correct.

ISO-8859-2 >> UTF-16 using Codepages ? Input string has 256 characters. ------------------------------------------------------------------------ Evaluating DirectConversion(string,chEncISO-8859-2,chEncUTF-16):widestring 100000 times, Time: 0,219 [s] : Result is correct.

....

ISO-8859-1 >> ISO-8859-2 using LConvEncoding ? Input string has 256 characters. -------------------------------------------------------------------------------- Evaluating LConvEncoding.ConvertEncoding(string,iso88591,iso88592):string 100000 times, Time: 0,873 [s]

ISO-8859-1 >> ISO-8859-2 using Charset ? Input string has 256 characters. ---------------------------------------------------------------------------- Evaluating Charset.getascii(Charset.getunicode(string,iso88591),iso88592):string 100000 times, Time: 9,079 [s]

ISO-8859-1 >> ISO-8859-2 using Codepages ? Input string has 256 characters. ---------------------------------------------------------------------------- Evaluating DirectConversion(string,chEncISO-8859-1,chEncISO-8859-2):string 100000 times, Time: 0,218 [s]

....

SHIFT_JIS >> UTF-8 using LConvEncoding ? Input string has 14843 characters. ---------------------------------------------------------------------------- Evaluating LConvEncoding.ConvertEncoding(string,cp932,utf8):string 1000 times, Time: 24,321 [s] Length(Result)=22078 Length(Reference)=22173 : 79 characters are different. Evaluating LConvEncoding.CP932ToUTF8(string):string 1000 times, Time: 24,414 [s] Length(Result)=22078 Length(Reference)=22173 : 79 characters are different.

SHIFT_JIS >> UTF-8 using Charset ? Input string has 14843 characters. ---------------------------------------------------------------------- Charset does not support conversions to UTF8, using utf8-unit for that Evaluating utf8.UnicodeToUTF8(Charset.getunicode(string,cp932)):string 1000 times, Time: 1,560 [s] Length(Result)=39233 Length(Reference)=22173 : 21798 characters are different.

SHIFT_JIS >> UTF-8 using Codepages ? Input string has 14843 characters. ------------------------------------------------------------------------ Evaluating DirectConversion(string,chEncCP932,chEncUTF-8):string 1000 times, Time: 0,234 [s] : Result is correct. Evaluating ConvertToUTF8(string,chEncCP932):string 1000 times, Time: 0,218 [s] : Result is correct. Evaluating CP932ToUTF8(string):string 1000 times, Time: 0,218 [s] : Result is correct.

Hmmm, the conversion SHIFT_JIS >> UTF-8 using the Charset-unit ended up with a complet mess. The reason is, that the large functionality of charset has no mean to convert Doublebyte charsets to Unicode. :(

The complete Testresults in the attachment...

I will publish the Testprogram on the bugtracker.

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed... Name: TestResults3.txt
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101204/36a03448/attachment.txt>

BTW, the testprogram needs FPC 2.5.1 to compile, because of the large reference stringconstants...

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Hello Lazarus-List,

Saturday, December 4, 2010, 3:34:50 PM, you wrote:

GF> I also agree, but as Jos? Mejuto wrote the tables for that GF> seem to be very large. I'm sure optimisation can be done but you GF> have also seen the comment from Marco van de Voort that the tables GF> are by now too big for the RTL.

Language specialization are not very large but quite complex to be used as there are several conditions and are expressed as some XML macro substitution.