An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100920/7dc94e94/attachment.html>

It's goal is to convert the most common codepages and do the rest with the existing OS libraries (iconv).

A complete conversion unit should be part of FPC.

True.
And the tables are quite big. Maybe the unit should be splitted into several units.

I would rewrite the entire unit, including most of the codepages from ftp.unicode.org, full support for UTF-8, UTF-16 and UTF-32, including LE and BE variants. And as a further utility, a conversion program for the mentioned codepagefiles, that will create a complete pascal include file with all the necessary tables and conversion functions.

What do you mean with full support for UTF-8, UTF-16 and UTF-32?

Mattias

--- Urspr?ngliche Nachricht ---
Von: Mattias Gaertner
An:
Betreff: Re: [Lazarus] rewriting of LConvEncoding

I agree, but that should be decided by the core developers.

I think with the rigth datamodel the tables can be reduced a lot in size. But nevertheless, it may be a good idea for those who don't need the asiancodepages.

I mean full conversion possibilities between old fashioned codepages and UTF, and between the different UTF-strings, always both directions.

I know that parts of this functionality is also in LCLProc, but it is also not complete.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

The core developers do not have a problem with this. Since it is mostly tables anyway, I see no reason not to do it.

Note that FPC does contain a some conversion code in the RTL, see the ucmaps directory and the charset unit.

Michael.

--- Urspr?ngliche Nachricht ---
Von: Michael Van Canneyt
An: Lazarus mailing list
Betreff: Re: [Lazarus] rewriting of LConvEncoding

The files in ucmaps are the files from ftp.unicode.org that I mentionned.

My opinion is that static tables are smaller and faster than dynamic mappings (especially for the asiancodepages), but memory could be an issue. Perhaps this can be solved by smartlinking.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

I use my own units to convert win1256 to utf8 or widestring http://sourceforge.net/projects/minilib/

In directory Unicodes

http://minilib.svn.sourceforge.net/viewvc/minilib/trunk/Unicodes/source/

You can install the and hook it as unicodemanager
e.g.

    ucpInstall(cp1256_mbtowc, cp1256_wctomb, True);

I ported it from libiconv at sourceforge
https://sourceforge.net/projects/libiconv/.
Need some one to port another files.

Zaher Dirkey
Best regards.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100921/66f42c5f/attachment.html>

2010/9/20 :

I would rewrite the entire unit

Great, just make sure you take all variables into account:

* code size (check if your tables are smartlinkable. The current ones are)
* speed
* maintenability

Mattias Gaertner wrote

And the tables are quite big. Maybe the unit should be splitted into several units.

In my tests the asian tables are smartlinked and won't be added unless necessary.

Splitting into several units is also possible, but then a registration mechanism should be added.

There is charset ?

True. And the tables are quite big. Maybe the unit should be splitted into several units.

Charset already does.

Explain static/dynamic in this context. Note that there is a tool to change the conversions into tables, so they can be linked in, already.

--- Urspr?ngliche Nachricht ---
Von: Felipe Monteiro de Carvalho
An: , Lazarus mailing list
Betreff: Re: [Lazarus] rewriting of LConvEncoding

My first idea of the structure then:

A ConvertEncoding-unit, containing all the necessary types,constants, function prototypes and the Unicodestuff. (conversions between the different UTF-variants). It could hold some jumptables for the conversion procedures, which are initalized to a local dummy function.

Then I do not generate include files, but a complete unit for every codepage we add. This unit will register itself to the ConverEncoding-unit in its initialisation part. So the user has full control over supported codepages of his application by simply using the the needed codepage-units.

Guy Fink

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

--- Urspr?ngliche Nachricht ---
Von: Marco van de Voort
An: , Lazarus mailing list
Betreff: Re: [Lazarus] rewriting of LConvEncoding

With the tool you mean the charset-unit? What I see is, that charset is not finished. It just offers a rudimentary way to read in the unicode.org textfiles, and some functions to find a mapping and convert one character. No support for complete string conversions, or UTF-8, UTF-16, UTF-32.

The tables are created dynamically via getmem and stored in a linked list. Every character is stored in a record : tunicodecharmapping, where Unicode is only definded as word, not cardinal. Thus UTF32 is not supported, UTF16 surrogates neither. The list is lineary scanned to find a certain mapping, not really optimal for fast conversions. The backconversion unicode > character is even worse than the algortihm in LConvEncoding, it searches linear through the mapping, this may take some time on big asian codepages. (LConvEncoding uses a dual Quicksearch)

Charset has absolutly no support to handle endianess of UTF-16 and UTF-32 strings.

With static tables, I mean a table in a const-section, compiled and linked into the code. This offers optimization possibilities to the compiler regarding indexed access to the table content. The tables can be optimized for best fitting datasize, latincodepages can mostly be encoded as UCS2. Others, especially asiancodepages but also some MAC-codepages need UCS4. The backconversion can be optimized trough individual functions for every codepage.

It is easy to write an utility which creates the sourcecode for such this codepage-units.

Best regards

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

I think one unit per group is sufficient. For finer granularity there is smartlinking.

Mattias

Ok, but than the registering of a mapping has to be made "by hand". Will say, if all the codepages in one unit are automatically registered during initialization, they compiler has to treat them as "used", even if the conversion functions are never really called. Or am I wrong?

What groups would make sense? As structured on ftp.unicode.org :

ISO8859 : codepages 1-15
Windows : cp12xx
IBM-PC : cp437 and so on
MAC : some of the MAC-mappings
EBCDIC : cp037, cp424, cp500, cp875, cp1026
MISC : ATARIST, NEXT, KOI8....

or grouped by characterspace :
Western europe (Latin)
Eastern europe (Kyrillic)
Eastasia, (or finer? Chinese, Japanese, Thai, Vietnamese......) Hebrew,
Greek
Special (like EBCDIC, ATARIST....)

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Guy Fink wrote

> the conversions into tables, so they can be linked in, already. With the tool you mean the charset-unit?

No, I mean fpc/rtl/units/creumap.pp that afaik generates statically linkable units from ISO files that plugin to charset.

What I see is, that charset is not finished.

Then finish it.

It just offers a rudimentary way to read in the unicode.org textfiles, and some functions to find a mapping and convert one character. No support for complete string conversions, or UTF-8, UTF-16, UTF-32.

Then make a good proposal to fix this. Preferably with patches.

The tables are created dynamically via getmem and stored in a linked list. Every character is stored in a record : tunicodecharmapping, where Unicode is only definded as word, not cardinal. Thus UTF32 is not supported, UTF16 surrogates neither.

UTF32 is nowhere supported at all with FPC atm, and to be honest, I don't see a reason to start now. The unicode Delphi's also don't provide a type for it. It is simply the most practical format, and the few places where it is typically used , like complex string routines and the like, can survive on hardcode handoptimized code. (IOW it is not really an user type)

Since despite what people think, UTF32 is extremely wasteful, and still not free from problems (codepoints vs chars, denormalized sequences etc)

Well, one of the reasons is that the unit is mainly used for embedded applications (which includes DOS and win9x nowadays) or special cases (like very, very compatible installers), since on normal targets the OS routines are used.

Nevertheless, I don't want to hide behind that. Certainly, charset is pretty much a one-off effort and can be improved. But please, when reengineering, keep in mind that the "special" uses are the main ones.

But if everybody tries to roll something new instead of improving existing functionality the we are getting nowhere.

Charset has absolutly no support to handle endianess of UTF-16 and UTF-32 strings.

I would add separate special functions for that. No need to bog down the standard functions that do the bulk of the work. IOW a special functions that do input validation at the perimeter, and functions that only do internal conversions (e.g. that you could base the widestring manager on)

With static tables, I mean a table in a const-section, compiled and linked into the code.

Have a you had looked up where and how (c)charset is used, you would have noticed

(see e.g. compiler/cp*

I don't see that last bit. Yes, some Asian codepages have some chars above the BMP, but afaik they are relatively rare. You might get more speed (due to better cache utilization) AND a better footprint by having a separate UCS2 table, and a separate check (and tables) for the surrogates.

It is easy to write an utility which creates the sourcecode for such this codepage-units.

See above.

Really, I don't know what to think of that order. I want to contribute to the project with my knowledge of more than 20 years programming in Pascal (since the very first days of Turpo Pascal). I surely will not waste my time on completing an algorhytm that I think is unappropriate for the problem!

Does not really make sense when after applying the patch there is nothing left from the original.

Is that really a reason not to start support for it? I don't think so. I even think it is a reason to support it, Delphi does not have full Unicodesupport, FPC will have.

UTF32 is there in the world, and yes it is wasteful.. And so what? Is that a reason to ignore it?

Well, one of the reasons is that the unit is mainly used for embedded applications (which includes DOS and win9x nowadays) or special cases (like very, very compatible installers), since on normal targets the OS routines are used.

These routines do not support all of the codepages. Further, the aim of a library is not to wrap some OS routines but to deliver functionality to the developer to help him solve his problem. Developers need solutions, not good words of how clean and ligthweigth the libraries are.

And if everybody holds on algorhytms which have been identified as beeing not appropriate to the problem you are getting nowhere either.

I have noticed... and now? Doesn' t improve the algorhytm. Perhaps it is better first to think over the right datastructures than to write down some trivial lines of code and to propagate that these have to stay now like that till the end of the days.

Sorry at this point for these hard words. I really appreciate the work done by the FPC and Lazarus-Team. It is a great piece of work and I think it will have a great future. It is out of that thinking that I would like to contribute my small part to the project.

But M. van de Voort, I will not continue the discussion on this level and in this tone.

My first intention was to improve LConvEncoding. I still think this functionality has to be in the RTL, but I also said at the beginning of this threat that it is to the core-developers to decide if it can be integrated there. Mattias Gaertner approved to this, and even named a COMPLETE conversion unit in its post. Felipe Monteiro de Carvalho also agreed.

If now others think that this is not wanted, no problem for me, the unit may stay in the LCL, I can live with that very well.

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

i don't see that as an "order"... more like "this is FOSS. you are free to fix things that you see and contribute them back to the community." ;)

I want to contribute to the project with my knowledge of more than 20 years programming in Pascal (since the very first days of Turpo Pascal).

ahhh... another oldtimer amongst us :P

I surely will not waste my time on completing an algorhytm that I think is unappropriate for the problem!

FWIW: *I* can understand that :)

that would depend on the "fix" wouldn't it? it also depends on the core "guidance team" and if they accept the fix... remember one goal is to not break existing functionality... maybe your fix/enhancement would/should use different unit and include names so as to not break what's already being used in thousands of projects? that way the core team can bring it in gradually and/or existing projects can convert to it on their time and needs ;)

one must also remember and take into account that delphi compatibility is a goal... that FPC and Laz have the option and ability to move further than delphi is a plus but delphi compatibility is still a requirement...

and what happens when delphi does add such capability? will your fix/enhancement be "updated" to match delphi?

on the surface, i'd say "no" but another question that comes to mind is why invest time in it if it goes nowhere?

i tend to agree with this, on the surface... however, much is actually done by providing wrappers so that existing functionality and compatibility can be maintained...

+1

one must remember that FPC's and Laz's smartlinking stuff is nowhere near like that in TP/BP or delphi... but i'm also not sure if that fits with what you are saying, either...

i'm not sure the perceived "tone" is what you seem to take it as...

since this is FOSS, i'd say to move forward keeping the above in mind and not hurting what is already out there... use what you can and write new stuff that may (eventually) replace it... that's the FOSS way from what i've seen over the years :)

What kind of support are you missing?

I don't think so. I even think it is a reason to support it, Delphi does not have full Unicodesupport, FPC will have.

What kind of applications will need such support?

IMO it's perfectly sufficient for 99.999% of all applications, when Unicode text can be stored and displayed.
For mere storage the encoding is irrelevant, or given by database data types.
For display purposes the OS specifies the encoding to use.

Further direct processing of such strings is limited to comparison, search, extraction and concatenation of substrings, what also is possible with every encoding, with no speed penalty. Transformations (upper, lower...) deserve according functions, that are provided by standard libraries, where again the libraries specify the supported encodings. Most such transformation applies *only* to the character based (alphabetic) codepages in the BMP, not to "word" based (Chinese, old Egypt...) codepages.

For all these purposes support of UTF-8 and -16 is perfectly sufficient.

The only place for 4 byte (UTF-32) characters might be an according char type, but the existence of ligatures and other constructs strongly suggest to use strings for storing even single character codes. For the same reason it's *not* wise to iterate through strings by index, instead iterator functions for the next/preceding character index have to be used. Pascal sets of such an char type are impractical, wasting 128MB of memory for *every single* set variable or constant. Does anybody know of an alphabetic codepage with more than 256 character codes?

UTF32 is there in the world, and yes it is wasteful.. And so what? Is that a reason to ignore it?

Please give only a *single* reasonable application, where UTF-32 would result in an improvement over the existing string types and encodings. I cannot remember any single user, who was *really* familiar with full Unicode text manipulation and all related complications, and who wanted to have a native UTF-32 encoding for strings.

The implementation and *continued* support of such additional libraries should be up to companies or (at least) appropriately skilled user groups, familiar with all implemented codepages. Everybody can start such projects, independently from any programming language and compiler. And there is no need that such libraries *must* become part of the core libraries, or that they *must* replace existing libraries. They can be implemented and used as additional libraries as well, and the *users* will judge about their value.

DoDi

waldo kitty wrote

one must also remember and take into account that delphi compatibility is a goal... that FPC and Laz have the option and ability to move further than delphi is a plus but delphi compatibility is still a requirement...

And the downfall of Lazarus LCL.

and what happens when delphi does add such capability? will your fix/enhancement be "updated" to match delphi?

And this is where the "being delphi compatible" pipe dream falls flat on it's face! First we get bitched at for not contributing, then we do, then delphi implements it in a totally different way (sometimes years down the line - and sometimes I think on purpose in a different way), then it is expected from us to break backward compatibility with FPC itself, and rewrite features to be yet again delphi compatible. No innovation - simply follow Delphi like lambs to the slaughterhouse. That is NO WAY to run a FOSS project.

I agree 100% with Graeme. FPC and Lazarus should follow its own path.

Ok, I fully agree. Since I started from LConvEncoding I planned to kee at least the existing interface, so no completly new solution anyway (from the side of the interface), I see no problem to integrate the existing charset interface. Internally it will be completely different, especially faster and smaller (I hope :-))

Thats what I have in mind. The only dependencies I have is the UTF8 and UTF16 functions from LProc. But as I understood it is also forseen to integrate these routines in the RTL. I think this would be the time to do that also and create a proper and complete character-conversion library.

- I've never seen UTF32 files in the wild. I assume however that smartlinking will adequately kill those routines, so it is not that much of a problem unless you use it internally.

If there is support fot UTF16, UTF32 does not really blow up the code. 2 lines of code to decode surrogate pairs, thats it.

I know that. The core-unit witchsupports UTF8, UTF16 and UTF32 does not need tables, so size is no matter there.

Support for codepages will be in separate units which will plugin to the core-unit. I still prefer a one unit per codepage solution, this will give the most flexibility and will not stress smartlinking to much.

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

:P yes, the good ald times, Turbo Pascal on 1 floppy, and the sources on the 2nd :-) and there was this 64KB limit (yes 64, not 640) *g*

Thats what I have in mind. It is clear that existing projects should continue to work...

Why not? But as long as Delphi does not have the capabilties, why should we restrict ourselve?

If you support UTF16, UTF32 is really not a great investment, neither in time nor in resources.

Adding functinalty does not mean that existings things have to been throwne away.

Ok, but you agree that it is easier for the compiler to optmize access to an array of WORD or DWORD, with a known size at compile time, than to a linked list of dynamic arrays of some record type?

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu

Hello all

So I have a first unit up and running. It is not yet finished and not yet bugfree, but it compiles and I have tested the UTF8 part.

This first tests for ISO-8859-1 and ISO-8859-2 showed a performance gain up to 33% compared to LConvEncoding. It should be much more for Doublebyte codepages.

The main work lies not in LConvEncodingNew (just the working name for now) but in ConverCPFile (in subdir Codepages). ConvertCPFile is an application to read a Codepage to Unicode descrition file (formated as those found on unicode.org) and generate an optimized pascal unit which can be compiled and linked in. There are some possibilities to influence the code generated.

My aim for this prerelease is to get echos and suggestions concerning the direcory structure and the naming conventions. Also comments and suggetions for the choosen strategy are welcome.

Greetings

______________________________________________________ powered by GLOBER.LU
Luxembourg Internet Service Provider
Hosting. Domain Registration, Webshops, Webdesign, FreeMail ...

Our professional Web Hosting plans include all the features you are looking for at the best possible price. www.globe.lu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LConvEncodingNew.zip
Type: application/zip
Size: 66179 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101016/9cc02cac/attachment-0001.zip>