Hello,

There was a discussion among lazarus developers about what kind of problems adding the cwstrings unit (libc widestring manager) for all UNIXes could bring.

Are there any cases where it would not work, or fail to link? For example with uclibc? Or does it always work if any libc is present?

thanks,

In our previous episode, Felipe Monteiro de Carvalho said:

There was a discussion among lazarus developers about what kind of problems adding the cwstrings unit (libc widestring manager) for all UNIXes could bring.

Adding to what ? Where? For GUI apps only it should be ok.

Are there any cases where it would not work, or fail to link? For example with uclibc? Or does it always work if any libc is present?

It doesn't require libc but iconv, Which happens to be in glibc, but not in others like BSD.

I don't know what uclibc does. But cwstrings would require adaptation if it doesn't, but OTOH how many GUI apps are made for uclibc?

Marco van de Voort wrote

Adding to what ? Where? For GUI apps only it should be ok.

To the LCL.

I don't know what uclibc does. But cwstrings would require adaptation if it doesn't, but OTOH how many GUI apps are made for uclibc?

Google for ulibc qtopia. It seams to be possible with Qtopia.

In our previous episode, Felipe Monteiro de Carvalho said:

> Adding to what ? Where? For GUI apps only it should be ok. To the LCL.

I assume that means graphical apps.

> I don't know what uclibc does. But cwstrings would require adaptation if it > doesn't, but OTOH how many GUI apps are made for uclibc? Google for ulibc qtopia. It seams to be possible with Qtopia.

That's not the question. Also that doesn't say much since they probably use configure logic to sort the differences out. And might have unicode disabled?

The question is if you want to support it, and who is going to do it. That person will have to find out if iconv is in uclibc or not. If not so, you'll also need to link iconv.

Felipe Monteiro de Carvalho wrote

There was a discussion among lazarus developers about what kind of problems adding the cwstrings unit (libc widestring manager) for all UNIXes could bring.

I don't know the widestring manager "cwstrings", so my question might sound silly. Why is there a libc requirement for widestring support? Isn't widestring support implemented in Object Pascal? Like UTF-8 used in LCL or fpGUI?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

Afaik all unicode support is based on iconv. Sooner or later.

Marco van de Voort wrote

Afaik all unicode support is based on iconv. Sooner or later.

What is iconv? I guess nothing to do with icons? :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

1) Go to google.
2) type in iconv
3) press "I'm feeling lucky"

After I sent the email I did: man iconv

iconv - Convert encoding of given files from one encoding to another

I still don't see why you need to rely on 3rd Party libraries to do character encoding conversions?

Maybe I don't understand the details that clearly, but is it really that hard to convert a UTF-8 string to a UTF-16 string or vis-versa? Surely such conversions could be implement in Object Pascal and then be totally cross-platform - not relying on iconv in Unix/Linux/*BSD or Cygwin under Windows etc..

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Take a look at what a widestring manager must implement:

http://www.freepascal.org/docs-html/rtl/system/twidestringmanager.html

The hardest parts to implement ourselves are ansi to widestring (because there are dozens of possible encodings for ansi) and uppercase/lowercase.

OK, but out of the list of encodings supported by iconv - how many are actually used. So start with the most used encodings and implement as required. Encoding conversions doesn't need to be perfect from the start.

Also from the iconv help:
"The following list contain all the coded character sets known. This does not necessarily mean that all combinations of these names can be used for the FROM and TO command line parameters. One coded character set can be listed with several different names (aliases)."

Which also means, iconv can't even use all FROM and TO combinations... So even that tool/library isn't perfect.

Am I understanding this correctly. Widestring is more for MBCS that Unicode. MBCS rely on encodings? Unicode strings like UTF-8 and UTF-16 do not have such encoding issues? (amount of possible encodings)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

At least UTF-8,16 and about 20 encodings (all European + Russian ones, and the most popular middle and east Asian languages).

So start with the most used encodings and implement as required. Encoding conversions doesn't need to be perfect from the start.

And everybody has to lug all those tables in their binaries. The point is that iconv is already installed and has those tables.

I see the Pascal only attempts mainly as a solution for embedded systems that don't need iconv, and only have to en/decode a limited set. Not as substitute for iconv.

Am I understanding this correctly. Widestring is more for MBCS that Unicode.

No. Widestring originally was UCS2, which is simplified UTF-16, and has been upgraded to full UTF-16 with some win2000 fixpack.

One can assume that up to date 2000/XP/Vista roughly support UTF-16.

MBCS rely on encodings? Unicode strings like UTF-8 and UTF-16 do not have such encoding issues? (amount of possible encodings)

Afaik MBCS are encoded in ansistring (array of 1 byte chars). It is more UTF-8 but not unicode related. It's more an extension of the codepage system to have codepages larger than 256 items for eastern Asian use.

Once you have unicode, you probably don't need MBCS except for import of legacy text in the Far East. And that doesn't need to be integrated into the system, but can be handled by custom systems. (depends on the complexity)

Hello Felipe,

Monday, September 15, 2008, 2:33:22 PM, you wrote:

FMdC> Take a look at what a widestring manager must implement:

FMdC> http://www.freepascal.org/docs-html/rtl/system/twidestringmanager.html

FMdC> The hardest parts to implement ourselves are ansi to widestring FMdC> (because there are dozens of possible encodings for ansi) and FMdC> uppercase/lowercase.

I have partially ready upper/lower (generic case, not locale dependent), sametext and some others in early stage like decompose and compose and also some iterators like next word, next line, next paragraph.

About codepage to unicode and viceversa I only have a bunch of them, mostly the 8859-x ones.

The "problem" is that the current code is not ready to be used as it has not been strongly tested. Also decompose and others are even not ready to be tested against the oficial test cases, so I do not know if the implementation is OK.

I'm now trying to reduce the tables size as some are by default too big, but some optimizations can be implemented like fixed offset tables for some upper/lower ranges which will improve speed and space at the same time.

Hello Graeme,

Monday, September 15, 2008, 2:50:16 PM, you wrote:

GG> Am I understanding this correctly. Widestring is more for MBCS that GG> Unicode. MBCS rely on encodings? Unicode strings like UTF-8 and GG> UTF-16 do not have such encoding issues? (amount of possible GG> encodings)

MBCS is table encoding dependent, this means that it is mostly an extension, while UTF-?? is an encoding for a unicode code point without ambiguity about the grapheme across all languages.

You can always go the way ansi -> unicode -> ansi.

(because there are dozens of possible encodings for ansi) and uppercase/lowercase.

Unicode provides tables for this. This is not hard but blows up executables and/or distro size.

load those tables dynamically - it's even possible to write a parser for the original tables from Unicode ;) (i did)

JoshyFun wrote

I have partially ready upper/lower (generic case, not locale dependent), sametext and some others in early stage like decompose and compose and also some iterators like next word, next line, next paragraph.

Are you building a pascal substitute for cwstrings?

Do you have your code publicly available?

In our previous episode, Ivo Steinmann said:

Even worse. Files to distribute with each binary, more installation support etc.