Hello,

Today I took a look at what would be necessary to implement controlling the output of the widestring manager, and implementing this looks rather trivial.

The only thing we need is a switch, to control if widestrings should be converted to the current locale or to utf-8. In the case of current locale one can use the current widestring function Wide2AnsiMoveProc . In case of utf-8 we can use the cross-platform UnicodeToUTF8 function.

I would start by adding a new field to TWideStringManager:

OutputUTF8ToStrings: Boolean;

Then when converting a widestring to a short string one should check which procedure to call.

I could check all places that call Wide2AnsiMoveProc and instead do:

if OutputUTF8ToStrings then UnicodeToUTF8()
else Wide2AnsiMoveProc ()

Or we could add a procedure which does this automatically. Or maybe something else.

So, which would be the prefered way?

thanks,

Op Tue, 27 Nov 2007, schreef Felipe Monteiro de Carvalho:

A more generic solution is necessary; one can want to output strings in a specific encoding, utf-8 is only one of the specific encodings that are demanded. Therefore a boolean is not a good solution.

Daniël

Daniël Mantione wrote

A more generic solution is necessary; one can want to output strings in a specific encoding, utf-8 is only one of the specific encodings that are demanded. Therefore a boolean is not a good solution.

Win32Wide2Ansi just calls WideCharToMultiByte, which reading ms docs:

http://msdn2.microsoft.com/en-us/library/ms776420.aspx

I couldn't find any way to output to an arbitrary encoding. I don't know if this situation is similar on linux.

So it may be impossible to implement what you suggest in a reasonable way.

Having said that I consider outputing always to an encoding which isn't the current locale and isn't utf-8 a very corner use case, if it has any a very corner case one could use a
special library to achieve it.

thanks,

Op Tue, 27 Nov 2007, schreef Felipe Monteiro de Carvalho:

This function has a code page parameter, which allows you to specify the code page you want to use?

It isn't a corner a text editor that should be able to read/write in specific encodings. Or perhaps you are coding a http client; documents on the web can be in any IANA approved encoding.

Daniël

Daniël Mantione wrote

This function has a code page parameter, which allows you to specify the code page you want to use?

umm ... I missread it and thougth that only the values on the table are possible =)

ok, so:

OutputFormat: TEncoding;
TEncoding = (eCurrentLocale, eUtf8, etc);

or something similar?

Indeed. I am in the designing stage for something similar for the DB components - they need code page remapping kind of functionality too - but ansistrings only, i.e. no widestring.

Michael.

Op Tue, 27 Nov 2007, schreef Felipe Monteiro de Carvalho:

Yes, that looks okay. However, I would use the IANA "cs" aliases as identifier names. Perhaps also use the IBM or Microsoft code page number as binary value, so you would get:

Tencoding = (csASCII             := 367,
             csPC850Multilingual := 850,
             csUTF8              := 65001,
             {...}
            );

Think carefully about the design. One issue will for example be that not every OS will support any encoding, some error handling and detection functionality will be necessary. The Win32 and Unix API's won't always cooperate with any design you think of.

Daniël

The whole point of the widestring manager system that it allows you to replace the default conversion routines with whatever conversion you wish. If you want UTF-8 then just write a set of widestring manager routines that do UTF-8. If you want ones that use the windows conversion APIs (which let you convert to any code page that is installed in the system by specifying it's codepage number) write ones that use the windows coversion APIs. If you want ones that use ICU write ones that use ICU and so on.

Ummm, I didn't think of it this way before, but it could be a solution.

What do other FPC developers think of this?

Should it be the responsability of the fpc user to write it's own routines to override the output of the widestring manager or should FPC provide this mechanism?

thanks,

Op Fri, 30 Nov 2007, schreef Felipe Monteiro de Carvalho:

The default widestring manager does not fullfill in all conversion needs and should be expanded to give users more control over conversions.

However, when expanding the functionality it should be done properly, not in a way that it will have to be changed over and over again. Not all functionality needs to work at once, if you just want to support UTF-8 in the beginning this is okay.

The API, however, should be solid and future proof, as it needs to be backward compatible.

Daniël

As discussed on irc today, this is a no go for the rtl. Ansistrings are encoded in the system encoding, this is a convention everything depends on like all Ansi* routines in sysutils. I give the bug report "I added widestringmanager.outputformat := soUTF8; to my program, now AnsiStrUpper doesn't work anymore", 1-2 weeks. It is fine if projects break with the convention, but then they have to deal with it by e.g. by overriding ansi<->wide conversion in the widestring manager.

The full discussion can be found here:
<http://www.hu.freepascal.org/fpcircbot/cgifpcbot?channel=lazarus-ide&fromdate=12-1-07&todate=12-1-07&linecount=9999&fromtime=10%3A00&totime=12%3A45&sender=&msg=>

Since we are talking about expanding the widestringmanager, I'm considering adding mblen. This because I encountered people using unit libc for it a few times. It is also a needed primitive to safely walk a string with any encoding style that has non fixed codepoint size.

FreeBSD and Linux support it, so OS X still needs to be checked. Windows is probably no problem, since the use probably originates in Delphi (sysutils(?).strnext).

I agree with Florian that adding a global conversion controlling boolean is a hack. From day 1 when we (=Lazarus) started to use utf8 I wanted a separate string type for that (with the default answer that I always can add this myself).
Mixing the meaning of the contents of a string is hard to maintain. This is error prone and not really a "typesafe" pascal way. From the logs I understand that there is a "real" managed utf8string type (other than utf8string = type string).
If this is the case I am pro using this type as container for utf8 encodings and leave ansi strings for system encodings as they were meant. FOr portability reasons, it would be great if it is possible to control what type the string type defaults to. Similar to the {$H+} which switches between shortstring and ansistring I think something like {$STRINGTYPE SHORT|ANSI|UTF8}

Marc

Marc Weustink wrote

From the logs I understand that there is a "real" managed utf8string type (other than utf8string = type string).

No, currently utf8string = type string is about all support implemented for such type.

Feel free to do so.

Indeed, patches for a utf8string unit are welcome ;)

I'm trying to find the windows eq first. It seems that charnext is only for MBCS, not UTF-8. (and I assume we want UTF-8 ?)

No. The rtl assumes that ansistrings always use the encoding used by the system.

Waht kind of patches are needed ? I just happen to notice that someone is adding unicodestring support ;)
would this be a replacemnt for the utf8strng ?.

Marc

No idea :) Lazarus complained that things are missing.

I just happen to notice that someone is adding unicodestring support ;) would this be a replacemnt for the utf8strng ?.

No. It will be an utf-16 string. On non windows unicodestring=widestring

Aahh... the wide string as the widestring should have been on windows. I got confused, since UTF8 is unicode too :)

Anyway, what I think for Lazarus is need is:
A) a transparent way of assigning from/to other stringtypes, (shortstring/widestring/ansistring), where under the hood the contents is converted.
B) a transparent way of passing as param of other stringtypes, (shortstring/widestring/ansistring), where under the hood the contents is converted.
C) a way of specifying the default type of the string type, to allow easier converiosn of existing codebases

ad A: From what I have seen with type UTF8String = type ansistring it is not possible to do with operator overloading since both types utf8string and ansistring are considered the same

ad B: there is no operator for this, so it acnt be overloaded

So imo, a real type is needed for this, but as I understood that would blowup the stringmanager.

Marc

This can be enabled.

Florian Klaempfl wrote

No. The rtl assumes that ansistrings always use the encoding used by the system.

And what if utf-8 is the system encoding? Which is the case on many unixes.

Not to mention that on this case AnsiStrUpper already doesn't work.

Then ansistrings are utf-8 encoded of course. What's the problem with this?

Not to mention that on this case AnsiStrUpper already doesn't work.

Well, unix people don't seem to care about completing the cwstring unit :) BTW: for other encodings it doesn't work either on unix ...

Procedure SetCWideStringManager;
Var
  CWideStringManager : TWideStringManager;
begin
  CWideStringManager:=widestringmanager;
  With CWideStringManager do
    begin
      ;
      ;
      ;
      ;
      ;
      ;
      {
      CharLengthPCharProc

UpperAnsiStringProc
LowerAnsiStringProc
CompareStrAnsiStringProc
CompareTextAnsiStringProc
}

      ;
      {
      StrICompAnsiStringProc
      StrLCompAnsiStringProc
      StrLICompAnsiStringProc
      StrLowerAnsiStringProc
      StrUpperAnsiStringProc
      }
      ;
      ;
    end;
  SetWideStringManager(CWideStringManager);

end;

Some code commented ...

And what about this one ?

will the following code be possible ?

procedure Foo(const S: Utf8String);
begin
end;
var
   X: ANsiString;
begin
   Foo(X);
end.
Will the correct conversion be called to be able to convert X to a
Utf8String ? (or will this be handled by the := operator ?)

What about C ?

Marc

Florian Klaempfl wrote on zo, 02 dec 2007:

Felipe Monteiro de Carvalho schrieb: Well, unix people don't seem to care about completing the cwstring unit :)

At least this unix person didn't know those routines were mandatory, otherwise I would have added them a long time ago. There don't seem to be any tests for them either which demonstrate that they need to be implemented, or which allow testing potential implementations (or the correctness of the Windows implementations).

Jonas

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Likewise. I even suggested last week to add one that is already there.

There don't seem to be any tests for them either which demonstrate that they need to be implemented, or which allow testing potential implementations (or the correctness of the Windows implementations).

That's also my problem.

Making such tests is not easy because they depend on the system locale. Using only latin characters make it work on all systems but with latin only also the default implemenation works and renders the test useless. Using non-latin characters might break because the locale of the test system isn't known. Maybe testing the locale in the tests and abort with a warning but with no error if the test is run on a non 8859-1/utf-8/cp1250 system might be a way to go.

You can make tests for this which work in all locales (although they obviously can't test characters which aren't supported by the used locale), it's just a bit more work. E.g.

{$codepage utf-8}
uses
  {$ifdef unix}
  cwstring,
  {$endif}
   sysutils;
var
   s: ansistring;
   w1,w2,w3: widestring;
   i: longint;
begin
   w:='éèà';
   w2:='ÉÈÀ';
   s:=w;
   w:=AnsiUpperCase(s);
   { filter out unsupported characters }
   for i:=1 to length(w) do
     if w[i]='?' then
       w2[i]:='?';
   if w  w2 then
     halt(1);
end.

Jonas

Florian Klaempfl wrote

BTW: for other encodings it doesn't work either on unix ...

Are widestrings supposed to be UTF-16 or UCS-2 on unix? Currently they are UTF-16, but a lot of routines seem to assume UCS-2 (like the generic WideStringToUCS4String, and the unix Lower/UpperWideString)

Jonas

It is a bit confusing, but for a while now UCS-2 uses UTF-16 as encoding. So even UCS-2 is nowadays not fixed length.

I guess you mean the UCS-2 standard from the times that each char was 2 bytes.

Marc

Marc Weustink wrote on di, 04 dec 2007:

Yes, I did. Thanks for the clarification, I'll fix everything to work with (variable length) UTF-16 then. Or does Windows still use fixed length UCS-2? And if so, does this UCS-2 use the code points which are reserved for surrogate pairs in UTF-16?

Jonas

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

When trying to compile Zeos lib using FPC 2.2.0 and 2.3.1, the compiler complains with a "Can't find unit ZAbstractRODataset", but the file is there, with exactly that name.

After some minutes trying to figure out what was the problem, I renamed the unit to its lowercase version, "zabstractrodataset.pas" and the compilation went ok.

Compiler and OS versions:
Free Pascal Compiler version 2.2.0 [2007/08/31] for i386 Copyright (c) 1993-2007 by Florian Klaempfl
Target OS: Linux for i386

Regards,

Leonardo M. Ramé
http://leonardorame.blogspot.com

____________________________________________________________________________________ Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs

y

Op Tue, 4 Dec 2007, schreef Jonas Maebe:

Before Windows 2000 Windows used fixed length, after Windows 2000 it was variable length.

And if so, does this UCS-2 use the code points which are reserved for surrogate pairs in UTF-16?

No, the only difference between UCS-2 is that in the original UCS-2 the surrogate code points were reserved, while in UTF-16 they are used.

Daniël

Op Tue, 4 Dec 2007, schreef Jonas Maebe:

This is correct for upper/lower case conversion; alphabets with upper/lower case needs are placed inside the BMP.

Daniël

Daniël Mantione wrote

Before Windows 2000 Windows used fixed length, after Windows 2000 it was variable length.

And what about windows 2000 itself? :-)

Micha