Hi,
For the people who don't know it already, how Unicode is handled in MSEgui for FPC 2.4:

- All GUI relevant string parameters and variables and all DB-Field strings are of type msestring, all characters are of type msechar.

- MSEgui has a set of file handling routines with parameters of type filenamety.

- For FPC 2.4 msestring = UnicodeString, msechar = UnicodeChar, filenamety = msestring.

- The MSEgui sources don't contain characters > #127 and therefore can be stored in any codepage and compiled with any -Fcxxx or even without -Fcxxx. Non ASCII string constants are defined in #xxxx format.

- User source normally is stored in utf-8 and compiled with -Fcutf8 -> user source can contain any string constant and is portable between platforms.

- Conversion between msestring and the system encoding and vice versa is handled transparently by the FPC widestringmanager. So exchange msestring with system routines is easy.

- Database strings are converted to/from msestrings while fetching/sending the data. The MSEgui DB-access components have an option to either use the system encoding by the assistance of widestringmanager or using utf-8.

- INI-textfiles are stored in utf-8 and therfore portable.

The user benefits:

- There is only one encoding in MSEgui framework, it is the same on all platforms.
- In the majority of cases the fast and convenient character access by index can be used. This is important for beginners.
- The current FPC UnicodeString is fast, simple, has little overhead and is well tested.

So please don't destroy this ideal solution by dropping current FPC UnicodeString in favour of the Delphi string which is complicated, less performant, will have many new bugs and is inscrutable by the average user! Please remember how many time it took to fix WideString. I don't want to go this ordeal again.

Martin

- There is only one encoding in MSEgui framework, it is the same on all platforms.

Which is performance wise neither a good decision.

Who says that? If you don't mess with code pages, the only different you'll might see is that UnicodeString gets two new fields: encoding and char size. However, this information is usually only used if you pass the string to a RawString parameters. Normal Unicodestring routines initialize these fields and that's it.

less performant,

In which regard?

will have many new bugs and is inscrutable by the average user! Please remember how many time it took to fix WideString. I don't want to go this ordeal again.

In our previous episode, Martin Schreiber said:

- The current FPC UnicodeString is fast, simple, has little overhead and is well tested.

Do you have any benchmark data on the overhead that delphi adds ? Because I don't think it is much slower if all strings are initialized on the same encoding (like they are on your system)

I'm no compiler devel, so I have no say in this, but even if widestring is retained there might be a significant revalidation. It will significantly turn the whole stringhandling upside down with many new routines and overloads.

I don't see why using D2009 strings would hurt that much.

If there is a compiler option to encode string constants as utf-16 or maybe an additional plain 16 Bit String type, that encoding would automatically be used for the "Strings" and thus all have the same encodings and in fact are reference counting WideStrings.

The string handling functions would need to compare the character size and encoding (both are stored in a 32 bit word), see that they are equal and now call the original wide string handling functions.

So maybe we just additionally need that "plain 16 Bit" encoding type for the D2009 Strings.

-Michael

Could be, if are writing a specialized command line tool or something like that.

But for GUI software the conversion speed impact is tiny, and the head-ache caused be a string type whose encoding is not know is by far much worse. Tons of string related code will stop working and need to be adapted at the gain of a couple of microseconds.

I don't see why FPC is going in the contrary direction of most of it's users and offering the contrary of what is desired (= fixed encoding strings). Sure you can claim you know better then us what is good for our software, but I don't see how that will end up well.

Further Pascal has always been very easy to use, but now we make it harder to use to gain a couple of CPU clocks, while the global tendency in the world is the opposite. Most inovations out there are designed to make software easier to build, not more performant. Garbage Collection for example, Java's virtual environment, etc. Because it is more important to be easier to use then being high performant for the vast majority of uses. Now is any one pushing into harder to use software but more performant? Does it make sense to do such a thing?

Felipe Monteiro de Carvalho wrote

But for GUI software the conversion speed impact is tiny, and the head-ache caused be a string type whose encoding is not know is by far much worse. Tons of string related code will stop working and need to be adapted at the gain of a couple of microseconds.

Generic unicode strings are not there for speed; their purpose is flexibility.

I don't see why FPC is going in the contrary direction of most of it's users and offering the contrary of what is desired (= fixed encoding strings).

You can use those as well. At least in D2009, you can declare a unicode string with a particular code page, and everything assigned to that string will be converted to that codepage.

Sure you can claim you know better then us what is good for our software, but I don't see how that will end up well.

Currently, the Unix rtl is built around ansistrings. The Mac OS X BSD- level filesystem functions however always require that their input is in UTF-8, and their output is also always in UTF-8. This is regardless of the LANG and of the LC_TYPE environment variables. Input from the keyboard will and to the terminal are of course in whatever encoding you set using those environment variables.

Currently, it is impossible to express this in the RTL. The new string type will make that possible, and generic unicodestrings without a forced type will mean that
a) you can pass in anything with an arbitrary encoding b) the RTL will take care of the necessary code page conversions when necessary (and unnecessary ones can be avoided to some extent)

Jonas

I can confirm there is not much overhead for the new UnicodeString. I was mislead by the Delphi {$stringchecks on} option and a misinterpreted comment from a FPC developer that it is not possible to check codepage compatibility at compiletime, sorry for that.
Some guesswork gained form my experiments with the cpstrnew branch, Win32, Russian locale, source in utf-8, {$codepage utf8}, please correct me if I am wrong:

UnicodeString
- always utf-16 encoded.
- str:= 'abc'; length(str) = 6, stringcodepage(str) = 1200.
- str:= 'abä'; length(str) = 6, stringcodepage(str) = 1200.
- no encoding checks by concanteation, concatenation does not work because of wrong length() value.
- setlength() of empty string creates CP 1200.

UTF8String
- str:= 'abc'; length(str) = 3, stringcodepage(str) = 65001.
- str:= 'abä'; length(str) = 4, stringcodepage(str) = 65001. Runtime widestringmanager.Wide2AnsiMoveProc().
- encoding checked by concatenation.
- utf8string:= utf8string + '123' needs conversion to UnicodeString and back.
- setlength() of empty string creates CP 1251.

String<1251>
- str:= 'abc'; length(str) = 3, stringcodepage(str) = 65001.
- str:= 'abä'; length(str) = 3, stringcodepage(str) = 1251.
  Runtime widestringmanager.Wide2AnsiMoveProc().
- str:= 'abc'; str:= str + '123'; needs conversion to UnicodeString and back.
- setlength() of empty string creates CP 1251.
AnsiString
- str:= 'abc'; length(str) = 3, stringcodepage(str) = 65001.
- str:= 'abä'; length(str) = 0, stringcodepage(str) = 1251.
  Runtime widestringmanager.Wide2AnsiMoveProc().
- str:= 'abc'; str:= str + '123'; needs conversion to UnicodeString and back.
- setlength() of empty string creates CP 1251.
RawByteString
- str:= 'abc'; length(str) = 3, stringcodepage(str) = 65001.
- str:= 'abä'; length(str) = 0, stringcodepage(str) = 1251.
  Runtime widestringmanager.Wide2AnsiMoveProc().
- str:= 'abc'; str:= str + '123'; needs conversion to UnicodeString and back.
- setlength() of empty string creates CP 1251.
- utf8str1:= 'abc'; cp1251str1:= utf8str1; needs conversion to UnicodeString
and back.
- utf8str1:= 'abc'; ansistr1:= utf8str1; no conversion. CP ansistr1 = 65001.
- ansistr1:= 'abc'; utf8str1:= ansistr1; no conversion. CP utf8str1 = 1251.

What are the differences of AnsiString and RawByteString?

Please report when you think cpstrnew branch is stable enough to be tested with MSEgui.

Thanks, Martin

I'am not sure how far these things are already fixed/supposed to work.

What are the differences of AnsiString and RawByteString?

Ansistring: system encoding
RawByteString: variable encoding, cannot be checked at compile time, when working with RawByteStrings, you've to take care of the newly introduced encoding field

Florian Klaempfl wrote

> What are the differences of AnsiString and RawByteString? Ansistring: system encoding

System encoding at compile time or run time?

Runtime, as it is currently.

Florian Klaempfl wrote

Ansistring: system encoding RawByteString: variable encoding, cannot be checked at compile time, when working with RawByteStrings, you've to take care of the newly introduced encoding field

Let's say there's a library that works with AnsiString as a string type with no encoding (as it's now).
Let's assume that the new branche is officially accepted. The library doesn't want the new feature, but it also doesn't want to change its code from AnsiString to RawByteString.

Would it be possible to make the following type declaration:

unit MainLibTypes;
type
  AnsiString = RawByteString;

to avoid using new auto-converters?

thanks,
dmitry

Well, an ansistring has an implicit encoding: system. This won't change and if one uses only unicodestrings and ansistrings, no change will be noticed.

No, because it would avoid auto-conversion from/to unicodestring as well.

Florian Klaempfl wrote

Well, an ansistring has an implicit encoding: system. This won't change and if one uses only unicodestrings and ansistrings, no change will be noticed.

Thanks. It's always nice to hear "No changes required" :)

No, because it would avoid auto-conversion from/to unicodestring as well.

does your answer means, that:
var
  r : RawByteString;
  w : WideString;
r:=w; // is impossible because of unknown encoding of RawByteString?

thanks,
dmitry

Well, I guess you're talking about lazarus ;) I don't see yet however, if and how the ansistring abuse in the lcl will bite one ...

No, it is possible becuase the encoding is known because it is stored in RawByteString, but the appropriate conversion will be selected at runtime.

Florian Klaempfl wrote

Well, I guess you're talking about lazarus ;) I don't see yet however, if and how the ansistring abuse in the lcl will bite one ...

not really. But it's good to know that LCL is safe as well :)

thanks,
dmitry

Length needs to be fixed. I know about the bug but had not looked at it yet.

What are the differences of AnsiString and RawByteString?

AnsiString has code page assigned to current system encoding. You can change it though using SetMultiByteConversionCodePage routine. RawByteString has no encoding. Therefore assigning to AnsiString converts right operand to AnsiString encoding first. Assigning to RawByteString does not invoke any conversion and as result of assigning RawByteString left operand will get the encoding or right operand.

Best regards,
Paul Ishenin.

Florian Klaempfl wrote

Well, I guess you're talking about lazarus ;) I don't see yet however, if and how the ansistring abuse in the lcl will bite one ...

Why should the (future) lcl use "ANSIString". I suppose using ANSIString for UTF-8 data was just a nasty hack to do Unicode with the _current_ FPC. (In contrast the MSE-GUI hack was using Widestring, which is less ambiguous and seems to work better on Windows, but needs more conversions on Linux.)

Of course the future LCL should use the dynamically encodes strings to allow for using utf-16 on Windows and utf-8 on Linux with the same source code. There should be appropriate means to define the string constants (in the LCL and in the user program) to be stored so that conversions are avoided (maybe on Linux, for optimization, the user would need to have the string constants meant for the GUI saved as UTF-8 and those meant for database handling in utf-16 <I'm quite illiterate with db-programming> ).

-Michael

On Linux xlib and xft have utf-16 interfaces.

Martin Schreiber wrote

> On Linux xlib and xft have utf-16 interfaces.

What exactly are xlib and xft and why does MSE-GUI seemingly use those while LCL seemingly uses something else ?

-Michael

xlib is the C-library used under Linux to communicate with the X-server, X-server is the graphic and user interface (keyboard, mouse...) engine for a common Linux installation. xft is a library for advanced font handling and rendering, based on xlib and xrender. xrender is an advanced rendering engine extension for the X-server. xft can render antialiased fonts for example.

Lazarus - contrary to fpGUI and MSEgui - does not talk to the X-server directly but uses an intermediate layer, the "widgetset". Some of the widgetsets have an utf-16 interface, the Lazarus main widgetset (GTK2) has an utf-8 interface AFAIK.

Martin

Martin answered this already. But just to be clear, xlib and xft have both utf-8 and utf-16 interfaces. fpGUI uses the utf-8 interfaces, whereas MSEgui uses the utf-16 interfaces.

Regards,
- Graeme -

Often strings are converted to UCS4 somewhere in the chain anyway so it doesn't matter. utf-16->UCS4 can be a little faster maybe.

Martin

Zitat von Martin Schreiber :

For graphic apps the text conversion is hardly measurable. UTF-16 is not sufficient under linux for file names, because Linux uses UTF-8 and allows invalid characters. This happens often when transferring files from Windows.
That's why the RTL file operations must use the system encoding under Linux and not UTF-16.
Afaik Mac OS X does not allow invalid UTF-8 characters for filenames. Windows has its codepages, with characters that have no unicode equivalent. But afaik this is a rare case, so for almost all programs UTF-16 is sufficient.

Mattias

So MSE-GUI creates it's own Widget set instead of using something like GTK. Is this really advantageous ?

-Michael

Michael Schnell wrote

So MSE-GUI creates it's own Widget set instead of using something like GTK. Is this really advantageous ?

Oh boy, this promises to start a long thread...

... which Jonas is likely to end if he gets fed up with it :-)

Michael.

2009/11/13 Michael Van Canneyt :

If Martin had a euro for every time someone asked him this, he'd be a very rich man, indeed :)

Henry

Michael Schnell wrote

So MSE-GUI creates it's own Widget set instead of using something like GTK. Is this really advantageous ?

Michael, you are joking, right? ;-)
Please join us on NNTP:
news://news.grid-sky.com/public.mseide-msegui.talk

to get answers to the question.

Martin

Well, at least one or two views out of many.

Michael.

I see that this should not be discussed here :) . Thanks ! -Michael

I did not yet try or consider to use the "GUI" part of MSEGUJI, as my intention is about doing embedded stuff (e.g. with NoGUIApplication). So sorry for may ignorance about this seemingly already thoroughly discussed issue.

Michael Schnell wrote

So MSE-GUI creates it's own Widget set instead of using something like GTK. Is this really advantageous ?

Definitely - depending on your needs. That's the whole reason we started fpGUI Toolkit as well. LCL, GTK, etc simply did not do what we needed. In creating our own GUI toolkit, we could implement everything that was required and get same behaviour and look on all supported platforms (another requirement for our projects).

Regards,
- Graeme -

I see.
Thanks,
-Michael