hello

i use the substr function to get some truncated string. these string are in french language and sometimes contains "?" or "?" etc.. char
When these charaters are in the string the wrong substr is return substr (pierre ? feu,11) return 'pierre ? f' not 'pierre ? fr'

length('?') return 2
utf8length('?') return 1

is a utf8Subst() like function exist?

thank
Claude

2011/2/10 claude Pomalo :

http://lazarus-ccr.sourceforge.net/docs/lcl/lclproc/utf8copy.html

Vincent

Op 2011-02-10 12:27, claude Pomalo het geskryf:

length('?') return 2

Length() returns bytes

utf8length('?') return 1

this returns the number of visual characters.

is a utf8Subst() like function exist?

utf8copy() somewhere in the LCL units.

Regards,
- Graeme -

It looks to me as if Length returns the byte count, while Utf8Length returns the character count.

is a utf8Subst() like function exist?

I'd use Pos to get the byte index of the part to copy or remove.

DoDi

Do you realize how useless that "documentation" is?

DoDi

Op 2011-02-10 16:24, Hans-Peter Diettrich het geskryf:

> > http://lazarus-ccr.sourceforge.net/docs/lcl/lclproc/utf8copy.html Do you realize how useless that "documentation" is?

:-)
I think Michael van Canneyt got it right with FPC and FCL documentation
- don't publish what isn't documented. Currently code-navigation is more useful than those pages.

I would love to know what MakeMinMax() does (I did not look at the code)? From the procedure name itself, it doesn't give a lot of clues. I would guess it returns some Min and Max values (based on the var parameters), but from what source? :-)

http://lazarus-ccr.sourceforge.net/docs/lcl/lclproc/makeminmax.html

Regards,
- Graeme -

Op 2011-02-10 16:20, Hans-Peter Diettrich het geskryf:

> is a utf8Subst() like function exist? I'd use Pos to get the byte index of the part to copy or remove.

Make that UTF8Pos() (in combination with UTF8Copy) because a UTF-8 character can be anything from 1?4 bytes long.

Regards,
- Graeme -

2011/2/10 Hans-Peter Diettrich :

> > http://lazarus-ccr.sourceforge.net/docs/lcl/lclproc/utf8copy.html Do you realize how useless that "documentation" is?

I think it tells people which unit to add to the uses clause. I think that is useful information.

Vincent

Vincent Snijders wrote

I think it tells people which unit to add to the uses clause.

Often enough it does not do that when requested for, as "F1" on an LCL procedure only gets you to that documentation, when the unit already is in the uses clause. =-O

-Michael

Hans-Peter Diettrich wrote

> length('?') return 2 > utf8length('?') return 1

I thinks according to the definition of UTF8String it's correct that Length(s) provides the byte count. I do hope that with "NewStrings" this some day might change, as it's quite confusing for anybody who does not want to be bothered with the Uniocde internals.

-Michael

Graeme Geldenhuys wrote

Make that UTF8Pos() (in combination with UTF8Copy) because a UTF-8 character can be anything from 1?4 bytes long.

Obviously UTF8Pos() and UTF8Copy() are a lot slower than Pos() and Copy(), and in many cases (i.e when that arguments of copy are obtained by Pos(), Length, ... ) Copy does what is requested. If single characters are the point of interest, this will not work. (Some easy to find and to read entry-level documentation on this would be nice, though :-) .)

-Michael

Hello Lazarus-List,

you wrote

> Make that UTF8Pos() (in combination with UTF8Copy) because a UTF-8 > character can be anything from 1?4 bytes long.

MS> Obviously UTF8Pos() and UTF8Copy() are a lot slower than Pos() and MS> Copy(), and in many cases (i.e when that arguments of copy are obtained

If no checks about utf8 integrity are performed they should not be that "lot slower", only a bit slower, at least utf8pos, utf8copy is for sure slower.
A different thing is that current implementation is a bit overengined which add some overhead.

Is it logical/safe that utf8 functions do not check utf8 integrity ? I'm talking about utf8pos, utf8copy, etc...

Length() is bound to the physical (array) size, a redefinition would break this established rule.

MBCS users had to live with this problem since ever, and UTF-8 is a MBCS. I'm not sure whether the difference between number of characters (glyphs) and number of codepoints can be eliminated by any approved convention.

IMO it's a good idea to forget about "char" in dealing with Unicode/UTF strings, and only use (sub)strings. This is not a major problem, since Pascal does not distinguish between char and string literals.

Obviously this code will fail with UTF-8 encoding:

   var a: char = '?'; //or '`a'?

and even UTF-32 may fail with ligatures or other character combinations.

Some "NewStrings" model IMO should at least distinguish between ASCII, ANSI and UTF strings:

ASCII: never convert, codes above #$7F are undefined (maybe raw data). ANSI: SBCS according to a specific codepage.
UCS2: a possible Unicode subset (BMP) of 2-byte (WideChar) characters. UTF: anything else, with unrelated character and byte counts.

This would make at least those coders happy, that are used to deal with SBCS, and writing applications for local/national use. All coders, in detail the English (ASCII) speakers, have to learn about UTF and MBCS when dealing with UTF strings (apart from assignment and display).

DoDi

Maybe make the sanity check optional with default true ? Or some unit flag, default true and the utf* routines could force check if told so ?

Not that I know anything about this code but why not let people who know what they are doing to skip the check or call check when they know they need to? (and as I said, in order not to break existing code, default would be to check always)

My ?0.000002.

Lukasz

Hans-Peter Diettrich wrote

Some "NewStrings" model IMO should at least distinguish between ASCII, ANSI and UTF strings:

With a future "NewStrings" implementation I mean a dynamically coded string typed that can hold e.g. "ASCII code page xxxx", "UTF8", "UTF16", or "UTF32" content and knows about what is stored and how. So "Length" with this type can be defined as "character count" and copy can work on character length and position, and automatically convert strings if they are coded differently.

Of course certain operations might be really slow if the encoding of the data is not appropriate.

-Michael

With a future "NewStrings" implementation I mean a dynamically coded string typed that can hold e.g. "ASCII code page xxxx", "UTF8", "UTF16", or "UTF32" content and knows about what is stored and how.

How would you determine the byte count for reading and writing text?

So "Length" with this type can be defined as "character count" and copy can work on character length and position, and automatically convert strings if they are coded differently.

I don't like automatic string conversion, because:

Of course certain operations might be really slow if the encoding of the data is not appropriate.

Consider what will happen when every procedure or component has its *own* idea of the "appropriate" encoding...

DoDi

Hello Lazarus-List,

you wrote

> Is it logical/safe that utf8 functions do not check utf8 integrity ? > I'm talking about utf8pos, utf8copy, etc...

LS> Maybe make the sanity check optional with default true ? LS> Or some unit flag, default true and the utf* routines could force check LS> if told so ?
LS> Not that I know anything about this code but why not let people who LS> know what they are doing to skip the check or call check when they know they need to? LS> (and as I said, in order not to break existing code, default would be to check always)

Current code does not perform sanity check, and I think most functions should not perform it, only conversion functions and a "sanitize" function should perform the checks, otherwise most functions will degradate in speed even when you know that the data is utf8 compliant. The same applies to UTF16.

If no checks about utf8 integrity are performed they should not be that "lot slower", only a bit slower, at least utf8pos, utf8copy is for sure slower.

I see no need for integrity checks, when the procedures are called with reasonable arguments. Before e.g. Copy can be called, the required parameters have to be determined, and *this* is where the use of the appropriate functions will automatically return valid arguments.

There exists no need for an utf8pos function, for use with an utf8copy, when Pos already returns the correct start index for Copy. Only the count parameter deserves different handling in utf8copy - where the determination of the byte count can be done once, e.g. in an (UTF8)ByteCount function. Then Copy can allocate immediately the requested number of bytes, then move the same number of bytes. The use of the ByteCount function is not required when the end index is already known, from e.g. another Pos call.

It also would help to ensure text integrity when indexed access to bytes/chars in (MBCS/UTF) strings simply would be dropped. Then either a different string type or different access methods have to be used, at the choice of the coder.

DoDi

Hans-Peter Diettrich wrote

How would you determine the byte count for reading and writing text?

e.g. when using a stream. good question. As, AFAIK, this is no more than a yet incomplete project in the svn, I don't know.

As always, comfort can be traded against speed. If the user wants speed he needs to take care that as few conversions as possible are done.

If he just uses this string type and does not explicitly enforce encoding no encoding is necessary but on exit and entry of his code. And the same code will work without re-coding for all codes used and entry and exit, provided they all are identical.

E.g. the Windows System API will use UTF-16, while the Linux System API uses UTF-8 for things like "caption" and "Text". The (even binary) unmodified user code will not need to do conversions for this kind of GUI work. (AFAIK, string constants are re-encoded on the first use, if necessary).

-Michael

Hello Lazarus-List,

you wrote

> Is it logical/safe that utf8 functions do not check utf8 integrity ? > I'm talking about utf8pos, utf8copy, etc...

HPD> There exists no need for an utf8pos function, for use with an utf8copy,

Nothing is needed for utf8copy ;) utf8pos is needed to return the "characters" position of an string it use for utf8copy or to display the information somewhere is a different matter.

HPD> when Pos already returns the correct start index for Copy. Only the

If not integrity checks are performed, yes, it returns a valid index position.

The current thicket of different character encodings should be reduced to the unavoidable minimum. I think the decision for UTF8 is very good. It is unicode based, has the least impact on memory and all other encondings have multiple bytes for characters too (with the drawback of increased memory consumption). The only reason to convert to other encodings (like ANSI) should be to satisfy non-UTF8 inferface requirements (like OS API).

And I believe that we need *two* string functions for index calculations: NumberOfBytes to determine the number of bytes (as Length does now) and NumberOfCharacters that reports the number of characters. Then everybody has to realize that in general NumberOfBytes is faster to calculate so it should be prefered (if the calculation of the number of characters is not required).

So for me the ultimate string type is UTF8 stored like ansistrings plus new functions to retrieve NumberOfBytes and NumberOfCharacters (even if Length already does the first task the new name would make its purpose much clearer).

Why determine an UTF-8 character index, when this one has to be converted later into an byte (char) index inside utf8copy?

DoDi

Hello Lazarus-List,

you wrote

> Nothing is needed for utf8copy ;) utf8pos is needed to return the > "characters" position of an string it use for utf8copy or to display > the information somewhere is a different matter.

HPD> Why determine an UTF-8 character index, when this one has to be HPD> converted later into an byte (char) index inside utf8copy?

Please, re-read my comment ( fixing my "it use" by "if used" :) ). The result is a character position index, not a need for use copy, but maybe for an information display.

J?rgen Hestermann wrote

I think the decision for UTF8 is very good.

AFAIK, the decision to use UTF8 is due to Linux using this encoding and so no conversion is done in the LCL system API. This of course is bad with Windows, as here the API uses UTF16 and everything needs to be recoded in the LC System API on entry and exit. Supposedly doing different string types - UTF8String vs (a reference counting version of UTF-16-encoded) WideString - for Linux and Windows at the LCL-user-Code interface is too confusing.

-Michael

On 02/11/2011 06:44 PM, J?rgen Hestermann wrote: > I think the decision for UTF8 is very good. AFAIK, the decision to use UTF8 is due to Linux using this encoding and so no conversion is done in the LCL system API.

No, that was just a nice goody.
The decision was made at a time where many Linux
distributions still use ISO character sets and most Windows used UCS-2.

UTF-8 was chosen, because the LCL should use only one string type for easy usage, UTF-8 supports the whole unicode range, there was no reference counted widestring in FPC and porting existing code is easier with UTF-8.

This of course is bad with Windows, as here the API uses UTF16 and everything needs to be recoded in the LC System API on entry and exit.

In almost all cases the overhead is insignificant compared to the GUI. For non gui tasks the overhead may be a problem, but that has nothing to do with the LCL.

Supposedly doing different string types - UTF8String vs (a reference counting version of UTF-16-encoded) WideString - for Linux and Windows at the LCL-user-Code interface is too confusing.

Mattias

AFAIK, the decision to use UTF8 is due to Linux using this encoding and so no conversion is done in the LCL system API.

IMO more important: no new string and char type (Wide...) is required, no duplicate set of stringhandling procedures. This may be essential for databases and communication as well.

This of course is bad with Windows, as here the API uses UTF16 and everything needs to be recoded in the LC System API on entry and exit.

The overhead may be neglectable in direct API calls, when these do real work. Strings in (visual) components can be converted once, into the internally used (OS display conforming) representation, and again the conversion overhead can be low until undetectable in the GUI.

Supposedly doing different string types - UTF8String vs (a reference counting version of UTF-16-encoded) WideString - for Linux and Windows at the LCL-user-Code interface is too confusing.

A *portable* UTF string implementation should be restricted, eliminating direct and indexed access to chars (which become substrings). A dedicated UTF16 class/type can be added at any time, as an optional package.

OTOH I agree that the weak (non-existing) distinction between Ansi and UTF8 strings is not pleasing. But here I'd establish a strong boundary between general (Unicode=UTF8) strings, and application specific strings of a single (immutable) codepage - remember that "Ansi" is not a single specific encoding, instead it's a collection of single-byte-char encodings, including UTF-8. Then the user can choose a specific codepage (or UTF-16) for use inside his application, with e.g. an AppString type. Then it's clear where conversions are required and have to be inserted automatically by the compiler.

The Delphi model, with differently encoded strings in the same string type, can result in much uncontrollable conversion overhead, easily outweighting the few possible optimizations with current AnsiStrings (assuming SBCS[1] only). The new ABI also is incompatible with existing DLLs of earlier Delphi/BCB versions, causing trouble with third-party components that are not available in the new ABI. Okay, no such problems exist with open source components, but not all Lazarus add-ons or apps are necessarily open source.

[1] With MBCS charsets the same rules apply as to UTF-8, so that UTF-8 can immediately replace all MBCS encodings. So the decision about new string types *only* affects current SBCS Ansi users, even ASCII users are not affected.

DoDi