Today I tried Lazarus 1.6 RC1 (on Windows 7 32 bit).

The installation hat a problem with missing access rights. I chose to uninstall the previous version but afterwards some folders of fpc 2.4 still existed and I did not have access rights to these folders. I am not sure whether this was caused by Lazarus or by the crappy rights management of Windows.
Well, in the end I could solve the problem.

But now I have a problem with UTF8Strings:
With this declaration

var S : UTF8String;

I want to delete a character

UTF8Delete(S,1,1);

but I get an error that the (var) parameter mismatches. S is UTF8String but expected is AnsiString (I thought that an AnsiString on its own no longer exists but that it has to be declared with a codepage).
UTF8Delete is declared with a "string" parameter (I hate these chameleons as I never know what they realy mean in a certain case).

My question:
How can I use UTF8Delete on an UTF8string?

J?rgen Hestermann wrote

How can I use UTF8Delete on an UTF8string?

You can't. Please read this :
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus "String" type is UTF-8 and it works now (almost) transparently without explicit conversions.

Juha

Then why does the compiler complain when I
feed UTF8Delete with an UTF8String?

J?rgen Hestermann wrote

Then why does the compiler complain when I feed UTF8Delete with an UTF8String?

As it told you. It got UTF8String but expected AnsiString. I think it would work with a typecast but that is quite useless. Just use a String instead.
In fact the Unicode support is now almost Delphi compatible at source level. Only code that depends on system codepage needs special attention.

Juha

Why do you insist in using UTF8Delete? Use just Delete:

var
   xUF: UTF8String;
begin
   Delete(xUF, 1, 1);
end;

Ondrej

Sorry, of course you want to specify position/length in real chars.

Ondrej

What if I compile my application with DisableUTF8RTL or (in the future) I compile with String=UnicodeString (utf-16) and I still need to use UTF8String with UTF8* functions (e.g. because an external library uses it)?

IMO, there should be overloaded versions for UTF8* functions that explicitely accept UTF8String.
Or am I wrong?

Ondrej

Ondrej Pokorny wrote

IMO, there should be overloaded versions for UTF8* functions that explicitely accept UTF8String. Or am I wrong?

You are right. I did not even think so far yet.
A typecast can be used as a workaround now.
UTF8String and AnsiString have the same memory layout so it should work.

Juha

This has nothing to do with the memory layout. Both Utf8String and AnsiString are AnsiString types and an assignment to each other or a typecast will result in a codepage conversion if necessary.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151210/642aee88/attachment.html>

What about:

UTF8Delete(AnsiString(Pointer(s)),1,1);

?

Mattias

UTF8Delete probably takes an AnsiString (or String) as var parameter and for var parameters the static codepages have to match exactly (String has CP_ACP while Utf8String has CP_UTF8).

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/57fc8055/attachment.html>

work.

While the typecast itself would probably work I strongly advice against it since you're relying on implementation details. Also I doubt that you can do this for var parameters...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/4ed3db24/attachment.html>

It is difficult to make efficient Free Pascal applications without such "tricks". ;-)

Martin

An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/d83bcd46/attachment.html>

it

> since you're relying on implementation details. Also I doubt that you

can

> do this for var parameters... > It is difficult to make efficient Free Pascal applications without such "tricks". ;-)

But then don't come and complain if such tricks don't work anymore for some reason or another...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/e555ba16/attachment.html>

IMO it's a valid issue. Please create a feature request in the bug tracker and - best - create a patch that introduces function overloads that accept UTF8Strings.

Ondrej
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/c6900f39/attachment.html>

UTF8Delete probably takes an AnsiString (or String) as var parameter and for var parameters the static codepages have to match exactly (String has CP_ACP while Utf8String has CP_UTF8). Just please help me understanding this: The unit LazUTF8 unit is for manipulating UTF8 strings only, correct?

Yes.

Is there any reason that all functions are declared with parameter type STRING instead of UTF8String?

Yes, because the FCL uses String.
See below.

For what else should I use UTF8delete if not for UTF8strings?

For "UTF8 strings".
An "UTF8String" and an "UTF8 String" are two different things for the compiler. See below.

And what happens if the string has a different encoding than UTF8? I doubt that the function will produce any useful result in this case because it expects UTF8.

Yes, invalid for chars above #127.

See here:
http://wiki.lazarus.freepascal.org/Better_Unicode_Support_in_Lazarus#Why_not_use_UTF8String_in_Lazarus.3F

Mattias

True.

Also I doubt that you can do this for var parameters...

FPC 3.0 eats it without a hint.

Mattias

An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/be35ea3d/attachment.html>

It does not work though. Interesting.
Using the new default UTF-8 system for LCL apps and FPC 3.0 on Linux, this works :

var
  uStr1, uStr2: String;
begin
  uStr1 := '???';
  uStr2 := uStr1;
  UTF8Delete(uStr2,1,1);
  ShowMessage('Original: ' + uStr1 + ',   Deleted: ' + uStr2);

but this shows garbage :

var
  uStr1, uStr2: UTF8String;
begin
  uStr1 := '???';
  uStr2 := uStr1;
  UTF8Delete(AnsiString(Pointer(uStr2)),1,1);
  ShowMessage('Original: ' + uStr1 + ',   Deleted: ' + uStr2);

Changing the last CharCount parameter for UTF8Delete makes no difference.

I was planning to add overloaded versions for procedures taking var parameters in LazUTF8. Copy/pasting code may be the only way to go.

Juha

"J?rgen Hestermann" wrote

What is the difference? The link does not tell me.

Did you actually read the wiki-page? The new UTF-8 system is explained there. It is a hack but a clever one.
The latest chapter by Mattias explains why UTFString should not be used now.

FYI Ondrej, having overloaded versions only for procedures taking var parameters makes no sense even when using DisableUTF8RTL. Lossy conversion would happen in every function using "String" type. Instead all functions must be copied and UTF8String must be used everywhere. The current functions make sense only with our new UTF-8 system which changes AnsiString encoding to UTF-8.

Juha

An "UTF8 String" is a String encoded in UTF-8. String and UTF8String are two different things for the compiler. My link explains some differences important for Lazarus.
For full technical differences read
http://wiki.lazarus.freepascal.org/FPC_Unicode_support and
http://wiki.freepascal.org/Character_and_string_types

The string codepage attribute was introduced for Delphi compatibility. The encoding hassle is built into Windows. The compiler cannot overcome it. It can only give you tools to make it easier.

Mattias

Without DisableUTF8RTL you can pass an UTF8String to a non "var" String parameter. The compiler will add some checks, but they don't change the encoding. Same for return values String to UTF8String.

With DisableUTF8RTL passing UTF8String to String changes encoding. So every function needs an overloaded wrapper function. No need to copy the implementation.

Mattias

The above literal requires {$codepage UTF8}.

uStr2 := uStr1; UTF8Delete(AnsiString(Pointer(uStr2)),1,1); ShowMessage('Original: ' + uStr1 + ', Deleted: ' + uStr2);

Works here.
Also on Windows.

Changing the last CharCount parameter for UTF8Delete makes no difference.

Works here.

I was planning to add overloaded versions for procedures taking var parameters in LazUTF8. Copy/pasting code may be the only way to go.

Mattias

Mattias Gaertner wrote

With DisableUTF8RTL passing UTF8String to String changes encoding.

Yes and a conversion between UTF-8 and a system codepage can be lossy.

So every function needs an overloaded wrapper function. No need to copy the implementation.

The functions then use String with system codepage. A wrapper would trigger useless conversions and potentially loose data. No, UTF8String really must be used in the implementation.

Juha

Mattias Gaertner wrote

The above literal requires {$codepage UTF8}.

Damn right, I forgot to test that.
It works.

Juha

[...] The functions then use String with system codepage. A wrapper would trigger useless conversions and potentially loose data. No, UTF8String really must be used in the implementation.

The job of the wrapper is to convert to type String without triggering the conversion of the content.

Mattias

Mattias Gaertner wrote

The job of the wrapper is to convert to type String without triggering the conversion of the content.

Ok, lots of ugly Pointer typecasts. Doable, yes.

Juha

An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/4970cae6/attachment.html>

"J?rgen Hestermann" wrote

I know the type "UTF8String" but what is an "UTF8 String" (which you say differs)?

"UTF8 String" is a String which has UTF-8 encoding. Our UTF-8 system changes the default codepage of String to UTF-8 and then (almost) everything magically works.
Read the wiki page again.

Juha

Mattias Gaertner wrote

http://wiki.freepascal.org/Character_and_string_types

I haven't seen that page yet. That is a brilliant explanation of the different string types. This really should live in the FPC Language Ref document too.

Thanks for sharing. ;-)

Regards,
- Graeme -

[...] >An "UTF8 String" is a String encoded in UTF-8. [...] I know the type "UTF8String" but what is an "UTF8 String" (which you say differs)?

You lost me here.

Maybe I can help you if you explain what you want to program and why do you think that UTF8String is needed for that.

Mattias

Just read my emails in this thread.
I have written it dozen of times:
I want to use UTF8Delete with an UTF8String (which seems natural to me because that's what this function was made for).

And I am wondering why these functions in LazUTF8 unit (which only work with UTF8 strings) do not use UTF8String as parameter.

type UTF8String = type ansistring(cp_UTF8)

Are there any other string types with UTF8 encoding in Free Pascal? As far as I know it can only be this one.
So it should be used for all functions which require an UTF8 string as parameter, no?
Any other encoding makes no sense (because these
UTF8 functions cannot be used with it).

Be aware that this wiki is outdated (not FPC 3.0). It says:

Currently, the type *UTF8String* is an alias to the type *AnsiString <http://wiki.freepascal.org/Character_and_string_types#AnsiString>*. It is meant to contain UTF8 encoded strings (i.e. unicode data) ranging from 1..4 bytes per character.

which is wrong.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20151211/a68858ef/attachment.html>

J?rgen Hestermann wrote

Are there any other string types with UTF8 encoding in Free Pascal?

Yes. The type "String" has a default encoding of UTF-8 when you use the new Unicode support in Lazarus.
See the wiki page for technical details.

Juha

Not necessarily. You can use SetCodePage() to change the code page of the string without triggering a codepage conversion by using the third parameter which is a Boolean that tells the function to either do a conversion (True; default) or not (False). You'd then need to declare the UTF8* routines as RawByteString and explicitly handle the type conversion.

That's at least more officially supported than ugly pointer typecasts...

(Note: it might be that some details don't work as I described them; if in doubt ask on fpc-pascal please)

Regards,
Sven

This is not entirely correct. Windows uses multi byte strings (one byte per character or more) and UTF-16 (which is mostly 2 Byte and 4 for surrogate pairs). The functions WideCharToMultiByte and MultiByteToWideChar which are also used inside FPC for string conversions both take a CodePage parameter that can also be CP_UTF8.

Regards,
Sven

I just checked the generated code and it's the same for both with and without Pointer. The question is whether this is intended to work. If I were you I definitely wouldn't rely on that until this question is answered (and then you can probably use it without the Pointer typecast...)

Regards,
Sven

I updated UTF8String.

The memory layouts are missing the codepage attribute.

Mattias

[...] And I am wondering why these functions in LazUTF8 unit (which only work with UTF8 strings) do not use UTF8String as parameter.

I'm sorry. This was answered several times. Probably there is some kind of misunderstanding, but I have no idea where it is.

Mattias

As far as I know, (current) Windows versions only use UTF16 internally. But it provides the old legacy ANSI functions too (which convert to UTF16). MultiByteToWideChar and WideCharToMultiByte are just helper functions to convert from arbitrary encodings to UTF16 (and back). But UTF-8 is nowhere used internaly in Windows (not even ANSI anymore, except the legacy functions which convert to and from UTF16) and you cannot use UTF8 as string encoding for WIN API functions. Otherwise we would not have this problem and could use UTF-8 as a standard for everything.

That's not really an option since it will break every single program using those functions.

AFAIK the Utf8* functions assume their input is UTF8 encoded (they do not check), so something like this should work?

{$ifndef NO_CP_RTL}
procedure Utf8Delete(var S: Utf8String; StartCharIndex, CharCount:
PtrInt); overload;
var
  Temp: String;
begin
  SetLength(Temp, Length(S));
  Move(S[1], Temp[1], Length(S));
  //nex step might not be needed?
  SetCodePage(RawBytestring(Temp), CP_UTF8, False);
  UTF8Delete(Temp, StartCharIndex, CharCount);
  SetLength(S, Length(Temp));
  Move(Temp[1], S[1], Length(Temp));
end;
{$endif}

Anyhow, as stated before, there should be noneed to use the type Utf8String in Lazarus programs.

Bart

Jonas has given me the following as a possible solution:

=== code begin ===

procedure UTF8Delete(var s: UTF8String; StartCharIndex, CharCount: PtrInt);
   begin
     ...
   end;
procedure UTF8Delete(var s: String; StartCharIndex, CharCount: PtrInt);
   var
     orgcp: tsystemcodepage;
     tmp: utf8string;
   begin
     orgcp:=StringCodePage(s);
     { change code page without converting the data }
     SetStringCodePage(s,CP_UTF8,false);
     tmp:=s;
     { keep refcount to 1 if it was 1, to avoid unnecessary copies }
     s:='';
     UTF8Delete(tmp,StartCharIndex,CharCount);
     { same as above }
     s:=tmp;
     tmp:='';
     SetStringCodePage(s,orgcp,false);
   end;

=== code end ===

Regards,
Sven

Bart wrote

Move(S[1], Temp[1], Length(S)); ... Move(Temp[1], S[1], Length(Temp));

No good. Moving the data in memory *twice* kills performance completely. Current Utf8Delete calls Delete which also moves data but only once.

Anyhow, as stated before, there should be noneed to use the type Utf8String in Lazarus programs.

Maybe not now but we must find a clean way to support it in future.

Juha

Bart wrote

Anyhow, as stated before, there should be noneed to use the type Utf8String in Lazarus programs.

i've been trying to follow along and keep up with this but this statement confuses me... how do you designate that a string is utf8 if you don't use that type?

i mean, i'm really old school... i normally use just string or string[xx]... sometimes ansistring... it is all so damned confusing now :(

i mean, i'm really old school... i normally use just string or string[xx]... sometimes ansistring... it is all so damned confusing now :(

It is no more confusing than in Delphi where the default String has UTF-16 encoding.
In Lazarus it is now UTF-8. Besides, it is amazingly compatible with Delphi at source level.
In any case the current system is much better than the old AnsiString
+ dedicated UTF...() functions hack.

Juha

J?rgen Hestermann wrote

Otherwise we would not have this problem and could use UTF-8 as a standard for everything.

What is the problem exactly?
Always call the Windows API 'W'-functions and use proper data types for them. The dynamic string encoding and conversion will take care of everything else.

Juha

Exactly that is my problem too.
There is a string type UTF8String (provided by Free Pascal) but we cannot use it with routines like UTF8Delete (which was designed to work on UTF8 strings only).
This is hard to understand.

I want to make my code readable and therefore I use(d) UTF8String types in all my programs and now I have to move back to the chimera STRING and somehow make
sure with crude system settings that its codepage is UTF-8 while UTF8String already *is* UTF-8.
That's confusing.

And I have read the "wiki"(s)
(there are so many and some are still refering to the older versions of FPC) but I don't get it:

Is it correct that now every ansistring has a static code page and a dynamic code page (as mentioned in
http://wiki.freepascal.org/FPC_Unicode_support)?

Is it correct that each ansistring type can store strings with any encoding (dynamic code page) but there is a "genuine" (static) code page which is preferred (in cases a target encoding needs to be chosen)?

What is the difference between

UTF8String = type ansistring(cp_UTF8);

and

String = type ansistring(cp_ACP);
when cp_ACP=cp_UTF8?

Both would have the same static code page (UTF-8) but may contain a different encoding at run time. So why can't I use an UTF8String in the same way as a String type as parameter?
Why are they not assignment compatible?

The problem is the "proper data type".

In the past I used UTF8String for all my strings
and converted to UTF-16 before feeding them into
Windows API 'W' functions.
That worked very well it was totally clear what happened.

Now these programs do not compile anymore
because UTF8Delete expects STRING and cannot
be used with UTF8String (which is realy rediculous: a function which only works on UTF-8 strings but
refuses to work with type UTF8String).
I have now to move back to type STRING and
somehow (did not find out yet) tell the system
that STRING should be UTF-8 all the time.

To add to the confusion there are multiple global
code page variables which can be switched to UTF-8 but I don't know which to use and what side effekts each have.

Before FPC 3.0 everything was clear but now
I don't know how to do it.

Where do you read this?
I only found:

"Since FPC 2.7.1 the default system codepage of the RTL can be changed to UTF-8 (CP_UTF8). So Windows users can now use UTF-8 strings in the RTL. "

It *can* be changed (but how?).

And what happens to the UTF8String type?
It can never be used anymore?
Why does it exist?

Which adds extra overhead for the default situation (not disableUtf8RTL). But this may very well be neglectable (I'm far too lazy to test).

(Nitpicking: the entire second implementation needs then to be ifdef-ed with IFnDEF NO_CP_RTL, beacuse if that define is true Utf8String equals String)

Bart

J?rgen Hestermann wrote

Is it correct that now every ansistring has a static code page and a dynamic code page (as mentioned in http://wiki.freepascal.org/FPC_Unicode_support)?

Yes.

Is it correct that each ansistring type can store strings with any encoding (dynamic code page) but there is a "genuine" (static) code page which is preferred (in cases a target encoding needs to be chosen)?

Yes.

CP_ACP is *never* CP_UTF8. It is *always* CP_ACP. So these two will *always* be different types. However the specific codepage that CP_ACP refers to is determined at runtime and that can indeed be CP_UTF8.

They are assignment compatible (all AnsiStrings are) and the compiler will insert a code page conversion if necessary (if CP_ACP refers to CP_UTF8 and if one of the two strings is CP_ACP the other CP_UTF8 then the conversion routine won't do a code page conversion of course). However since they are different types you can't simply pass a AnsiString(CP_UTF8) to a var parameter of type AnsiString(CP_ACP).

Regards,
Sven

J?rgen Hestermann wrote

"Since FPC 2.7.1 the default system codepage of the RTL can be changed to UTF-8 (CP_UTF8). So Windows users can now use UTF-8 strings in the RTL. " It *can* be changed (but how?).

DefaultSystemCodePage := CP_UTF8;

Bart

J?rgen Hestermann wrote

The problem is the "proper data type".

It is UnicodeString (or maybe WideString) with Windows API 'W' functions. Everywhere else it is just String.

In the past I used UTF8String for all my strings and converted to UTF-16 before feeding them into Windows API 'W' functions. That worked very well it was totally clear what happened.

That was kind of dummy thing to do because UTF8String was an alias for AnsiString then.
You could have used "String" always.
Now UTF8String is no more an alias.

I have now to move back to type STRING and somehow (did not find out yet) tell the system that STRING should be UTF-8 all the time.

"did not find out yet" -- it means you still did not read the wiki page. :( Why not?

Before FPC 3.0 everything was clear but now I don't know how to do it.

A valid option is to continue using FPC 2.6.4 and Lazarus 1.4.x, especially if your SW is in maintenance mode and/or it depends very much on system codepage in strings.

Still, in most cases our new UTF-8 system "just works".

From earlier post: Why are they not assignment compatible?

This is a wrong list to ask that. The issue has been discussed and explained many times in FPC lists during the past ~ 7 years. Let's not start the same cycle again here.

Juha

Yes, internally Windows uses UTF-16, but if you set your Windows Ansi code page or at least the current thread's locale to UTF-8 (indirectly by choosing a locale that has UTF-8 as code page, I don't know one right now though) then the *A functions *do* work with UTF-8, simply because they use the current locale's code page to convert from Ansi to Unicode and in this case Ansi includes UTF-8.

Regards,
Sven

The page says among other things :
"The new mode is enabled automatically when compiling with FPC 3.0+." Then it explain the technical details how it was implemented.

Juha

Sven Barth wrote

Yes, internally Windows uses UTF-16, but if you set your Windows Ansi code page or at least the current thread's locale to UTF-8 (indirectly by choosing a locale that has UTF-8 as code page, I don't know one right now though) then the *A functions *do* work with UTF-8, simply because they use the current locale's code page to convert from Ansi to Unicode and in this case Ansi includes UTF-8.

If that would be true then we could simply switch the Windows (API) interface to UTF-8 and nobody needs to worry about code pages anymore. I doubt that this can be done.

How do you set the Win API interface to now
expect UTF-8 strings in all *A functions?
Any link?

i don't do delphi... never have and never will ;)

In Lazarus it is now UTF-8. Besides, it is amazingly compatible with Delphi at source level. In any case the current system is much better than the old AnsiString + dedicated UTF...() functions hack.

ok... so do we just define a var as string and stuff any old thing into it? if so, /how/ does it know that two or more bytes are for one code point or whether they are actually separate bytes? i still do a lot of stuff with the old >127 CP437 graphical characters (especially the single and double line box drawing characters) and things are really messed up at times...

Then why does it say:
"Since FPC 2.7.1 the default system codepage of the RTL can be changed to UTF-8 (CP_UTF8)" It should say:
"Since FPC 2.7.1 the default system codepage of the RTL *is* UTF-8 (CP_UTF8)" That's contradicting to your cite and lets room for other interpretations.

Also, in other mails and wikis I read that cp_ACP *can* be UTF-8 (on windows) but it is not nessesarily so. But how it is set to UTF-8 is left in the dark.

Then it explain the technical details how it was implemented.

This may be true for those who coded the new string mode or those who wrote the wiki but not for the readers who tries to find out without knowing it already.

Thanks for your answers.
At least some of my questions are answered now.
The wikis are not clear about it.
Especially, the role of CP_ACP is not clearly documented.

So I need to add this to my program(s) now?
Where?
In each unit?
In the main program?

Juha Manninen wrote

That was kind of dummy thing to do because UTF8String was an alias for AnsiString then. You could have used "String" always. Now UTF8String is no more an alias.

What an arrogant answer!
I read it like:
"You should have known that UTF8String was just introduced to later make it incompatible to all system routines. It was just there to irk programmers."

Why wasn't it mentioned before that you should not use it? And what was the purpose of UTF8String (and what is it now)?

Again a very arrogant attitude.
I read it dozen of times but it is totally confusing and contradicting.

Still, in most cases our new UTF-8 system "just works".

I see how "it works". Thank you.

If you write a program in Lazarus which uses the LCL (which in turn uses LazUtils) then this is done for you by the LazUtils package at initialization of the fpcadds unit.
(Mind you, it does NOT do this when you define disableUtf8RTL in projectoptions or on commandline.)

And it needs to be done only once.

Why in the world would you think so?

Unless you actively alter DefaultSystemCodePage in code in your program (as LazUtils does, see above), the value of this is determined at program start and is dependant on you current system locale (cp1252 on West-European Windows, UTF8 on most Linux).

Bart

Because Juha wrote

Please read this : http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus "String" type is UTF-8 and it works now (almost) transparently without explicit conversions.

and

The page says among other things : "The new mode is enabled automatically when compiling with FPC 3.0+."

That should tell me that STRING is now UTF-8 all the time. And I wondered about the text in the wiki which says that it *can* be changed. But if it not only can be changed but already *is* changed then the wiki text needs a change too.

As said: The docu in the wikis is very confusing and contradicting, fully understandable only for those who already know the details.

J?rgen Hestermann wrote

Again a very arrogant attitude. I read it dozen of times but it is totally confusing and contradicting.

There is no need for such a tone, please!
"We" ty to explain things as best as we can, but somehow fail. That's unfortunate, but no need for calling names.

Grasping the concepts of the new CP aware strings and all it's implications is not that easy.

Switching to fpc 3.0 may indeed force you to make some adjustements in your code.
Nevertheless, most of it will work out of the box, as long as you do not fight the system.

I also feared things to get broken when I switched. An example: my backup program which deals with unicode filenames outside my codepage (e.g. chinese) needed about 20 lines code changed (mostly due to the fact that it uses a function variable that in it's signature has a TSearchRec, of which the definition is not compatible with fpc 2.6). If all that would have been hardcoded it would have required no text below may either add to the confusion, or it may clear it up a little.

You also need to consider where we come from.
Lazarus has strived to be a fully unicode compatible environment. So, a string type was needed that serves the full Unicode range. This could either have been some AnsiString type or WideString.

The only AnsiString type that covers full Unicode is UTF-8 encoded AnsiStrings: the elements are single bytes, and each codepoint (a "character") can consist of 1-4 bytes.
One of the advantage being that it is fully backwards compatible with ASCII.

At that time there was no such thing as CP aware strings.

So, Lazarus treated _all_ AnsiStrings as being UTF8 encoded. And it does not check it it really is UTF8 encoded. "We" (Lazarus developers) just make sure that it is.

Delphi had a type Utf8String, so we needed that too. Why not then (as we had Utf8Strig = String) call all our strings Utf8String? Because of Delphi compatibility (and the staggering amount of chages it would require to the existing codebase).
Somewhere in that process it was decided for the Utf8* routines _not_ to have Utf8Strings as parameters or retrun type.
And there were convincing arguments for that decision.

And hence, at that point in time, there would (for programmers using Lazarus) bee no need to use the type that is the legacy we carry with us today.

Now comes along CP aware strings, which must be comaptible with Delphi of course.
So now UTF8String String.
And you face the fact that routines with var or out paramterers of type String cannot be used anymore with type Utf8String.

Of course that sucks (especially from your point of view). And now we discuss how to fix that (see the proposal of Jonas).

We also suggest that you may reconsider the need of the use of the Utf8String type in your code.
As we have tried to explain, there really is no need to use that, provided you do not use the disableUtf8RTL define.

I can see why you felt it was logical to have your strings being of type Utf8Strings, but unfortunately for you, it now turns out to be not such a good choice.

Bart

J?rgen Hestermann wrote

Am 2015-12-12 um 18:21 schrieb Juha Manninen:

[...]

especially those readers from the ancient past (TP/BP days) who are trying to catch up to the modern future...

especially those readers from the ancient past (TP/BP days) who are trying to catch up to the modern future...

LCL has supported UTF-8 for > 10 years. As a long time Lazarus user you should know it.
For most cases things did not change very much. Earlier you had to use the explicit UTF8...() functions, now you don't need them. Things are better!

Usage of UTF-8 multi-byte codepoints is explained here : http://wiki.freepascal.org/UTF8_strings_and_characters

Juha

J?rgen Hestermann wrote

But if it not only can be changed but already *is* changed then the wiki text needs a change too.

I improved the text in wiki.

As said: The docu in the wikis is very confusing and contradicting, fully understandable only for those who already know the details.

It is not contradicting. I hope my editing made it more understandable. However you really must understand something about FPC's new string types and their usage.
There is plenty of info available, it will not be repeated in a wiki page explaining Lazarus UTF-8 support.

Juha

[...] > In Lazarus it is now UTF-8. Besides, it is amazingly compatible with > Delphi at source level.

Just to clarify:
It is amazing how small the percentage of most program sources is that handles non English characters. That's why most Delphi string code works without changes with Lazarus.

Note: UTF8 has 1 to 4 bytes per codepoint. See http://wiki.lazarus.freepascal.org/UTF-8.

Here are examples how to work with UTF-8 strings.
http://wiki.lazarus.freepascal.org/UTF8_strings_and_characters

i still do a lot of stuff with the old >127 CP437 graphical characters (especially the single and double line box drawing characters) and things are really messed up at times...

Unicode has the box drawing characters as well and so does UTF-8.
FPC 3.0 write and writeln are now clever enough to convert to console codepage. You can now simply use the ? character in writeln and in LCL captions and it will work cross platform, e.g. any Windows with any language and even on Linux and Mac OS X.
See here:
http://wiki.lazarus.freepascal.org/Better_Unicode_Support_in_Lazarus#Writing_to_console

Mattias

Thanks for the example.

Mattias

[...] If you write a program in Lazarus which uses the LCL (which in turn uses LazUtils) then this is done for you by the LazUtils package at initialization of the fpcadds unit.

To be exact: It is the unit LazUTF8 of package LazUtils that sets the UTF-8 in the RTL.

Mattias

Juha Manninen wrote

On Sat, Dec 12, 2015 at 8:38 PM, wrote: > especially those readers from the ancient past (TP/BP days) who are trying > to catch up to the modern future... LCL has supported UTF-8 for > 10 years. As a long time Lazarus user you should know it.

I think we all know this, but this is not the point.

For most cases things did not change very much. Earlier you had to use the explicit UTF8...() functions, now you don't need them.

What does this mean?
If I use DELETE on a String type (that is an UTF-8 string) does it now use UTF8Delete internally?
Does it delete bytes or code points?
Does LENGTH now give back the number of bytes or the number of characters (when used on an UTF-8 String)?

Things are better!

Maybe, but without fully understanding the mechanism it is worse in the first place.

Bart wrote

There is no need for such a tone, please!

That's what I thought too as I read Juha's answer
telling me that I was just too "dummy" to have used UTF8Strings.

Grasping the concepts of the new CP aware strings and all it's implications is not that easy.

True. Still many here say that you just need "to read the wiki" and all is clear. But that's not true.

And hence, at that point in time, there would (for programmers using Lazarus) bee no need to use the type I would suggest to remove this type.

I can see why you felt it was logical to have your strings being of type Utf8Strings, but unfortunately for you, it now turns out to be not such a good choice.

It is of course quite easy to rename string types, but that's not the point. It is just totally confusing that Free Pascal (still) provides an UTF8String type but it cannot be used anymore (not even for routines that *only* work on UTF-8 strings).
If UTF-8 is the choice (which is good) then STRING should be replaced by UTF8String so everybody instantly knows what he is using.

But I still don't understand the rationale behind the decision to not allow ansistring(A) to be fed into a var parameter of ansistring(B). I would expect that *all* ansistrings are assignement compatible. Insisting on having the same *static* codepage does not make sure that these strings have the same (dynamic) code page at run time. Assigning 2 strings with different static code pages has the same risk as if their static code pages are equal, they still may or may not have the same encoding.

Allowing all ansistrings to be compatible would also remove the problem with UTF8String and LazUTF8 routines.

J?rgen Hestermann wrote

Then I would suggest to remove this type.

FPC and Lazarus are 2 different projects. Please make such requests in FPC's lists if you must.

Repeating the same whining again and again about Lazarus Unicode choice does not help anybody.
The wiki pages are not SO bad. You could learn this thing if you wanted. Besides, you have alternatives. Namely:
1. Use -dDisableUTF8RTL.
2. Continue using FPC 2.6.4. The plan is to keep latest lazarus 1.6 and trunk compatible with it. Then you still get the nice new IDE features when using FPC 2.6.4.

Juha

{$mode nitpicking}

LauUtf8 unit sets (amongst others) some WideStringManager methods.

FPCAdds unit sets DefaultSystemCodePage to CP_UTF8:

{$ifdef UTF8_RTL}
initialization
  SetMultiByteConversionCodePage(CP_UTF8);
  // SetMultiByteFileSystemCodePage(CP_UTF8); not needed, this is the
default under Windows
  SetMultiByteRTLFileSystemCodePage(CP_UTF8);
{$IFEND}

Bart

J?rgen Hestermann wrote

> And hence, at that point in time, there would (for programmers using > Lazarus) bee no need to use the type Utf8String at all. Then I would suggest to remove this type.

No, it is there for Delphi compatibility, so it should not be removed.

It is just totally confusing that Free Pascal (still) provides an UTF8String type but it cannot be used anymore (not even for routines that *only* work on UTF-8 strings).

Yes,it it confusing.

If UTF-8 is the choice (which is good) then STRING should be replaced by UTF8String so everybody instantly knows what he is using.

But changing all strings in Lazars/LCL to Utf8String is not going to happen. No Lazarus code would be compatible with Delphi anymore if we did.

But I still don't understand the rationale behind the decision to not allow ansistring(A) to be fed into a var parameter of ansistring(B).

Var parameters must be of exactly the same type.
This is what makes Pascal a safer tool than C for example.

I would expect that *all* ansistrings are assignement compatible.

They are assignment compatible.

Allowing all ansistrings to be compatible would also remove the problem with UTF8String and LazUTF8 routines.

I don't think it is that easy.

We should fix the specific UTF8* functions to accept UTF8String parameters.

Bart

Yes, and it does even more.

I wanted to point out that just using package LazUtils does not enable the UTF-8 mode. This is important for non LCL applications.

Mattias

Utf8String is part of the FPC RTL and is just a shorthand for AnsiString(CP_UTF8). That the LCL provides String routines that don't take an Utf8String, but an AnsiString, aka AnsiString(CP_ACP) is not the fault of the Utf8String type, so it will stay. Period.

But I still don't understand the rationale behind the decision to not allow ansistring(A) to be fed into a var parameter of ansistring(B). I would expect that *all* ansistrings are assignement compatible.

They *are* assignment compatible, just like ShortString and UnicodeString are assignment compatible. But they are different types and just like you can't pass a UnicodeString to a function expecting a ShortString var parameter you can't pass an AnsiString(CP_UTF8) to a function expecting a AnsiString(CP_ACP). Simply consider them as different types with automatic type conversions between them...

Regards,
Sven

I'll come back to you regarding this. It *is* possible, but it is not feasible - which is why it isn't used more - and I'm currently a bit too busy to write an example...

Regards,
Sven

Utf8Delete is a LCL function, not a RTL function while Delete is an RTL function. The latter does not know about the former.

Does it delete bytes or code points? Does LENGTH now give back the number of bytes or the number of characters (when used on an UTF-8 String)?

In both cases its Byte. I *think* we have some functionality to deal with code points as well, but I'd need to look that up as I'm not working with these that much...

Regards,
Sven

i don't because i'm just barely dipping my toes into the UTF-8 pool... one of my first tasks was to convert today's mess back to CP437 for posting in pure text environments... it wasn't too hard but it was tedious with tables for converting quoted-printable, html entities and unicode so that things like

   ©
   #$C2#$A9
   =A9
   &#169

are all converted to the old style "(c)" copyright... the same for "(tm)" and others... 1/4, 1/2, 3/4 are additional examples...

For most cases things did not change very much. Earlier you had to use the explicit UTF8...() functions, now you don't need them. Things are better!

i'm hoping to dip further into the pool in the next few years...

Usage of UTF-8 multi-byte codepoints is explained here : http://wiki.freepascal.org/UTF8_strings_and_characters

You mean you convert from Unicode to CP437 system codepage? Uhhh... most people do the exact opposite and want to get rid of the horrors of local codepages using any means possible. What means "pure text environments"? Unicode is also pure text, only the encodings are more sane.

i'm hoping to dip further into the pool in the next few years...

Now you convert from Unicode to CP437. Next year you start to convert from CP437 to Unicode.
Ok. You may be doing something dummy now but it is not my business of course... :)

Juha

yes... plain text readers and editors cannot handle the fancy mess of today's world... they only know CP437 or maybe CP850...

Uhhh... most people do the exact opposite and want to get rid of the horrors of local codepages using any means possible.

most people are not writing utilities for genuine old-school text-mode BBSes, either ;)

What means "pure text environments"? Unicode is also pure text, only the encodings are more sane.

pure text as in only the 255 characters that CP437 offers... think DOS v6.xx command line... no graphics other than the few dozen or so characters offered

127 in CP437... i've already mentioned the single and double line frame

characters... there's also the shading (stippled?) and solid block characters...

no, not "dummy" at all... it has everything to do with the technology target... in one case, we're converting emailed news letters to plain CP437 text for posting in old school BBS message bases... pdf crap is out as it any sort of html or unicode cruft... there are no QWK or BlueWave offline readers that can render any of that mess...

BTW: yes, Fidonet and WWIVnet and similar old school BBS message networks still exist today... get back to the real roots of computer based comms B)

> > You mean you convert from Unicode to CP437 system codepage? yes... plain text readers and editors cannot handle the fancy mess of today's world... they only know CP437 or maybe CP850...

Wow, I don't know what text editors you use, but any text editor worth using should at least support Unicode. If not, that text editor has seen updates for 10-15+ years.

On a side note:
Just over a year ago I reviewed tons of text editors. I needed something that could preview other encodings (in case auto-detect didn't work 100%) so I could decide what should be used, warn me about inconsistent encodings or possible loss of data, then do actual conversions, have great regex support and have many other great features. My conclusion was that EditPad Pro
[http://www.editpadpro.com/] was miles ahead of the competition. Yes it isn't free, but the price is dirt cheap for what you get, including monthly updates and great support. Even though it is a Windows application, it runs perfectly under FreeBSD and Linux using WINE.

Regards,
- Graeme -

J?rgen Hestermann wrote

As said: The docu in the wikis is very confusing and contradicting, fully understandable only for those who already know the details.

This is obvious by the always repeating and long winding discussions on that issue. It supposedly can't be avoided, as, IMHO the concept introduced by Delphi is really flawed and not decently usable for a development environment that is supposed to support portability.

When I had been requested to, I wrote a wiki article describing a concept that (IMHO) could solve the problem - while not breaking existing Delphi or Larzarus/fpc applications (unless the do very low level access to strings), and not introducing noticeable overhead. But of course this this is not along the lines of Delphi (e.g. it defines a new type and uses it for Strings and hence in many classes and functions).

-Michael

we're talking about text editors on traditional old school BBSes and offline readers... the old qedit is one that comes to mind for use with offline readers though some real masochists used edlin ;)

the online editors used /on/ BBSes are generally line oriented or "full screen"... they may be built into the BBS package or they may be an add-on that is executed like a door... "full screen" editors use ANSI sequences for some cursor positioning...

we're talking about being able to use these editors in terminal programs like Telix, ProComm, Qmodem, {COMMO}, etc...

we're talking about text editors on traditional old school BBSes and offline readers... the old qedit is one that comes to mind for use with offline readers though some real masochists used edlin ;)

I thought BBS was made obsolete by e-mails, internet forums and social media sites. Apparently not.

Juha

yup, definitely not... there aren't as many as there used to be, that's very true, but they do still exist... i've got four running here using three different packages and if pressed i could likely dig up addresses for many more... the big thing, today, is that we've been able to use virtual modems which speak old style serial comms on the one side and telnet, vmodem or possibly ssh on the other side... some systems also support rlogin as well as having integrated ftp and web capability...

but, we've drifted way O/T for this thread... this branch only came up because of what i try to do and what i do have to do in certain cases dealing with BBSes and custom utils written for them...

BTW: there is at least one relatively modern BBS written with FPC and i'm aware of at least two others being ported to FPC from TP/BP... there's also at least one ANSI terminal that i think is also done with FPC... it interfaces with SDL2, though, and that's way outside my realm...

yup, definitely not... there aren't as many as there used to be,

wow, very interesting.

more... the big thing, today, is that we've been able to use virtual modems which speak old style serial comms on the one side and telnet, vmodem or possibly ssh on the other side... some systems also support rlogin as well as having integrated ftp and web capability...

I ran AdeptXBBS (OS/2) for about 3 years. It alread supported back in 1994 serial comms via modem, telnet, nntp etc. It was an awesome piece of sotfware.

BTW: there is at least one relatively modern BBS written with FPC and i'm aware of at least two others being ported to FPC from TP/BP.

Yeah, the good old BBS days had loads of tools written with TP. RemoteAccess BBS software was very popular in South Africa. It was also written using Turbo Pascal.

Regards,
- Graeme -

.., but off-topic.

Please continue such a topic on fpc-other.

Mattias

Fixed in r50850.
Proposed for merging to 1.6RC2.

Bart

Better read Jonas' comments on that commit first:

Regards,
Sven

Sven Barth wrote

> The code that was committed for UTF8Delete(utf8string) in > http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/lazutils/lazutf8.pas?root=lazarus&r1=50850&r2=50849&pathrev=50850 > makes no sense:

Can you continue this discussion on the Lazarus devel list? I discussed it there with Mattias.

Bart