Hi Experts,

A friend of mine wants to port an application from Delphi 7 to a be a Lazarus 64 Bit Windows application.

This (huge) application only features a very limited GIU, but uses strings lot for "uncoded" characters, going in and out via TCP/IP. Any automatic type conversion would kill the application.

So it would be necessary that fpc is configured to use the (old) 8 bit Strings. These might be used with UTF8 encoding for the GUI stuff (and for some limited locations in the business-code with *explicit* conversion).

Is this possible with the current version of Lazarus ?

As I lost the clue on the planned path regarding code-aware strings: will this be possible with future versions of Lazarus ,as well ?

Thanks,
-Michael

If you code well, none will happen.

It is possible today. You don't need an 'old' style fpc.

Just use RawByteString as stringtype for all parameters and variables in calls where you don't want code page conversions to occur. The compiler will not touch the codepage then.

Michael.

Michael Schnell wrote

Any automatic type conversion would kill the application. [...] will this be possible with future versions of Lazarus ,as well ?

I guess you mean automatic encoding conversion for strings. (?) The automatic conversion is NOT a feature of Lazarus. It is a feature of FPC 3.0+.
If you don't want it then you can continue using FPC 2.6.4. It is a valid choice in some cases, thus Lazarus will support FPC 2.6.4 for some time to come.

Juha

Michael Van Canneyt wrote

Just use RawByteString as stringtype for all parameters and variables in calls where you don't want code page conversions to occur. The compiler will not touch the codepage then.

Ok, true. That works, too. :)

Juha

I suppose a more clear question would be...

is it possible for now and the foreseeable future to have FPC (3.0+, not just 2.x) make the "string" type (not rawbytestring or anything else) behave as it was behaving always since the beginning of time without any source code changes but (say) via some compiler argument?

I suppose it is worth to have something like this avoid breaking code that exists out there (some might consider it as already broken, but for many if it works - especially if it worked for years - it isn't broken and IMO that is a valid stance to take).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160212/964f9612/attachment-0001.html>

Kostas Michalopoulos wrote

is it possible for now and the foreseeable future to have FPC (3.0+, not just 2.x) make the "string" type (not rawbytestring or anything else) behave as it was behaving always since the beginning of time without any source code changes but (say) via some compiler argument?

No. "String" type triggers conversion of dynamic encoding when needed. RawByteString does not.
I am surprised you didn't know it. The topic was discussed in FPC mailing lists again and again for many years.
And yes, often the same arguments were repeated by the same person who started this thread.

I suppose it is worth to have something like this avoid breaking code that exists out there (some might consider it as already broken, but for many if it works - especially if it worked for years - it isn't broken and IMO that is a valid stance to take).

For old code that is in maintenance mode, a valid option is to use FPC 2.6.4. For actively developed new code the dynamic string encoding conversion is a huge improvement.

Juha

Michael Van Canneyt wrote

Just use RawByteString as stringtype for all parameters and variables in calls where you don't want code page conversions to occur.

This would mean rewriting the complete application. This obviously is not desired .

-Michael

Juha Manninen wrote

The automatic conversion is NOT a feature of Lazarus. It is a feature of FPC 3.0+.

Of course I do know this. That is why I ask. AFAIK, fpc can be configured to use the new string type or not to use it.But will this be possible for a Lazarus application.

If you don't want it then you can continue using FPC 2.6.4. It is a valid choice in some cases, thus Lazarus will support FPC 2.6.4 for some time to come.

This might be a possible option, but it would be better to use the current fpc and use compile switches to have it use the "old" 8 bit string type.

Moreover " for some time to come." is not a really helpful option.

-Michael

Tish...

{$define String:=RawByteString}

and you are done.

Juha Manninen wrote

And yes, often the same arguments were repeated by the same person who started this thread.

At the time I did those researches and discussions, my colleagues here were interested in porting their Delphi application to Linux using Lazarus and I tried to help. They finally decided do delay the task, and now, as we have Windows 10 IOT, they supposedly are not interested any more at all, as Delphi applications (services without a GUI) do run on Win 10 IOT

Now a friend of mine is faced by a similar task. But here he wants to keep a (small) GUI and he does not want to go for Linux but for 64 Bit.

But the challenge is rather similar, and again there *might* be hope to convert a professional application and programmer team from Delphi to Lazarus.

-Michael

I'll let him know...

-Michael

Michael Van Canneyt wrote

{$define String:=RawByteString} and you are done.

Awesome!

FPC: 1
Delphi: 0

:)

Regards,
G.

I don't thinks this is true.

Many built-in functions and RTL functions and classes (such as "pos" and "TStrings") explicitly use "string" as parameter and property type. If they are not re-compiled they will not work at all or not as expected .

-Michael

Then you are, as they say, f*d...

Michael.

FPC 3.0+ does not arbitrarily change the encoding of strings. That's why the LCL can use UTF-8 in AnsiStrings even without the new UTF-8 mode. It works pretty similar to FPC 2.6.4.

The many encoding discussions raised from the problem that the LCL uses UTF-8 strings in RTL classes which expect system encoding. Your friend does not have this problem.

Your friend can use the LCL for the GUI with disabled UTF-8 mode. He has to check (convert) the strings between his TCP/IP code and the LCL controls (e.g. UTF8ToSys). As he has only a small GUI this should be easy.

As I lost the clue on the planned path regarding code-aware strings: will this be possible with future versions of Lazarus ,as well ?

The new UTF-8 mode is easier for new projects. Many old projects still use the old mode. So chances are high that it will be supported the next years.

Mattias

Michael Schnell wrote

Now a friend of mine is faced by a similar task. But here he wants to keep a (small) GUI and he does not want to go for Linux but for 64 Bit.

Both Delphi and FPC now support 64-bit. I don't know how that relates to dynamic String encoding.

But the challenge is rather similar, and again there *might* be hope to convert a professional application and programmer team from Delphi to Lazarus.

Sure there is hope. See the new UTF-8 support:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

The only problem comes if you depend on Windows system codepage a lot. That's why there is :
http://wiki.freepascal.org/Lazarus_with_FPC3.0_without_UTF-8_mode Note: Even there FPC 3.0 still has the dynamic string encoding but it is still useful.

Juha

True, although the wording might be misleading.
FPC 3.0+ does not arbitrarily change encoding of strings. That's why the LCL can use UTF-8 in AnsiStrings even without the new UTF-8 mode.
It then works pretty similar to FPC 2.6.4.

Mattias

Juha Manninen wrote

Sure there is hope. See the new UTF-8 support: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

I'll let him know...

And of course I will take read this myself, too.

In fact he does not want "Better_Unicode_Support" but *no_unicode_support* ;-)

But maybe this is possible somehow....

The code does not depend on Windows system just
receives text (characters) from TCP/IP, converts it using databases and some "embedded"/"realtime" conditions and sends it out via TCP/IP. Not windows or other codepages "implicitly" involved. If some o the work needs codepage conversion, this is done explicitly in user code.

-Michael

[...] {$define String:=RawByteString} and you are done.

Won't compile when calling RTL functions with var parameters.

I fail to see why RawByteString should do less conversions than using AnsiString, when porting a Delphi 7 application. Can you give an example?

Mattias

Juha Manninen wrote

Sure there is hope. See the new UTF-8 support: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

This page says:

"This page covers Unicode support in Lazarus programs (console or server, no GUI) and applications (GUI with LCL) using features of FPC 3.0+. "
"RTL with default codepage UTF-8"

That would mean that the type "String" is 8 Bit ?

That would mean that 8 bit Strings are used in TStrings and hence in TStringList ?

That would mean that (e.g.) storing 8 bit "String"s in a TStringList would not do any conversion at all ?

And it would mean that pos() and friends work on 8 bit Strings (and wist simple byte-positions and length) ?

If string constants and strings coming from and going to the GUI are implicitly converted, I suppose this is manageable.

Is all this is here to stay, I think my friend could use Lazarus and fpc for his task.

-Michael

AFAIK, any 64 bit Delphi forces UTF-16 Unicode and cant do what is described in the wiki page you mention.

-Michael

This is exactly what I seemed to remember from multiple discussions in multiple forums, and this is why I recommended my friend to take a look at Lazarus/FPC.

the LCL *can* use UTF-8

How to set it to do so ? (I may be able guess, but as I said I am asking for a pal who is rather new to Lazarus/FPC, so explicit pointers are appreciated. ...)

even without the new UTF-8 mode.

How to set this mode when compiling with Lazarus ? What mode do you think is more appropriate when most work is done on strings that just have no or an unknown encoding ?

Thanks a lot !
-Michael

Mattias Gaertner wrote

Your friend can use the LCL for the GUI with disabled UTF-8 mode.

Sounds great !

He has to check (convert) the strings between his TCP/IP code and the LCL controls (e.g. UTF8ToSys). As he has only a small GUI this should be easy.

No problem. The current D7 code does this,anyway, it just converts to/from ANSI with the appropriate code page,and now would need to convert to/from UTF8 .

The new UTF-8 mode is easier for new projects. Many old projects still use the old mode. So chances are high that it will be supported the next years.

Am I correct assuming that "The new UTF-8 mode" is this http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus ?

We will try to find out if this mode or the "old" is more suitable for the task in question.

-Michael

Yes, 8bit per Char.

That would mean that 8 bit Strings are used in TStrings and hence in TStringList ?

Yes.

That would mean that (e.g.) storing 8 bit "String"s in a TStringList would not do any conversion at all ?

Yes.

And it would mean that pos() and friends work on 8 bit Strings (and wist simple byte-positions and length) ?

Yes.

If string constants and strings coming from and going to the GUI are implicitly converted, I suppose this is manageable.

Without UTF-8 mode you are responsible to convert the LCL UTF-8 strings to whatever needed (probably system code page, UTF8ToSyS).

With UTF-8 mode all strings are by default UTF-8 and implicitely converted when using functions like FileExists or
TStrings.LoadFromFile. If you want to work in system code page you have to convert back and forth between RTL+LCL and your code.

When converting a Delphi 7 application it depends on what charset(s) the application supports to decide which mode is easier/better.

Just saying "uncoded strings in TCP/IP" is not enough information. That can be done in both modes.

Is all this is here to stay, I think my friend could use Lazarus and fpc for his task.

Mattias

On Fri, Feb 12, 2016 at 9:16 AM, Michael Van Canneyt <

How to achieve the same effect from command line?

thanks,
Dmitry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160212/a5326226/attachment.html>

As I understand it, calling

Procedure MyProcedure(S : RawByteString) ;
begin
end;

with

Type
   T = Type Ansistring(850);
Var
   D : T;
begin
   MyProcedure(d)
end;

will not cause any code page conversion. While if it was

Procedure MyProcedure(S : String) ;

Then the compiler would first convert the code page to DefaultSystemCodePage or CP_ACP

Michael.

[...] > even without the new UTF-8 mode. How to set this mode when compiling with Lazarus ?

See here
http://wiki.freepascal.org/Lazarus_with_FPC3.0_without_UTF-8_mode

What mode do you think is more appropriate when most work is done on strings that just have no or an unknown encoding ?

If your strings don't have an encoding the compiler treats them as default encoding and so the mode does not matter.
Your friend's application won't pass such strings to the GUI, so no problem with the LCL.
You just have to be careful when mixing strings and widestrings (just like Delphi), or strings with non default codepage (AFAIK Delphi 7 does not have them, but maybe you use a package that does).

If your friend use string literals in Delphi they are encoded in system code page. Here might be some traps.

Mattias

Am 12.02.2016 17:16 schrieb "Dmitry Boyarintsev" <

Just don't. There are things that are not worth the trouble and this is one of them.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160212/4cb90d99/attachment.html>

I normally would not advise such a trick.
But if the alternative is 'rewriting the whole application'... then probably anything is worth a shot...

Michael.

In this particular case (of porting Delphi 7 code straight to FPC 3.0.0) is a matter of backward compatibility.

I could agree that there are very little options left:
* Delphi (whatever version supports 64bit) - changes in code are needed
* FPC 3.0.0 - {$define String:=RawByteString} must be added to every unit (or a single .inc file if used)
* FPC 2.6.4 - no changes needed (?)
So it might be indeed a thing not worth the trouble.

thanks,
Dmitry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160212/0c5ea0e9/attachment.html>

AFAIK this was introduced with Delphi 2009. Not available in Delphi 7.

Mattias

Dmitry Boyarintsev wrote

In this particular case (of porting Delphi 7 code straight to FPC 3.0.0) is a matter of backward compatibility.

Yep.

Not only to D7, but also to legacy fpc and Lazarus versions, that (as we all know) were modeled after D7 (and friends).

I think that it is *very* desirable to provide configuration options to provide full backwards compatibility (while still allow to use as many of the new features as possible when explicitly writing coding for this).

-Michael

This does sound good. :-)

-Michael

Mattias Gaertner wrote

If your friend use string literals in Delphi they are encoded in system code page. Here might be some traps.

He will easily be able to handle this .

Thanks a lot !

-Michael

Michael Schnell wrote

I think that it is *very* desirable to provide configuration options to provide full backwards compatibility (while still allow to use as many of the new features as possible when explicitly writing coding for this).

Uhhh...
Michael Schnell, how is it possible you behave like a complete newbie with FPC's String encoding issues, after so many years of arguing about it in various mailing lists?
Please show some respect to the people who made it work and even documented it. For example these questions :

That would mean that the type "String" is 8 Bit ? That would mean that 8 bit Strings are used in TStrings and hence in TStringList ? That would mean that (e.g.) storing 8 bit "String"s in a TStringList would not do any conversion at all ? And it would mean that pos() and friends work on 8 bit Strings (and wist simple byte-positions and length) ?

... were answered in the wiki pages I linked to you, and they are obvious after just minimal testing.
If you find a bug then let us know, otherwise please show some positive attitude. The new UTF-8 system "just works" in most situations.
I remember you repeated the same arguments against FPC's new String type in FPC lists during many years again and again. Let's not do the same here.

Juha

Am 12.02.2016 17:37 schrieb "Dmitry Boyarintsev" <

: On Fri, Feb 12, 2016 at 11:24 AM, Sven Barth

one of them.

In this particular case (of porting Delphi 7 code straight to FPC 3.0.0)

is a matter of backward compatibility.

I could agree that there are very little options left: * Delphi (whatever version supports 64bit) - changes in code are needed * FPC 3.0.0 - {$define String:=RawByteString} must be added to every unit

(or a single .inc file if used)

* FPC 2.6.4 - no changes needed (?) So it might be indeed a thing not worth the trouble.

As long as you don't change the default code page strings in 3.0.0 should behave very close to 2.6.4. The only exceptions would be API that talk to the system (e.g. file APIs, though AFAIK we haven't really touched those yet and the FindFirst API).
But in the end one should simply try it and after all one should have testcases that help one discover problems anyway...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160212/ad74e88a/attachment.html>

because of this:

Michael Van Canneyt wrote

Then you are, as they say, f*d...

Juha Manninen wrote

... were answered in the wiki pages I linked to you, and they are obvious after just minimal testing.

This is what I did assume and just wanted a confirmation for, so that I can send this information in a compact and approved form to the said friend .

If you find a bug then let us know, otherwise please show some positive attitude. The new UTF-8 system "just works" in most situations.

Sorry I did not intend to express any negative attitude, but on the contrary am very happy that Lazarus/fpc can provide exactly that functionality (8 bit Strings - including TStrins, pos , etc - in a 64 Bit application) that (AFAIK ) is not possible with Delphi (which I don't have).

I remember you repeated the same arguments against FPC's new String type in FPC lists during many years again and again. Let's not do the same here.

IMHO your memory is incomplete on that behalf. On the contrary I argumented against the "intermediate" Lazarus way of using UTF8 *without* "new strings" (which - as you correctly say, seemingly is exactly hat my said friend would like to use (but here, he does not care about the LCL GUI functions). In fact I always was pro "New Strings",

Later I argumented against the way New Strings have been implemented (very close to the "sloppy" way Delphi does this), which (among other IMHO not really good stuff) does not provide for using TStrings for "uncoded" / "unkowinly coded / "transparently coded" 8 bit strings. As you might know I did a description of an IMHO better way to do this here:
http://wiki.lazarus.freepascal.org/not_Delphi_compatible_enhancement_for_Unicode_Support .

-Michael

Michael Schnell wrote

Sorry I did not intend to express any negative attitude, but on the contrary am very happy that Lazarus/fpc can provide exactly that functionality (8 bit Strings - including TStrins, pos , etc - in a 64 Bit application) that (AFAIK ) is not possible with Delphi (which I don't have).

Ok, then I misunderstood.
The new UTF-8 system has been criticised, rightfully so, for changing the String default codepage to UTF-8 and loosing the original Windows system codepage.
In ideal situation FPC would allow changing the encoding of String but leave AnsiString as is.
Now String = AnsiString and our solution is a semi-hack, although much less of a hack than the old explicit UTF8...() conversion functions. Still, in my experience the solution is good. In a typical situation dealing with Unicode I/O it works like magic. When dealing with Windows system codepage, few lines of explicit conversion are needed.

I have also learned that the encoding wars are quite useless. It is possible to write code that works with both main encodings at source level, by using String for individual codepoints and with some helper functions.
The real complexity of Unicode is beyond encodings.

Juha

Juha Manninen wrote

The real complexity of Unicode is beyond encodings

Yep.This I learned from the discussions. E.g. even with a technically perfect UTF string implementation, the problem that exactly the same printable text can be encoded in multiple ways can't be overcome.

-Michael