Hello,

I would like to add a new package to Free Pascal into the directory packages/fputf8 which will provide classes and routines for UTF-8 applications. It might grow to include things like:

* full unicode tables and conversion routines
* Equivalents of RTL classes which are not adequate for UTF-8 applications and/or will still not be in the wide RTL like TFileStream and TStringList
* Equivalents of RTL routines with UTF-8 parameters

Just announcing to check if there is opposition to Free Pascal distributing such a package.

thanks,

In our previous episode, Felipe Monteiro de Carvalho said:

I don't see the point of this. These things are related to Lazarus UTF8 decision and thus logically belong in Lazarus, till they are superceded in 2.8 by proper unicode support.

Please do not include this package yet.

Currently, hard work is being put in the unicode string and codepage string routines in the RTL.

Chances are that you will duplicate a lot of functionality, and this should be avoided at all costs.

So I suggest you wait till the dust settles around the new string type, and then check
a) what is still needed. Maybe we will have solved it for you. b) whether it should not be put in the RTL instead of a separate package.

Michael.

Marco van de Voort wrote

These things are related to Lazarus UTF8 decision and thus logically belong in Lazarus,

Not really. One can write a non-visual library which uses utf8string as its main string type, why not? But the RTL is not adequate for utf8string projects and will still not be after the Unicode RTL is completed.

This is the case of fpvectorial and fpspreadsheet for example. If there is no fpc package with UTF-8 routines then it becomes impossible to write utf-8 non-visual packages which do not depend on the LCL. FPSpreadsheet already contains code copied from the LCL utf-8 routines, I want to stop this multiplication of code copies and unify all routines in one base place, which logically should be in Free Pascal.

till they are superceded in 2.8 by proper unicode support.

The Unicode RTL is not satisfactory for utf-8 projects. One cannot seriously expect to be able to write a project which uses utf8string if you don't even have a utf-8 TStringList to share between various libraries.

Please do not include this package yet.

I'd rather find a consensus on this now then to postpone yet again. I have patches to merge about this and I have some time now which is something which can disappear at any moment.

So I suggest you wait till the dust settles around the new string type, and then check a) what is still needed. Maybe we will have solved it for you.

I am 100% sure that it will not solve my problems.

b) whether it should not be put in the RTL instead of a separate package.

Wasn't this rejected a short while ago?

You can always commit them to a branch so the changes can be discussed.

In our previous episode, Felipe Monteiro de Carvalho said:

Not really. One can write a non-visual library which uses utf8string as its main string type, why not? But the RTL is not adequate for utf8string projects

True, because we never planned to have support for such utf8 only libraries in the first place.

and will still not be after the Unicode RTL is completed.

I don't see why not.

Then factor out the relevant LCL independent stuff to an external project.

If it is non visual, it doesn't automatically mean it should go into FPC.

1) There is an unit ansistrings in Delphi for just that.
2) And if you use the UTF8 spin of the unicode RTL, the default TStringlist is UTF8.

This is in fact a reason not to include it in fpc. You commit some quick-and-dirty stuff, and then it's forgotton, or abandoned.

So: rather not.

Not to my knowledge.

First wait till the dust settles. You can always park your routines in lazarus CCR in the meantime.

Michael.

Felipe, I agree with you in theory, but I'd still recommend you to move utf8string together with fpvectorial and fpspreadsheet into Lazarus.

This way, you can develop much faster, without the need to fight for your changes,
and without the need for a half-duplicate CCR repository.

As for visual vs non-visual distinction, Lazarus can be used to produce console applications just as well, so moving to Lazarus does not automatically means depending on LCL.

Alexander Klenin wrote

This way, you can develop much faster, without the need to fight for your changes,

Others call this fighting "review" and consider it as important part to improve software quality.

Marco van de Voort wrote

True, because we never planned to have support for such utf8 only libraries in the first place.

...

Then factor out the relevant LCL independent stuff to an external project. If it is non visual, it doesn't automatically mean it should go into FPC.

You know very well that this is not a workable solution. It would be a huge work to set up such an intermediary project between FPC and Lazarus.

I think that's a very radical posture. I can even understand that you may not want utf-8 support in the RTL which you develop, that you may not want to use fputf8 in your projects, but block packages which you don't develop??? You don't have to use it if you don't like it.

1) There is an unit ansistrings in Delphi for just that.

Not nearly what I want. Just to start with, ansistring is not the same as utf8string.

2) And if you use the UTF8 spin of the unicode RTL, the default TStringlist is UTF8.

Also not a solution, because then fpvectorial and fpspreadsheet would not be able to compile in other RTL modes.

Felipe Monteiro de Carvalho wrote

Also not a solution, because then fpvectorial and fpspreadsheet would not be able to compile in other RTL modes.

What? <confused> You mean you are seeking the solution upstream? Seems the design of those units is lacking.

I'd say there is a continuum between those extremes, and (unfortunately) from my point of vew, FPC review is sometimes rather close to the former. I have been burned by this myself.

Anyway, what I suggest is IMO a good compromise and should satisfy both sides -- Felipe can continue development of his packages unobstucted, while the quality of FPC will not suffer.

I do not really see any downsides -- perhaps there is a section of users who use FPSpreadsheet and NOT use Lazarus, but I suspect this section is very small.

FPC has to fit the need of a lot of users so we must be very carefull what we add and what not. Everything we add and which goes into a release must be maintained. We cannot easily kill packages if a maintainer disappears or if the functionality is already in base units. Unicode support is currently under heavy reconstruction so it's not a good idea to add new packages for unicode.

Anyway, what I suggest is IMO a good compromise and should satisfy both sides -- Felipe can continue development of his packages unobstucted, while the quality of FPC will not suffer.

That's why I proposed a branch and that's what branches are for.

For example a TString* class for UTF-8 might even go into classes. But this needs to be done carefully.

Alexander Klenin wrote

I'd say there is a continuum between those extremes, and (unfortunately) from my point of vew, FPC review is sometimes rather close to the former. I have been burned by this myself.

When?

Anyway, what I suggest is IMO a good compromise and should satisfy both sides -- Felipe can continue development of his packages unobstucted, while the quality of FPC will not suffer.

As I remember there is a package "lnet" which can be somehow installed via internet. I don't remember the details but I believe I've seen this a half year ago or so. Why fpspreadsheet and fpverctorial together with new utf8 package can't be follow this way of distributing?

Those users who need them can install then using some fp package install utility.

I also don't see the big need for hosting the majority of FPC packages in the FPC svn repository. Why not to leave there only the basic things like windows, osx, gtk and other platform packages and to remove everything else outside?

Best regards,
Paul Ishenin.

Paul Ishenin wrote

As I remember there is a package "lnet" which can be somehow installed via internet. I don't remember the details but I believe I've seen this a half year ago or so. Why fpspreadsheet and fpverctorial together with new utf8 package can't be follow this way of distributing?

Not a solution for the utf-8 package because then Lazarus would not be able to use it easily.

lnet is only a test package for that functionality. It hasn't been opened yet in general.

Jonas

Florian Klämpfl wrote

That's why I proposed a branch and that's what branches are for.

So far we had two against and one maybe a small part of it ... seriously I think it is unrealistic to expect that if there is virtually no support from fpc developers for the proposal now, if I would implement it in a branch, then support for this would magically build up.

I think we should agree to disagree: Let's delete fpvectorial from FPC, I'll export it into lazarus/components/fpvectorial and we move our separate ways.

No, the branches (in svn, at least) are for code that is an alternative/in-progress
imlpementation of some features, not for features rejected from the main branch.

For example a TString* class for UTF-8 might even go into classes. But this needs to be done carefully.

There is, again, a continuum between careful development and stangation. While acknowledging great work that FPC team has done on the former, I'd venture to say that is came uncomfortably close to the latter.

Felipe Monteiro de Carvalho wrote

I think we should agree to disagree: Let's delete fpvectorial from FPC, I'll export it into lazarus/components/fpvectorial and we move our separate ways.

+1
Among other things, this will simplify management of TAChart's fpvectorial back-end ;-)

Well, I consider an utf-8 package as an alternative/in-progress implementation to the current wip on unicode support.

not for features rejected from the main branch.

If you like, we can also create svn.freepascal.org/svn/fpc/stagging ;) I don't see it as rejected, see e.g. also MvC's mail.

To avoid re-starting old flame, I'll send you an example privately.

I also don't see the big need for hosting the majority of FPC packages in the FPC svn repository. Why not to leave there only the basic things like windows, osx, gtk and other platform packages and to remove everything else outside?

There is also a balance there -- while I agree that currently FPC has some obvoiusly unneeded packages (a52/dts for example), there are some benefits for both users and developers when using a common repository.

Alexander Klenin wrote

There is, again, a continuum between careful development and stangation. While acknowledging great work that FPC team has done on the former, I'd venture to say that is came uncomfortably close to the latter.

I think http://wiki.freepascal.org/FPC_New_Features_Trunk looks quite good.

Apart from that, I agree with the sentiment that it does not make sense trying to stuff everything and the kitchen sink under the FPC umbrella. Everything that's distributed that way automatically becomes the responsibility of the "core" developers of the day, and that collection of source code has grown much faster than the number of people maintaining it (and in a number of cases, maintainers have also disappeared). So distributing packages separately generally indeed makes much more sense for everyone (more flexibility for the maintainer, less responsibility and stagnation risk for the FPC project due to additional responsibilities).

And in this particular case, there does remain the fact that the cpstrnew adaptations are far from finished and that part of the functionality in this "utf-8 package" may well become part of the RTL or FCL. If there is then also a separate utf-8 package used by fpvectorial, and possibly also by user programs, you can bet that people would (understandably) complain if functionality is then removed from this utf-8 package to prevent code and maintenance duplication.

That's also why Florian mentioned the branch: does not cause wrong expectations (users won't start using that package), the code is available in svn and therefore it can be used for finishing the cpstrnew work where appropriate, and before/when merging to trunk we can first look at what should be removed or changed to prevent duplication of functionality. But again: as a non-FPC package you indeed avoid this overhead and the uncertainty that when the time merging arrives, you won't be able to merge everything you want.

Jonas

I'd venture to disagree.

Attached is a small graph from the bugtracker activity. As you can see, the bugfix rate remains constant, which refutes 'stagnation'. We do work, even if it is not so 'visible'.

Many things are not even in this graph, as Jonas pointed out with his WIKI link.

The difficulty is:

The more code there is, the more it must be maintained. The team does not grow at the same rate as the lines of code in the files we distribute, meaning that virtually the same amount of people must maintain more code.

Now, to counter the perception of stagnation, you can help:

I suspect faster release schedules could help to counter this perception. So: Please help us working on fppkg, which should allow a much more free release schedule for FPC, and especially for the packages.

Using fppkg as a distribution mechanism would release the FPC team from the need to maintain everything in the FPC subversion. People could add packages to the repository, and release updates of their , we (mainly Joost, in fact) are converting the FPC packages to this new mechanism, fixing bugs as we go along. You can help by testing this, and converting more packages to the new fppkg.

Michael.

As I said, I did not ever deny you work, of course you do. Still, "steady bugfixing" and "dymanic evolution" are not quite the same, although related.

Now, to counter the perception of stagnation, you can help: Please help us working on fppkg

Looking at the bugtracker, I found a single non-assigned issue #18321, which talks about some "webdesign" package I know nothing about. So I where can I look at the todo-list?

Slowly, we (mainly Joost, in fact) are converting the FPC packages to this new mechanism, fixing bugs as we go along. You can help by testing this, and converting more packages to the new fppkg.

Perhaps I can take care of a single package -- for example, numlib improvement was discussed recently on the forum. However, I am afraid my work will be rejected again
as was the case with fpdoc.
So I'd consider working on numlib (+ math unit, which is obviously closely related),
if I'd get either commit access or somebody who will timely commit my patches.

Basing an opinion on 1 occurrence - however painful - is not very scientific :-)

The amount of patches being rejected is a small minority. But yes, it sometimes happens.

So I'd consider working on numlib (+ math unit, which is obviously closely related), if I'd get either commit access or somebody who will timely commit my patches.

You should contact Marco Van De Voort, he currently looks after numlib.

Michael.

Do I understand correctly that this is the "New String" string handling (to be optionally, by default or exclusively used) ?

Do I understand correctly, that with theses strings automatic conversion between different string encodinge (e.g. locale based ANSI <-> Unicode and UTF-8 UTF-16) is provided in the RTL ?

-Michael

Yes.

Do I understand correctly, that with theses strings automatic conversion between different string encodinge (e.g. locale based ANSI <-> Unicode and UTF-8 UTF-16) is provided in the RTL ?

Yes. Each string will have a codepage identifier associated with it. if, during an assign operation, a difference exists, a conversion will occur. (there are of course some exceptions)

Michael.

GREAT !

I wonder when Lazarus will be able to follow and use this string type for the complete library.

-Michael

The earliest point is when the next big release after 2.6 is released (note: I don't mean 2.6.x here, more a 2.8). As 2.6 is currently prepared you can estimate how long that will take.

Regards,
Sven

I see. ;-)

Unfortunately Delphi is way ahead on this behalf, but I also know that Delphi provides a lot of traps for users converting their "ANSI" projects to use NewStrings (even if they don't need Unicode at all).

-Michael

Not really :-(

I wonder when Lazarus will be able to follow and use this string type for the complete library.

The LCL will use one unique encoding, most probably UTF-8 as the only losless Unicode representation. This means that strings with different encoding are converted whenever passed to a control, or read from it.

DoDi

This might be true and it is obviously the way to go when running on Linux.

But with dynamically encoded Strings, the LCL does not need to force a certain encoding on it's user code interface. It _could_ internally use UTF-8 on Linux and UTF-16 on Windows with the same source code (only some differences in the direct system API related code that is OS-specific anyway). In the user code auto-conversion is supposed to happen whenever necessary.

I suppose this would result in faster programs on Windows.

Of course auto-conversion in the user code might slow down certain programs hen the user somehow forces a certain coding. And inappropriate user code might implicitly rely on a certain internal coding of a string and work only on a certain OS.

-Michael