Hello Lazarus,

I'm now "upgrading" a Lazarus 0.9.24 DB project to 0.9.25 with by default unicode support. The problem is that the database can not currently be upgraded to be unicode compliant so it is using the default codepage meanwhile all the visual objects like dbgrid are unicode.

The problem is that when a text field has a "special" character (codepage) the string fails to be rendered displaying nothing (blank line) and I'm unable to find a way to insert the Utf8ToAnsi and AnsiToUtf8 conversions somewhere.

My layout is a non visual firebird connection which is linked to the visual object after form creation and the connection is linked to a SQLQuery -> DataSource -> DBGrid

May anybody help me in the best way to insert the necessary conversions ?

Note: I'm not using the fields definition in the SQLQuery object as at design time the database connection is not present in the form.

You could override the TDataset.DataConvert procedure or the TDataset.Translate procedure and set the Transliterate property to true of the TStringFields of which you want to convert the data.

If you e.g. use a TSQLQuery, you can add your own
TTransSQLQuery=class(TSQLQuery) and override one of the above procedures.

Joost

Op woensdag 23-07-2008 om 18:39 uur [tijdzone +0200], schreef JoshyFun:

Hello Joost,

Wednesday, July 23, 2008, 10:46:44 PM, you wrote:

JvdS> You could override the TDataset.DataConvert procedure or the JvdS> TDataset.Translate procedure and set the Transliterate property to true JvdS> of the TStringFields of which you want to convert the data.

JvdS> If you e.g. use a TSQLQuery, you can add your own JvdS> TTransSQLQuery=class(TSQLQuery) and override one of the above JvdS> procedures.

Hmmm... it sounds interesting and "quite" simple to transform in full unicode when the DB be upgraded to unicode too.

This means that I must create my own visual object derived from i.e. TSQLQuery and add it to the lazarus components, but at least looks like a good option if I'm not able to find a "automagical" way to do it without special components.

A lot of thanks.

Hi
Be warned that this will fail on some databases
Example, ZEOS components, Lazarus 0.9.25 Unicode UTF-8, database encoding UTF-8
Tables contain ansi & extended including Chinese & Hebrew If you fail to set the driver to UTF-8, the transliteration is done before the table object receives it
The result is ????? instead of Chinese & Hebrew

This email is utf8 encoded & the following lines are supplied as a test reference
The Polish line may look like ansi but the Z has a punctuation mark above it which will be lost if you do conversion

Hebrew : ??? ?? ????? ?????????? NLSO <http://dual-core/nlso/index.php> Chinese : ???????????????? <http://dual-core/nlso/index.php> English : This site uses NLSO technology <http://dual-core/nlso/index.php> Polish : Ta strona uz.ywa technologi NLSO <http://dual-core/nlso/index.php>

These will only show correctly if you have international support on your system,
The database will only work correctly if there is NO conversion or compatible conversion eg: utf8 to ucs2 or ucs2 to ucs4 The objective for smooth operation should be to have the table fields utf8 encoded, all communication between your application & the database also utf8 encoded
To achieve this with firebird, I have no clue
To achieve it with Zeos using MySql is easy when you know how Drop a ZConnection on your form
set the ZConnection properties to

character_set_client=utf8
character_set_connection=utf8
character_set_database=utf8
character_set_results=utf8
character_set_server=utf8
character_set_system=utf8
collation_connection=utf8_general_ci
collation_database=utf8_general_ci
collation_server=utf8_general_ci
Codepage=utf8

This will force the Mysql driver to speak utf8 in all communication Failure to do this with MySql driver will make all attempts to transliterate data absolutely useless on the above language samples as the data is mangled by the driver anyhow
I suggest you look first at the table encoding, then the communication with your server
If these are right then you should have no need for further transliteration, use pure utf8

Have fun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080724/3f061089/attachment.html

No. It's iso-8859-1 and the ???? are just plain ascii.

Mattias

The html part of them mail is also iso-8859-1 but the characters are html encoded. In the html view you can see the characters. But there was no UTF-8 in the mail.

Here is UTF-8:

Hebrew : ??? ?? ????? ?????????? NLSO
Chinese : ????????????????
English : This site uses NLSO technology
Polish : Ta strona u?ywa technologi NLSO

Mattias

Hi
When I look at the message source, I see content-encoding set to UTF-8 When I created the message, I set the content to UTF-8, What Thunderbird or the mail list do with it after that I cant say I can confirm that your mail was UTF-8, base64 encoded

I have had loads of problems with database encoding & transfer of data, though I didn't expect the same with these mails
This thread clearly demonstrates to the reader what goes wrong when encoding standards change & the potential pitfalls when changing Lazarus to UTF-8
Mattias is Absolutely right about the encoding, yet Thunderbird tells me the encoding is UTF-8
This brings me back to the original point about database encoding If you use UTF-8 in the Lazarus application, ensure that your database driver speaks UTF-8 & your tables are also using UTF-8 Anything else & your risking data mangling & seeing ???????????????? or some other junk where your text should be

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080724/5f1a5983/attachment.html

Hello Jesus,

Thursday, July 24, 2008, 1:19:36 AM, you wrote:

JR> I think the best would be to fix your database, if you don't

Yes, convert the database to utf8 should be the best solution, but currently that's impossible as there are different software using it and I can not control all of those. So meanwhile the other software can not work with utf8 data I should not upgrade the database (even we are using FB 1.5).

JR> really need to have multiencodings then avoid to use it and fix

No multiencoding is needed, only ISO8859-1 is needed and all fields have the same encoding (no encoding at all in fact).

[...]
JR> this. Then setting the expected charset in ibconnection should be JR> all what you need to do, because ib server should send you strings JR> in requested encoding and on saving it should convert back to JR> field charset (provided that when storing data in target field the JR> conversion is legal, that's it)

The problem raises when the character sequence (without encoding) looks like a utf8 special character group but the result code is invalid as Lazarus try to convert it and raises an exception which will stop all the data gathering.

I'll try to get the data manually and see if the problem is something in the conversion routines (my problem, not a conversion problem) or the way the data is being feeded.

I'll do more tests and post here the results.

Hello Ivan,

Thursday, July 24, 2008, 1:23:36 AM, you wrote:

IG> Hi
IG> Be warned that this will fail on some databases IG> Example, ZEOS components, Lazarus 0.9.25 Unicode UTF-8, database IG> encoding UTF-8
[...]
IG> I suggest you look first at the table encoding, then the communication IG> with your server
IG> If these are right then you should have no need for further IG> transliteration, use pure utf8

My wish is to continue using "locale" data, no unicode data in both sides but using the more recent versions of lazarus than 0.9.24. Ok, I can compile Lazarus without unicode support but as someday the whole system should be updated to unicode I want to work in a mixed mode and a way to do so should be possible without a great problem.

Basically I wish to receive data from DB in ISO8559-1 locale (the system one, so default locale) and data should be converted to UTF8 to be displayed by DBAware components. The edition in DBAware components could could create a character that has no representation in current locale but in that case I have no problem in get a '?' mark or any other symbol, but I can not deal with continuos exceptions out of my control, blanked lines or the worst case characters with changed representation ('?' -> 'c').

I'll investigate it and try to create some kind of filter that allows me to get a clean ISO8559-1 -> UTF8 -> ISO8559-1 without too much code to be added in many places and without exceptions.

Thank you for your help.

Hello,

I had found part of my problems with the ISO8859-1 database and DBAware components, to be more precise the problem of empty strings when original data are not empty. The problem is the Utf8ToUnicode in the fpc RTL as this function when found an invalid UTF8 sequence (because it was originally "codedpage") it simply returns '' without any kind of "visual" notification like the usual '????' strings.

I know that an invalid UTF8 sequence ruins the whole string and I do not know if should be desirable to continue parsing adding a mask char like the '?' one.

PS: Maybe this discussion should be moved to the FPC list ?

Why is there an invalid UTF-8 string?

I know that an invalid UTF8 sequence ruins the whole string and I do not know if should be desirable to continue parsing adding a mask char like the '?' one.

You can safely continue parsing. You can even jump to somewhere into an UTF-8 string and find the next character start. The problem is that you can no longer reverse without data loss. OTOH returning an empty string is even more data loss ... .

PS: Maybe this discussion should be moved to the FPC list ?

The conversion must be somewhere, either in the db controls (lazarus), or in the db connector (fcl). Because the FCL prefers system encoding it might be a more lazarus problem.

Mattias

Hello Mattias,

MG> Why is there an invalid UTF-8 string?

Because the string is locale with a '?' char and next char produces an invalid UTF8 sequence and the UTF8 decoder of FCL simply clears the string and returns.

MG> You can safely continue parsing. You can even jump to somewhere MG> into an UTF-8 string and find the next character start. MG> The problem is that you can no longer reverse without data loss. MG> OTOH returning an empty string is even more data loss ... .

That's OK as the coding is wrong (whichever cause, it is not important).

> PS: Maybe this discussion should be moved to the FPC list ?

MG> The conversion must be somewhere, either in the db controls (lazarus), MG> or in the db connector (fcl). Because the FCL prefers system encoding it MG> might be a more lazarus problem.

But this "bug" is located in "wustrings.inc" file which comes from FPC RTL and it has been found when looking for a solution for my DB problems. This function is being called before each DRAWTEXT call when String is being used, so in DB components everytime. When the function finds an invalid UTF8 sequence the result is a blank (zero bytes) widestring.

I had ported my own UTF8Decode function from an old VB code and it works as expected from my point of view (except for the $FFFF unicode char which is reported as wrong, but it will be solved) so maybe the FPC team wants to replace their code with this one which passes the usual stress tests available in internet. Of course it is a bit more slow as it checks the invalid sequences more strictly.