Hi,

I working on a Firebird database that has a default Charset = UTF8. I'm using the SqlDB package via tiOPF to save and read my data from the database.

I have a table with a field defined as Char(2) and I have set it with a value "en" which represents a Country Code.

When I read back that value, by TCountry unit tests fail with the following message.

==========================
1) textrunner.SQL Database tests.TTestCountry.TestCountry_ReadList: ETestFailure
at $0807F9B4
"Check #1: Failed on ID
Expected:
"en"
But was:
"en "
==========================

Further investigation revealed that when I read back a Char(2) field via TField.AsString the resulting string is 8 bytes wide. I can see where this value comes from - Firebird obviously uses 4 bytes per UTF-8 character, because that is the largest amount of bytes a UTF-8 character can use. Not very efficient, but that is what Firebird 2.1 does when the Charset = UTF8.

Now I was wondering how to resolve this.

1) Switch all my Char() fields to VarChar() which resolves the problem.

2) Trim(lField.AsString) before I store it in each TCountry.Code property.

3) SqlDB does the TField.AsString trimming for any Char() fields.

Is option #3 viable? This will resolve any similar issue to all future developers using Firebird with Char() type fields and Charset=UTF8.

Or should I opt for options #1 or #2 instead. Option #1 is probably the least amount of effort because the project I am working on is new, so there is no existing database that need to be converted. Problem being that another developer down the line might add a new field of type Char() and then we sit with the same problem again.

What is your thoughts on this. As far as I know MSEgui does automatic trimming of spaces in the TField, so should this maybe be done in SqlDB (Interbase/Firebird) as well?

Regards,
- Graeme -

You may never trim these fields;
The database contains always the declared number of characters: the value is trimmed with a space till the number of declared characters is reached. TField.AsString returns what is in the database, which seems to be 8 characters in your case.

No, because it is fundamentally wrong.

The question should be:
Why does Firebird report 8 characters to your application ?

Michael.

Yes Firebird returns 8 characters for a Char(2) database field. Firebird is very happy with inserting "en", but when read it back with FPC, it returns "en "

If I run a select query using the FlameRobin database tool against that exact same database and table, FlameRobin returns "en". FPC's SqlDB doesn't.

==============[ database DDL ]==================== CREATE DOMAIN D_OID AS Char(36)

  COLLATE UTF8;

CREATE TABLE COUNTRY(
OID D_OID NOT NULL,
ID Char(2) NOT NULL,
NAME Varchar(50) NOT NULL,
CONSTRAINT PK_COUNTRY PRIMARY KEY (OID)
);
=================[ end ]====================

When I ask FlameRobin to export the Country table data as INSERT statements, it does it correctly with the correct spacing as shown below.

==========================
INSERT INTO COUNTRY (OID, ID, NAME) VALUES ('ZA

          ', 'ZA', 'South Africa');

==========================

36 characters for the OID field and 2 characters for the ID field. Just like I defined it in the DDL.

So why does FlameRobin handle Char() fields as I expected, but FPC's SqlDB does something different. So who is right and who is wrong?

No, because it is fundamentally wrong.

I did not mean with a simple Trim() function. I meant being more precise and still return the correct amount of characters as defined in the DDL. Or maybe do some UTF8ToString() conversion or something if field charset = UTF8.

The question should be: Why does Firebird report 8 characters to your application ?

Only via FPC's SqlDB - not via FlameRobin. See above.

Regards,
- Graeme -

That is why I wrote 'to your application', obviously.

This will require a serious amount of investigation and debugging of TDataset and TSQLConnection. Because according to IBConnection.pp, the field size is determined solely by what firebird reports in SQLLen:

   else case (SQLType and not 1) of
     SQL_VARYING,SQL_TEXT :
       begin
         TrType := ftString;
         if SQLLen > dsMaxStringSize then
           TrLen := dsMaxStringSize
         else
           TrLen := SQLLen;
       end;

Which means firebird reports 8 to your application.

Michael.

Michael Van Canneyt wrote

Which means firebird reports 8 to your application.

So does this issue only occur on my system and not on others? Do you uses charset=UTF8 in your databases, and Firebird reports the correct size for Char() fields?

I'll put together a small test case and sample Firebird database so I can rule out tiOPF as a possible cause. I'll also double check using different database components. I have SqlDB, FBLib and IBX (from Delphi
7) available here.

I'll see if I can put together a C-source example as well, which talks directly to the firebird library. Hell, I can't even remember when last I wrote a C program - hopefully the included Firebird examples will help. :-(

If all goes well, can narrow down the search. I would be nice to know if this issue is only on my system though. Could anybody else test this?

Regards,
- Graeme -

I never use Char fields, and I also don't use charsets in Firebird...

I'll put together a small test case and sample Firebird database so I can rule out tiOPF as a possible cause. I'll also double check using different database components. I have SqlDB, FBLib and IBX (from Delphi 7) available here.

IBX will present problems; I had problems with field sizes when using IBX and Delphi 7 and firebird 2.1, which seems to use UTF-8 for metadata. It might be interesting to see how they solved the problem in later Delphis.

Michael.

Op dinsdag 21-07-2009 om 16:41 uur [tijdzone +0200], schreef Michael Van Canneyt:

Yes, Firebird creates a char field with size 8 when the charset is set to utf8, and you pass a width of 2. It's a documented feature, and could look strange at first when using char-fields. But it's completely normal.

And I guess that Flamerobin always trims char-fields. Which is ok when you are showing data to the end-users, but not for the connection-component.

Joost.

Op dinsdag 21-07-2009 om 17:23 uur [tijdzone +0200], schreef Michael Van Canneyt:

Delphi doesn't support Firebird at all for now.

Joost

It depends what you mean with Support. I use D7, IBX with about 1000 Firebird databases since 6 years without any problems.

So 'support' is relative :-)

Michael.

Michael Van Canneyt wrote

I never use Char fields,

I can live with that...

and I also don't use charsets in Firebird...

..but not with this! Here are just a few issues with using NONE as the default charset in a Firebird database.

* Firebird does NOT default to your system locale. It ignores anything on you OS, so all databases on any OS in any locale are treated the same.

* Firebird can't correctly uppercase, lowercase or no accent non-ASCII characters

* Firebird sorting of non-ASCII characters very wrong for most occasions

* Nothing in the database itself documents how the bytes stored should be interpreted as characters

* When the same database is simultaneously or in succession used by systems with different system default character sets, the data will silently change meaning.

Our system will use accented characters and sorting will be important. We will also be accessing data from various locations (differing locales).

I'm Google'ing now for "firebird charset utf8" and reading the various articles. But I'll still be testing FBLib and IBX to see what they do and if they handle it the same as SqlDB.

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I think what Joost is saying is that since D7, IBX has been dropped (I think) and dbExpress is the replacement. By default there is no driver for Firebird with dbExpress, but you can purchase 3rd party drivers for dbExpress+firebird.

CodeGear is _considering_ (not final yet) adding Firebird support in post-D2009 releases due to popular demand.

So 'support' is relative :-)

That statement still holds true. :)

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

No problem, since this is Windows, all with the same CP.

* Firebird can't correctly uppercase, lowercase or no accent non-ASCII characters

All this is handled on the client app. For the 4 fields in the database where it matters, we have a 2nd (indexed) field which contains a lowercased value of the value of the original field. Takes slightly more room, but is much faster.

* Firebird sorting of non-ASCII characters very wrong for most occasions

Sorting never happens by firebird, all on client.

* Nothing in the database itself documents how the bytes stored should be interpreted as characters

All handled by app.

* When the same database is simultaneously or in succession used by systems with different system default character sets, the data will silently change meaning.

Not a problem in my context.

Just to say that using NONE doesn't need to be a problem in all cases.

In general, I think your approach is correct, though.

IBX does, as far as I can gather.

Michael.

Later, I think.

and dbExpress is the replacement. By default there is no driver for Firebird with dbExpress, but you can purchase 3rd party drivers for dbExpress+firebird.

Since firebird comes with 'legacy gds32.dll' install option, IBX or the interbase driver should work fine. The API hasn't changed in years.

Always has :-)

Michael.

Op dinsdag 21-07-2009 om 19:30 uur [tijdzone +0200], schreef Michael Van Canneyt:

We also use it, but I know a few things that won't work...

Joost.

Graeme Geldenhuys wrote

What is your thoughts on this. As far as I know MSEgui does automatic trimming of spaces in the TField, so should this maybe be done in SqlDB (Interbase/Firebird) as well?

Don't confuse Firebird field datasize (in bytes) and field character length (in characters/code points). MSEgui SqlDB fork queries the character length by metadata. The utf-8 encoded database text will be converted to widestring and stored in the dataset by variable length widestrings. tmsestringfield.size will be set to the character length instead to the data length.
Firebird trims CHAR fields for UTF8 to character length (byte length = 4*character length) and with the old UNICODE_FSS to byte length (= 3*character length), see
http://www.destructor.de/firebird/charsets.htm

Martin

OK Michael, I did some more research inside the system tables of a Firebird database. There is a field that stores the "length" in bytes and the "length" in characters. By default it seems that SqlDB only every looks at the "byte length"

See the select statement below which retrieves metadata for a know table. fields in question are "f.rdb$field_length" and "f.rdb$character_length"

What is SqlDB looked at the meta data of a string field and if it is set to UNICODE_FSS (pre v2 setting) or UTF8 (post v2 setting) and use the appropriate field to determine the "length".

So if the rdb$field_type is 14 (TEXT) or 37 (VARYING) and maybe 40 (CSTRING) and rdb$character_set_id = 4 (UTF8), then it rather reports the rdb$character_length, rather than the rdb$field_length (bytes).

That way, SqlDB can return copy(fieldvaluestring, 0, character_len) as the actual field text value, which trims off the padding of spaces.

SELECT
r.rdb$field_name as field_name
,f.rdb$field_type as field_type
,f.rdb$field_sub_type as field_sub_type
,f.rdb$field_length as field_length
,f.rdb$character_length as field_charlength
,f.RDB$CHARACTER_SET_ID as field_charset_id
,c.rdb$character_set_name as field_charset
FROM
rdb$fields f
INNER JOIN
rdb$relation_fields r ON f.rdb$field_name = r.rdb$field_source LEFT OUTER JOIN
RDB$CHARACTER_SETS c ON f.rdb$character_set_id = c.rdb$character_set_id WHERE
r.rdb$relation_name = 'MODULE'

Regards,
- Graeme -

This is not feasable, because it works only for actual fields. Expressions for instance will not work, and secondly, it requires a second query per executed query, which is simply unacceptable from a performance point of view. So this proposal is not an option.

Firebird reports the theoretical length of each field in the result set, we should use that in all cases.

That way, SqlDB can return copy(fieldvaluestring, 0, character_len) as the actual field text value, which trims off the padding of spaces.

If you look carefully, you'll see that the padding of spaces happens in code in the case of a CHAR field. Maybe we should do something about that.

Michael.

Michael Van Canneyt wrote

This is not feasable, because it works only for actual fields. Expressions for instance will not work,

I have not tried expressions yet, but will give it a try and see what Firebird actually does. No point in assuming or guessing. ;-)

and secondly, it requires a second query per executed query, which is simply unacceptable from a performance point of view.

No, not a second query, just keeping track of extra (other) meta data information which was retrieved from the first API call to Firebird.

Firebird reports the theoretical length of each field in the result set, we should use that in all cases.

In that case, TField and TParam is NOT Delphi compatible.

From the Kylix 3 and Delphi 7 documentation:

=============================
TField.Size
-----------
Returns various sizes based on data type.

ftString Size is the maximum number of characters in the string. ...
=============================

TParam.Size
-----------
Specifies the number of characters in a string-type parameter. =============================

In both cases, Size is the number of "characters" in the string, NOT the size in bytes as the current SqlDB - Firebird implementation is doing.

So Char(2) should return Size = 2. Irrespective if the character set in the database is ASCII or UTF8 etc... So SqlDB should be using the "f.rdb$character_length" field and *not* the "f.rdb$field_length" for string types.

Regards,
- Graeme -

I'm fine with padding for CHAR() field types if the content is less than the Char() length.

eg: Char(5) field definition should always return 5 characters irrespective if you insert only 2 character of data.

The UTF8 implementation is greatly flawed in Firebird. It does not adhere to the max character length as defined by Char(x).

The UTF-8 value of "en" IS "en" because UTF-8 is a variable byte length implementation. Plus the first 254 (there abouts) characters in UTF-8 only take up 1 byte per character. The UTF-8 encoded string of "en" in NOT "en " like Firebird is returning!

In Summary:
-------------
There are two problems here.

1) SqlDB and Firebird are reporting the wrong TParam.Size and TField.Size results. SqlDB is using byte length instead of character length.

2) The UTF-8 implementation of Firebird is seriously flawed. Firebird makes as if UTF-8 is a fixed byte algorithm and just returns rubbish results from a Char(x) field and breaks the DDL rule of what the maximum character length is. Using the metadata, SqlDB *can* fixes this by using something like: copy(fieldvalue, 0, MaxCharacterLength)

Regards,
- Graeme -

My databases are HUGE, and I don't think that such a query is appropriate.

Given that neither supports UTF-8, the documentation is not really relevant, I'd say.

I agree that we need a solution, but I'm not convinced your solution is correct or even desirable.

Don't forget also that for unicode, the number of characters differs from the number of bytes. The Firebird API predates this, and so does the Delphi TDataset implementation. Currently I think that introducing a new field type TUTF8field or TUnicodeField is the right way to go. This field can then take care of any conversion issues.

The other solution is to change the interbase layer so it reports a TWideStringField instead of TField if the charset is UTF-8.

Michael.

Applies for UNICODE_FSS not UTF8. As I wrote before: "
Firebird trims CHAR fields for UTF8 to character length (byte length = 4*character length) and with the old UNICODE_FSS to byte length (= 3*character length), see
http://www.destructor.de/firebird/charsets.htm
"

Martin

Michael Van Canneyt wrote

> No, not a second query, just keeping track of extra (other) meta data > information which was retrieved from the first API call to Firebird. My databases are HUGE, and I don't think that such a query is appropriate.

It's got nothing to do with the size of your database. It is simply SqlDB that is using the wrong field to report the size of the Char(x) field definitions.

Lucky for you, Embarcadero now has all its Delphi help available online. Nothing has changed in Delphi 2009 help. Here are links to D2009's online help. In both cases, the Size property is referring to characters and not byte length.

http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/DB_TParam_Size.html http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/DB_TField_Size.html

So SqlDB with Firebird is in fact wrong when it returns Size = 8 for a Char(2) with UTF8 charset enabled.

I agree that we need a solution, but I'm not convinced your solution is correct or even desirable.

Well, Firebird makes no sense regarding it's behaviour. If you have an UTF-8 encoded string as follows:

  s := 'en';   // assume s is a UTF8 String type

What is the length of that string? Firebird would argue that it's 8 bytes. But the Unicode organisation says it's 2 bytes - I tend to agree. The ASCII character set is represented in UTF-8 and they work as they did in ASCII. They also only take up 1 byte per character.

Firebird now tells me that the content of the variable s is now equivalent to "en " when read back from the DB, but it is definitely not the case. "en" in ASCII or UTF-8 is still only "en" without the rubbish padding!

PostgreSQL also supports the UTF-8 character set in databases. Surprise, surprise TParam.Size and TField.Size report the value of Char(x). Also the return values read from the Char(x) field don't contain any space padding on the right unless the actual text is less than the Char(x) definition. Also the character length NEVER exceeds the Char(x) definition.

I'll report this issue to the Firebird developers as well. Whoever implemented the UTF-8 support in Firebird was a total idiot, and knew nothing about Unicode.

But in the mean time we can fix the SqlDB issue and work around the Firebird Char(x) issue as I explained before.

Don't forget also that for unicode, the number of characters differs from the number of bytes. The Firebird API predates this, and so does

Firebird was a total rewrite in C++ for v1.5 or v2 (I can't remember exactly which). That was pretty recent, so there is no excuse like legacy code for such crappy Unicode support.

Regards,
- Graeme -

Martin Schreiber wrote

> Applies for UNICODE_FSS not UTF8. As I wrote before:

UNICODE_FSS and UTF8 character set implementation as incorrectly implemented in Firebird. Firebird is changing the data when read back from the DB, by padding it with spaces when it is not required.

Firebird also violates the type definition of Char(x) when UTF8 charset is specified. Firebird returns a string longer than the maximum characters allowed in a Char(x) field.

Regards,
- Graeme -

Which field should it use according to you then ?

Yes, but assume that a size of 2 is returned. This means a buffer of 2 bytes (in ansistring byte=character) will be reserved for the data.

What happens if some strange unicode string of 4 or even 8 bytes is returned by Firebird ? A Buffer overflow...

So SQLDB "agrees with firebird" and reserves 8 bytes because that is the max what can be returned. The problem being that if an ansistring is requested, 1 char = 1 byte...

Hence my idea that TStringField is not suitable, but TUTF8Field is needed, where size 2 is reported as it should, but a buffer of 8 is reserved for the string. Or else a 'charsize' property must be introduced in TFieldDef and TField.

But the padding is added by SQLDB, not by firebird.

The problem is deeper than you see, and is not related to SQLDb, but to the implicit assumption in TBufDataset that for TStringField, 1 char = 1 byte:

The situation is that TBufDataset gets a fielddef for a string field, with size N. So it reserves a buffer of N bytes, because 1 char = 1 byte. This assumption is not correct for UTF-8 encoded strings.

As a consequence, my prediction is that, because it reports a size in characters, the postgres implementation will suffer of buffer overflows as soon as strange (=multibyte) unicode characters are returned.
Whereas the firebird implementation will not buffer overflow since it returns a size in bytes (but instead it suffers from the problem you encountered).

We need to foresee the possibility where the size in characters is different from the size in bytes. If I am correct, the TWideStringField accomodates for this, it uses a character size of 2.

Well, I'm not versed enough in Unicode to comment on their implementation.

I do know that they went to extreme lengths to be backwards compatible, which probably accounts for the weird behaviour.

I would like Joost to comment on the whole issue, he is best placed to discuss a possible remedy for this problem:

1. Foresee a charsize field in TStringField
2. Map UTF8 fields to widestring and convert them on the go.
3. Introduce a TUTF8field which equals TStringField, but which reserves a buffer of 4 bytes per character.

At that point, the firebird connection must be adapted to report the character length instead of the buffer length.

Michael.

Try to enter 'äöüéà' in a char(5) utf-8 encoded PQ field in SqlDb. ;-)

Martin

I see Martin understood since long what I was trying to say :-)

Michael.

Correct.

We need to foresee the possibility where the size in characters is different from the size in bytes. If I am correct, the TWideStringField accomodates for this, it uses a character size of 2.

Graeme will not accept it because he wants to use ancient Greek and Klingon AFAIK. ;-)
MSEgui uses variable size storage for strings in tmsebufdataset. It was difficult to implement but it is worth the effort.

Martin

Michael Van Canneyt wrote

Which field should it use according to you then ?

"f.rdb$character_length" to report TField.Size and TParam.Size See below...

OK Michael, you are confusing what TField.Size means. You also don't seem to take into account TField.DataSize. See the following URL.

http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/DB_TField_DataSize.html

TField.Size and TParam.Size report back the x number of "characters" irrespective of what character set is being used. This value should be the same as the Char(x) type definition.

TField.DataSize reports back the amount of bytes required to store the value.

Example 1.
-----------
Data stored is "en" in a field defined as Char(2) with UTF8 charset. UTF-8: 0x65
UTF-8: 0x6E
TField.Size must equal 2
TField.DataSize must equal 2 (1 byte per character)

Example 2.
-----------
Data stored is "豈更" in a field defined as Char(2) with UTF8 charset. UTF-8: 0xEF 0xA4 0x80
UTF-8: 0xEF 0xA4 0x81
TField.Size must equal 2
TField.DataSize must equal 6 (3 bytes per character)

Example 3.
-----------
Data stored is "e" in a field defined as Char(2) with UTF8 charset. UTF-8: 0x65
UTF-8: 0x20 // 1 space character for padding
TField.Size must equal 2
TField.DataSize must equal 2 (1 byte per character)

What happens if some strange unicode string of 4 or even 8 bytes is returned by Firebird ? A Buffer overflow...

In that case SqlDB is not using TField.DataSize like it is supposed to.

So SQLDB "agrees with firebird" and reserves 8 bytes because that is the max what can be returned.

Why to use the TField.DataSize to reserve the correct about of bytes.

But the padding is added by SQLDB, not by firebird.

Oh! It seems that FBLib also makes the same mistake as SqlDB then. I tested on both SqlDB and FBLib and they behaved the same, so assumed the fault seems to me, then it is more a general Object Pascal implementation mistake. Luckily I haven't posted a bug report to Firebird then.

I want to try using C source code, but I'm to rusty in C. I couldn't even get my test program to compile. :-)

The problem is deeper than you see, and is not related to SQLDb, but to the implicit assumption in TBufDataset that for TStringField, 1 char = 1 byte:

I think it's more a case of TField.DataSize not being taken into account, and always assumes TField.Size and TField.DataSize are the same for Char(x) field definitions.

As a consequence, my prediction is that, because it reports a size in characters, the postgres implementation will suffer of buffer overflows as soon as strange (=multibyte) unicode characters are

Just did a test. PostgreSQL reports back the correct TField.Size, but somewhere the content is being clipped. I ran this through a modified tiOPF with SqlDB_PG persistence layer.

I'll make a cleaner example that only uses SqlDB to confirm this issue.

==============================
There was 1 failure:

1) textrunner.SQL Database tests.TTestCountry.TestCountry_ReadList: ETestFailure
at $0807FA56
"Check #3: Failed on ID
Expected:
"豈更"
But was:
"豈e2""
==============================

Regards,
- Graeme -

I know, but I'm afraid of the huge memory usage and fragmentation it IMHO inevitably leads to. That's why I'm looking for an alternate solution at a deeper level.

Somehow Delphi 2009 must also be confronted with this, with their native unicode support. Hm.

Michael.

On a quick test, but via tiOPF, it seems to clip the UTF-8 text, but doesn't throw an error. My unit tests detected the mistake though. But I need a cleaner test that *only* uses SqlDB, which I will do next.

Graeme will not accept it because he wants to use ancient Greek and Klingon AFAIK. ;-)

You remembered! :-)

Regards,
- Graeme -

Good point.

> So SQLDB "agrees with firebird" and reserves 8 bytes because that is > the max what can be returned. Why to use the TField.DataSize to reserve the correct about of bytes.

-> This seems like a good solution. We'll have to look at this.

That's currently exactly so, and is what needs to be fixed, however, this is at a deeper level as SQLDB.

If it's clipped, that means the copy operation to the dataset buffer takes care of the overflow.

Michael.

Hello Michael,

Friday, July 24, 2009, 3:31:32 PM, you wrote:

MVC> We need to foresee the possibility where the size in characters is MVC> different from the size in bytes. If I am correct, the TWideStringField MVC> accomodates for this, it uses a character size of 2.

Same problem with TWideStringField, unicode can be 32 bits... TVeryWideString ? :)

But there are more problems (not tested recently) like:

Table with field NAME VARCHAR(5) UNICODE_FSS

Trying to write "abcdé" which is perfectly valid raises and exception in SQLDB as it is 6 bytes in length.

If you write this field with other application (not FPC) the field is accepted and at least in past versions SQLDB is unable to read it, or returns a malformed UTF8 String "abcd?".

Since that problems I'm using Firebird in SQLDB with UTF8 data over regular (VAR)CHAR fields without the UNICODE_FSS. I can not perform uppercase and like selects but at least works in a predictable way.

In our previous episode, Michael Van Canneyt said:

Somehow Delphi 2009 must also be confronted with this, with their native unicode support. Hm.

Afaik D2009 always works in 2 byte units. Surrogates are a problem of the user I guess.

I don't fully understand why this function must return exact characters and not e.g. words.

TField.Size is the declared size in characters. What the size of the data buffer is, is another matter, and depends largely on design decisions. That's what we need to resolve.

Michael.

In our previous episode, Michael Van Canneyt said:

The question is if it still does in D2009. I doubt it. I think it is encoding-base unit. (1 or 2)

Marco van de Voort wrote

> TField.Size is the declared size in characters. The question is if it still does in D2009. I doubt it. I think it is encoding-base unit. (1 or 2)

I do not have D2009, but the D2009 online help clearly states that the Size property is in characters for string type fields.

TField.Size, and then read about ftString:
http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/DB_TField_Size.html

TField.DataSize:
http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/DB_TField_DataSize.html

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Ah, this I don't know, I don't have D2009.

Michael.

In our previous episode, Michael Van Canneyt said:

> The question is if it still does in D2009. I doubt it. I think it is > encoding-base unit. (1 or 2) Ah, this I don't know, I don't have D2009.

Note that it was a guess based on what I have seen in D2009. Nowhere there is talk about surrogates, and while they say "character" they mean wchar_t not codepoints.

Op vrijdag 24-07-2009 om 12:49 uur [tijdzone +0200], schreef Graeme Geldenhuys:

We can add parameter for this in TSQLConnection, Delphi does the same, it's a quite common approach. (ie: an option: should char-fields be trimmed.

Which is should. You are thinking in a 'real-world' situation. But you have to look at the TDataset/TStringField model. It allocates DataSize bytes for each string per record. And, yes, in case of an utf-string, that has to be 4*character-count. This works perfectly fine.

They made some decisions you would have done differently. But sqldb is not there to hide db-features(bugs?) form the end-developer. It's only a layer to connect to the database. Post-processing should be done by the end-programmer, imho.

I don't have time to read the complete thread, but these are the basics.

Joost

Op vrijdag 24-07-2009 om 12:55 uur [tijdzone +0200], schreef Michael Van Canneyt:

This could also be a property, or some function that can be overloaded. (and then in such a way it works for all fieldkinds) This will make Martin Schreiber also happy, I think...

Joost.

Op vrijdag 24-07-2009 om 15:31 uur [tijdzone +0200], schreef Michael Van Canneyt:

And, Graeme, we need a field that is accesible through the Firebird-C-API, not by some query we can run.

1. Foresee a charsize field in TStringField

Yes, or (ab?)use .Size for that, as we also have .DataSize. Depends on Delphi2009, I would say.
Normally for TStringFields, Size is DataSize-1. I don't know if it leads to any problems when we change that.

And note that this is not implemented in sqldb, nor TBufDataset, but TDataset/TStringField itself...

2. Map UTF8 fields to widestring and convert them on the go.

With a property to set this, this seems ok

3. Introduce a TUTF8field which equals TStringField, but which reserves a buffer of 4 bytes per character.

Yes, or make it variable-length. In that case we can use part of
Martin's code. And not

At that point, the firebird connection must be adapted to report the character length instead of the buffer length.

Which can be difficult, I don't know if they added an api-call for this. Else we have to calculate the length ourselves by looping through the characters. But if TDataset should do that or the end-uprogrammer, I don't know.

Joost.