Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2012-03

Variant vs Pointer

17 messages · 7 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Marcos Douglas

Hi,

The TField class have the FValueBuffer: Pointer attribute and have also SetData(Buffer: Pointer,...) method that all T***Field subclasses use to set themselves.
The methods AsInteger, AsString, etc use FValueBuffer to convert in an especific type. So, FValueBuffer could be anything.

My question is:
What would be the difference in performance if FValueBuffer were a Variant type?

Marcos Douglas

Reply by email

Marcos Douglas
to Marcos Douglas
Marcos Douglas wrote 12 lines
Hi, The TField class have the FValueBuffer: Pointer attribute and have also SetData(Buffer: Pointer,...) method that all T***Field subclasses use to set themselves. The methods AsInteger, AsString, etc use FValueBuffer to convert in an especific type. So, FValueBuffer could be anything. My question is: What would be the difference in performance if FValueBuffer were a Variant type? Marcos Douglas

Nobody knows?

I need implements a similar structure but I need know if the use of Variant have very cost.

Marcos Douglas

Reply by email

Unknown author
to Marcos Douglas
Marcos Douglas wrote 18 lines
On Wed, Feb 29, 2012 at 10:38 PM, Marcos Douglas wrote: > Hi, > > The TField class have the FValueBuffer: Pointer attribute and have > also SetData(Buffer: Pointer,...) method that all T***Field subclasses > use to set themselves. > The methods AsInteger, AsString, etc use FValueBuffer to convert in an > especific type. So, FValueBuffer could be anything. > > My question is: > What would be the difference in performance if FValueBuffer were a Variant type? > > Marcos Douglas Nobody knows? I need implements a similar structure but I need know if the use of Variant have very cost.

It has a cost.

The reason for using a buffer in tdataset is that a record's data normally is located in one continuous buffer, from which a value is picked from the right spot (including strings) You can't have that with variants.

Michael.

Reply by email

Marcos Douglas
to Unknown author
Show quoted text 32 lines
On Fri, 2 Mar 2012, Marcos Douglas wrote: > On Wed, Feb 29, 2012 at 10:38 PM, Marcos Douglas wrote: >> >> Hi, >> >> The TField class have the FValueBuffer: Pointer attribute and have >> also SetData(Buffer: Pointer,...) method that all T***Field subclasses >> use to set themselves. >> The methods AsInteger, AsString, etc use FValueBuffer to convert in an >> especific type. So, FValueBuffer could be anything. >> >> My question is: >> What would be the difference in performance if FValueBuffer were a >> Variant type? >> >> Marcos Douglas > > > Nobody knows? > > I need implements a similar structure but I need know if the use of > Variant have very cost. It has a cost. The reason for using a buffer in tdataset is that a record's data normally is located in one continuous buffer, from which a value is picked from the right spot (including strings) You can't have that with variants.

Hm... right.
So, if I will have a class like a TParam, that have FValue: Variant attribute, Would you change the implementation to not use Variant?

Marcos Douglas

Reply by email

Ludo Brands
to Marcos Douglas
Show quoted text 17 lines
>> I need implements a similar structure but I need know if the use of >> Variant have very cost. > > > It has a cost. > > The reason for using a buffer in tdataset is that a record's data > normally is located in one continuous buffer, from which a value is > picked from the right spot (including strings) You can't have that > with variants. Hm... right. So, if I will have a class like a TParam, that have FValue: Variant attribute, Would you change the implementation to not use Variant?

Not sure what you want to do but as Michael explained, the internal format in tdataset is very much linked to the source of the data. Databases don't understand pascal variants. They use only strictly typed data on the wire. Using a variant as an internal format would only introduce overhead and eventually data loss (fe. precision on decimals) without any gain. The TField variant properties should be considered a convenience only and be avoided as much as possible.

If your new class has no links with the outside world you can use whatever you want.

Ludo

Reply by email

Alberto Narduzzi
to Marcos Douglas: Hm... right. So, if I will have a class like a TParam, that have FValue: Variant…
Show quoted text 7 lines
> The reason for using a buffer in tdataset is that a record's data normally > is located in one continuous buffer, from which a value is picked from the > right spot (including strings) You can't have that with variants. Hm... right. So, if I will have a class like a TParam, that have FValue: Variant attribute, Would you change the implementation to not use Variant?

your class could use whatever you want, of course, but the overhead of the Variant (a silly invention, IMHO, most probably MS related...) isn't worth.
Your programs, more or less, will always know what data they are working with, hence you'll be safe and satisfied enough with the TField interface. Don't generalize if there isn't actually any need for it.

I may be wrong, of course, but I strongly doubt it... ;-)

Just my 2c

Cheers, A.

Reply by email

Marcos Douglas
to Ludo Brands: Not sure what you want to do but as Michael explained, the internal format in tdataset is…
Ludo Brands wrote 28 lines
> >> I need implements a similar structure but I need know if > the use of > >> Variant have very cost. > > > > > > It has a cost. > > > > The reason for using a buffer in tdataset is that a record's data > > normally is located in one continuous buffer, from which a value is > > picked from the right spot (including strings) You can't have that > > with variants. > > Hm... right. > So, if I will have a class like a TParam, that have FValue: > Variant attribute, Would you change the implementation to not > use Variant? > Not sure what you want to do but as Michael explained, the internal format in tdataset is very much linked to the source of the data. Databases don't understand pascal variants. They use only strictly typed data on the wire. Using a variant as an internal format would only introduce overhead and eventually data loss (fe. precision on decimals) without any gain. The TField variant properties should be considered a convenience only and be avoided as much as possible. If your new class has no links with the outside world you can use whatever you want.

Think in my class like a Business Object.
The struct is very similar with TParams it has many TParam. The data, in a TParam, is save in a Variant type.
I need something like:
u := TmyBO.Create; // the name does matter
u.Attr['name'].Value := 'Marcos'; // an instance is created and the name is 'name' and the type is string

The attributes are created on demand and the data is saved into a Variant, like TParam.
If I do not use a Variant, I need to create one class to each type that I will use, right?

If Variant is so bad, why TParam is implemented using it?

Marcos Douglas

Reply by email

Marcos Douglas
to Alberto Narduzzi: your class could use whatever you want, of course, but the overhead of the Variant (a…
Alberto Narduzzi wrote 24 lines
>> The reason for using a buffer in tdataset is that a record's data >> normally >> is located in one continuous buffer, from which a value is picked from >> the >> right spot (including strings) You can't have that with variants. > > > Hm... right. > So, if I will have a class like a TParam, that have FValue: Variant > attribute, Would you change the implementation to not use Variant? your class could use whatever you want, of course, but the overhead of the Variant (a silly invention, IMHO, most probably MS related...) isn't worth. Your programs, more or less, will always know what data they are working with, hence you'll be safe and satisfied enough with the TField interface. Don't generalize if there isn't actually any need for it. I may be wrong, of course, but I strongly doubt it... ;-) Just my 2c Cheers, A.

Well, if I won't generalize I need to create one class for each type, like TField does, but saving the data in a especific type variable. I would like to avoid it but...

Marcos Douglas

Reply by email

Ludo Brands
to Marcos Douglas: Think in my class like a Business Object. The struct is very similar with TParams it has…
Show quoted text 14 lines
Think in my class like a Business Object. The struct is very similar with TParams it has many TParam. The data, in a TParam, is save in a Variant type. I need something like: u := TmyBO.Create; // the name does matter u.Attr['name'].Value := 'Marcos'; // an instance is created and the name is 'name' and the type is string The attributes are created on demand and the data is saved into a Variant, like TParam. If I do not use a Variant, I need to create one class to each type that I will use, right?

Or define an internal storage format and convert the different types to that format. That is also what variants are doing. The difference is that variants have pre-defined assignment routines and conversions between the different variant types which don't always fit what you want (date formats, decimal points, precision, characters set, etc.). Assign fe. 1.2 to a variant and it will store it as a double while you perhaps meant a 32 digit precision numeric value.

If Variant is so bad, why TParam is implemented using it?

That is a very good question. Delphi compatibility is probably the answer.

Just look at the VarFmtBCD custom variant format created to fit the higher precision numeric data types and all the difficulties to get it right.

Ludo

Reply by email

Marcos Douglas
to Ludo Brands
Ludo Brands wrote 22 lines
> > Think in my class like a Business Object. > The struct is very similar with TParams it has many TParam. The data, > in a TParam, is save in a Variant type. > I need something like: > u := TmyBO.Create;  // the name does matter > u.Attr['name'].Value := 'Marcos';  // an instance is created and the > name is 'name' and the type is string > > The attributes are created on demand and the data is saved into a > Variant, like TParam. > If I do not use a Variant, I need to create one class to each type > that I will use, right? > Or define an internal storage format and convert the different types to that format. That is also what variants are doing. The difference is that variants have pre-defined assignment routines and conversions between the different variant types which don't always fit what you want (date formats, decimal points, precision, characters set, etc.). Assign fe. 1.2 to a variant and it will store it as a double while you perhaps meant a 32 digit precision numeric value.

I'm thinking to use JSON (TJSONData and subclasses) or at least the same idea. Could you give an example to use a internal storage format? Do you talk about Pointer and conversions?

Show quoted text 9 lines
> If Variant is so bad, why TParam is implemented using it? > That is a very good question. Delphi compatibility is probably the answer. Just look at the VarFmtBCD custom variant format created to fit the higher precision numeric data types and all the difficulties to get it right. Ludo

Well, TParams in Delphi works fine.
My new class will works with DBMS data format (integer, varchar, etc) very similar with TParams (I inherited of TParams, at the moment) with more functionalities.

The TParams class, AFAIK, should work well with DBMS data format, don't?

Marcos Douglas

Reply by email

Luiz Americo Pereira Camara
to Marcos Douglas: Think in my class like a Business Object. The struct is very similar with TParams it has…
Marcos Douglas wrote 8 lines
Think in my class like a Business Object. The struct is very similar with TParams it has many TParam. The data, in a TParam, is save in a Variant type. I need something like: u := TmyBO.Create; // the name does matter u.Attr['name'].Value := 'Marcos'; // an instance is created and the name is 'name' and the type is string

TJSONObject (unit fpjson) does exactly that

I use this way extensively

Luiz

Reply by email

silvioprog
to Luiz Americo Pereira Camara

2012/3/3 Luiz Americo Pereira Camara :

Show quoted text 15 lines
On 3/3/2012 10:08, Marcos Douglas wrote: > > Think in my class like a Business Object. > The struct is very similar with TParams it has many TParam. The data, > in a TParam, is save in a Variant type. > I need something like: > u := TmyBO.Create;  // the name does matter > u.Attr['name'].Value := 'Marcos';  // an instance is created and the > name is 'name' and the type is string TJSONObject (unit fpjson) does exactly that I use this way extensively Luiz

Me too. TJSONObject is very nice. Please see Fields and Params in:

https://github.com/silvioprog/lazwebsolutions/blob/master/Core/LWSCGI.pas

In my personal tests I noticed that TJSONObject is very fast.

Silvio Clécio ==================================== Site - <silvioprog.com.br> LazSolutions - <code.google.com/p/lazsolutions> ====================================

Reply by email

Marcos Douglas
to silvioprog
silvioprog wrote 22 lines
2012/3/3 Luiz Americo Pereira Camara : > On 3/3/2012 10:08, Marcos Douglas wrote: >> >> Think in my class like a Business Object. >> The struct is very similar with TParams it has many TParam. The data, >> in a TParam, is save in a Variant type. >> I need something like: >> u := TmyBO.Create;  // the name does matter >> u.Attr['name'].Value := 'Marcos';  // an instance is created and the >> name is 'name' and the type is string > > TJSONObject (unit fpjson) does exactly that > > I use this way extensively > > Luiz Me too. TJSONObject is very nice. Please see Fields and Params in: https://github.com/silvioprog/lazwebsolutions/blob/master/Core/LWSCGI.pas In my personal tests I noticed that TJSONObject is very fast.

Thanks Luiz and Silvio.
If I will implement using subclasses for each type, I think this is a good choice for now.

But I'm still curious about the Ludo's implementation.

Marcos Douglas

Reply by email

Ludo Brands
to Marcos Douglas: I'm thinking to use JSON (TJSONData and subclasses) or at least the same idea. Could you…
Show quoted text 17 lines
> Or define an internal storage format and convert the different types > to that format. That is also what variants are doing. The difference > is that variants have pre-defined assignment routines and conversions > between the different variant types which don't always fit what you > want (date formats, decimal points, precision, characters set, etc.). > Assign fe. 1.2 to a variant and it will store it as a double while you > perhaps meant a 32 digit precision numeric value. I'm thinking to use JSON (TJSONData and subclasses) or at least the same idea. Could you give an example to use a internal storage format? Do you talk about Pointer and conversions?

When you are thinking of using JSON you can use AnsiString as the internal format eventually adding a datatype tag to avoid conversion data loss as much as possible.
I have never used TJSONData before but used a homebrewn json implementation to transfer arbitrary database data.

Show quoted text 21 lines
>> If Variant is so bad, why TParam is implemented using it? >> > > That is a very good question. Delphi compatibility is probably the > answer. > > Just look at the VarFmtBCD custom variant format created to fit the > higher precision numeric data types and all the difficulties to get it > right. > > Ludo Well, TParams in Delphi works fine. My new class will works with DBMS data format (integer, varchar, etc) very similar with TParams (I inherited of TParams, at the moment) with more functionalities. The TParams class, AFAIK, should work well with DBMS data format, don't?

Again, the higher precision numeric and decimal datatypes as used in fe. Oracle are poorly supported with variants. If higher precision numerics, date/time formats, decimal points and other locale related conversions aren't an issue for you, and if you don't want strict compile time type checking, please do use variants.

Ludo

Reply by email

Marcos Douglas
to Ludo Brands
Ludo Brands wrote 23 lines
> > Or define an internal storage format and convert the > different types > > to that format. That is also what variants are doing. The > difference > > is that variants have pre-defined assignment routines and > conversions > > between the different variant types which don't always fit what you > > want (date formats, decimal points, precision, characters > set, etc.). > > Assign fe. 1.2 to a variant and it will store it as a > double while you > > perhaps meant a 32 digit precision numeric value. > > I'm thinking to use JSON (TJSONData and subclasses) or at > least the same idea. Could you give an example to use a > internal storage format? Do you talk about Pointer and conversions? > When you are thinking of using JSON you can use AnsiString as the internal format eventually adding a datatype tag to avoid conversion data loss as much as possible. I have never used TJSONData before but used a homebrewn json implementation to transfer arbitrary database data.

Well, if I will use JSON, I can use TJSONData for this job because it have all conversions types were implemented.

Show quoted text 27 lines
> >> If Variant is so bad, why TParam is implemented using it? > >> > > > > That is a very good question. Delphi compatibility is probably the > > answer. > > > > Just look at the VarFmtBCD custom variant format created to fit the > > higher precision numeric data types and all the > difficulties to get it > > right. > > > > Ludo > > Well, TParams in Delphi works fine. > My new class will works with DBMS data format (integer, > varchar, etc) very similar with TParams (I inherited of > TParams, at the moment) with more functionalities. > > The TParams class, AFAIK, should work well with DBMS data > format, don't? > Again, the higher precision numeric and decimal datatypes as used in fe. Oracle are poorly supported with variants. If higher precision numerics, date/time formats, decimal points and other locale related conversions aren't an issue for you, and if you don't want strict compile time type checking, please do use variants.

Thanks for these tips.
I will use the JSON implementation in fpjson unit.

Marcos Douglas

Reply by email

Marco van de Voort
to Ludo Brands: When you are thinking of using JSON you can use AnsiString as the internal format…

In our previous episode, Ludo Brands said:

When you are thinking of using JSON you can use AnsiString as the internal format eventually adding a datatype tag to avoid conversion data loss as much as possible.

ansistring is a decimal type. This incurs a binary to decimal conversion per se (which is afaik not needed in SQL if you use parameters)

Show quoted text 6 lines
> The TParams class, AFAIK, should work well with DBMS data > format, don't? > Again, the higher precision numeric and decimal datatypes as used in fe. Oracle are poorly supported with variants

Variants or the current sqldb implementation?

If higher precision numerics,a date/time formats, decimal points and other locale related conversions aren't an issue for you, and if you don't want strict compile time type checking, please do use variants.

Btw, there is a third route if you don't like stringly typing. In a past job I had a "import" app that imported the schemas from the db, transformed them (with additional lists of "exceptions") and generated pascal source code for all the associated objects and collections and the queries to load/store them.

For really big jobs with a lot of exceptions (e.g. homegrown database layouts that are determined externally) this can be a great solution.

Reply by email

Ludo Brands
to Marco van de Voort
Show quoted text 10 lines
In our previous episode, Ludo Brands said: > When you are thinking of using JSON you can use AnsiString as the > internal format eventually adding a datatype tag to avoid conversion > data loss as much as possible. ansistring is a decimal type. This incurs a binary to decimal conversion per se (which is afaik not needed in SQL if you use parameters)

In oracle the number data type is closer to decimal than binary. Binary integer is converted to number for storage and binary float binary double are fairly recent data types (>=10g). So it is dependant on the database back-end.

There are many other oracle data types, far less used, that have to be represented in their string representation such as timestamp (up to 9 digits fractional second precision), timestamp with timezone and the interval data types.

Show quoted text 10 lines
> > The TParams class, AFAIK, should work well with DBMS data > > format, don't? > > > > Again, the higher precision numeric and decimal datatypes as used in > fe. Oracle are poorly supported with variants Variants or the current sqldb implementation?

Variants in general. The current sqldb implementation does a good effort with the VarFmtBCD custom variant type although its use requires a carefully check of every variant assignment and operation to be sure there are no hidden type conversions involved. The compiler will not do it for you.

Ludo

Reply by email