Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2011-10

sqldb how to find a record fast with primary key

13 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Felipe Monteiro de Carvalho

Hello,

I am using sqldb to connect to a postgres database. Using wiki instructions, my final object is a TSQLQuery for each table. So, this basically is a TDataset and to read fields I know that I can use First(), Next(), EOF() and RecNo

But these are rather slow. Many fields in the tables refer to the primary key in other tables, which does not match the RecNo. How can I jump fast a table to the record which has a particular PrimaryKey?

thanks,

Felipe Monteiro de Carvalho

Reply by email

Martin Schreiber
to Felipe Monteiro de Carvalho
Felipe Monteiro de Carvalho wrote 11 lines
Hello, I am using sqldb to connect to a postgres database. Using wiki instructions, my final object is a TSQLQuery for each table. So, this basically is a TDataset and to read fields I know that I can use First(), Next(), EOF() and RecNo But these are rather slow. Many fields in the tables refer to the primary key in other tables, which does not match the RecNo. How can I jump fast a table to the record which has a particular PrimaryKey?

Use a local index and tlocalindex.find() (MSEgui). SQLdb has local index too AFAIK.
MSEgui has additionally tlookupbuffer and fast lookup field linking by local index and direct data access without dataset scrolling.

Martin

Reply by email

Unknown author
to Felipe Monteiro de Carvalho
Felipe Monteiro de Carvalho wrote
Ping, No sqldb specific answers on this one? I did some googling and also searched for TPrimaryKey or LocalIndex in sqldb without results ... =(

To my knowledge there is no fast search/locate in sqldb. Maybe we should have a look at Martin Schreibers' implementation.

Michael.

Reply by email

Martin Schreiber
to Unknown author
Show quoted text 9 lines
On Fri, 21 Oct 2011, Felipe Monteiro de Carvalho wrote: > Ping, > > No sqldb specific answers on this one? I did some googling and also > searched for TPrimaryKey or LocalIndex in sqldb without results ... =( To my knowledge there is no fast search/locate in sqldb. Maybe we should have a look at Martin Schreibers' implementation.

Most code is in lib/common/db/msebufdataset.pas. Lookup buffers are in lib/common/db/mselookupbuffer.pas. Heavy stuff. ;-)

Martin

Reply by email

Felipe Monteiro de Carvalho
to Martin Schreiber
Martin Schreiber wrote
Most code is in lib/common/db/msebufdataset.pas. Lookup buffers are in lib/common/db/mselookupbuffer.pas. Heavy stuff. ;-)

How much from the other stuff would I need to compile your DB modules?

Any beginner guides to stablish a connection, open a table, read/write some fields, go to a different record, go to a record which has FieldX=Y?

Does it support Postgres?

Does it work in Mac OS X?

Felipe Monteiro de Carvalho

Reply by email

Unknown author
to Martin Schreiber: Most code is in lib/common/db/msebufdataset.pas. Lookup buffers are in…
Martin Schreiber wrote 12 lines
On Friday 21 October 2011 16.30:18 wrote: > On Fri, 21 Oct 2011, Felipe Monteiro de Carvalho wrote: >> Ping, >> >> No sqldb specific answers on this one? I did some googling and also >> searched for TPrimaryKey or LocalIndex in sqldb without results ... =( > > To my knowledge there is no fast search/locate in sqldb. > Maybe we should have a look at Martin Schreibers' implementation. > Most code is in lib/common/db/msebufdataset.pas. Lookup buffers are in lib/common/db/mselookupbuffer.pas. Heavy stuff. ;-)

Well, sqldb has the indexes. It's just an additional index routine.

Michael.

Reply by email

Ludo Brands
to Unknown author: To my knowledge there is no fast search/locate in sqldb. Maybe we should have a look at…
Show quoted text 10 lines
> > No sqldb specific answers on this one? I did some googling and also > searched for TPrimaryKey or LocalIndex in sqldb without results ... =( To my knowledge there is no fast search/locate in sqldb. Maybe we should have a look at Martin Schreibers' implementation. Michael.

You can use indices and locate with TSQLQuery as follows:

  SQLQuery1.AddIndex('idx_no_art','no_art',[]);
  SQLQuery1.IndexName:='idx_no_art';
  SQLQuery1.Open;
  ...
  SQLQuery1.Locate('no_art','200295',[]);

Unfortunately, Locate uses a simple linear search. Note also issue 0020514.

Ludo

Reply by email

Martin Schreiber
to Felipe Monteiro de Carvalho: How much from the other stuff would I need to compile your DB modules? Any beginner…
Felipe Monteiro de Carvalho wrote
On Fri, Oct 21, 2011 at 4:50 PM, Martin Schreiber
Show quoted text 5 lines
> Most code is in lib/common/db/msebufdataset.pas. Lookup buffers are in > lib/common/db/mselookupbuffer.pas. Heavy stuff. ;-) How much from the other stuff would I need to compile your DB modules?

The whole non visual MSEgui infrastructure.

Any beginner guides to stablish a connection, open a table, read/write some fields, go to a different record,

Almost the same as sqldb.

go to a record which has FieldX=Y?

Use tlookupbuffer.find() for example. In order to get a lookup value without scrolling use the returned bookmark in tmsebufdataset.currentbmas*[] or use a lookup field without setting LookupCache.

Does it support Postgres?

Yes.

Does it work in Mac OS X?

No.

BTW, because of the dead of BerliOS MSEide+MSEgui and MSEuniverse are now Gitorious:

http://gitorious.org/mseide-msegui
http://gitorious.org/mseuniverse

The SVN repositories are not updated anymore.

Martin

Reply by email

Felipe Monteiro de Carvalho
to Ludo Brands: You can use indices and locate with TSQLQuery as follows:…
Ludo Brands wrote 6 lines
You can use indices and locate with TSQLQuery as follows:  SQLQuery1.AddIndex('idx_no_art','no_art',[]);  SQLQuery1.IndexName:='idx_no_art';  SQLQuery1.Open;  ...  SQLQuery1.Locate('no_art','200295',[]);

Thanks, that's really interresting, but it does not seam to be useful in my case. Basically my application is a cgi app which receives a request, does some modifications in the database and then sends back some other data based on the request and on the database and then it just quits.

I get the value of the primary key of the table from the request, so I though that because it is the primary key I would be able to quickly jump to it. But it seams that not? From what I understood the solutions are first going through the entire table and indexing it to make future lookups faster. But this does not seam to use the fact that my field is the primary key.

From my experience doing a loop

while not SQLQuery.EOF do
compare
SQLQuery.Next

is *really* slow.

But I doubt that first indexing the whole table and then looking up the value 1 time will be any faster.

I am thinking of moving to FastCGI, but I'm not sure if then I would be able to index the table and gain speed in future requests. maybe...

Felipe Monteiro de Carvalho

Reply by email

Martin Schreiber
to Felipe Monteiro de Carvalho
Felipe Monteiro de Carvalho wrote 7 lines
I get the value of the primary key of the table from the request, so I though that because it is the primary key I would be able to quickly jump to it. But it seams that not? From what I understood the solutions are first going through the entire table and indexing it to make future lookups faster. But this does not seam to use the fact that my field is the primary key.

Can't you do the lookup in the DB server by a SQL join for example?

Martin

Reply by email

Ludo Brands
to Martin Schreiber
Can't you do the lookup in the DB server by a SQL join for example?

Or, if you need do some processing in between, use 2 queries and build the "where" clause for the 2nd from the results of the 1st. If you don't need the full table, don't use select *.

Ludo

Reply by email

Michael Van Canneyt
to Felipe Monteiro de Carvalho: Thanks, that's really interresting, but it does not seam to be useful in my case.…
Felipe Monteiro de Carvalho wrote 28 lines
On Fri, Oct 21, 2011 at 5:22 PM, Ludo Brands wrote: > You can use indices and locate with TSQLQuery as follows: >  SQLQuery1.AddIndex('idx_no_art','no_art',[]); >  SQLQuery1.IndexName:='idx_no_art'; >  SQLQuery1.Open; >  ... >  SQLQuery1.Locate('no_art','200295',[]); Thanks, that's really interresting, but it does not seam to be useful in my case. Basically my application is a cgi app which receives a request, does some modifications in the database and then sends back some other data based on the request and on the database and then it just quits. I get the value of the primary key of the table from the request, so I though that because it is the primary key I would be able to quickly jump to it. But it seams that not? From what I understood the solutions are first going through the entire table and indexing it to make future lookups faster. But this does not seam to use the fact that my field is the primary key. > From my experience doing a loop while not SQLQuery.EOF do compare SQLQuery.Next is *really* slow.

Yes, but of course this is the complete wrong way. Of course this is slow.

You must do a

With TSQLQuery do
   begin
   SQL.Text:='SELECT * FROM MyTable where KeyField=:MyKey';
   ParamByName('MyKey').AsString:=Mykey; // MyKey you got through the request.
   Open;
   try
     if (EOF and BOF) then
       // no data, raise an exception
       Raise Exception.CreateFmt('Value "%s" is not an existing key value',[MyKey]);
     // Do some stuff with the record
   finally
     Close;
   end;
   end;

Then you will get only the record you need, and it will be lightning fast if you have used the primary key as the field to select on.

Michael.

Reply by email