Hello:

I'm trying to sort a TBufferDataset by several criteria, and I'm a little confused.

There is no sort method, so I decided to use indexes, but I have a lot of doubts.

When you close the TBufferDataset, all the data is cleaned.

TbufferDataset has a MaxIndexesCount that limits the number of indexes. By default this property is intialized to 2, and two indexes are created and used by default as soon as you call addIndex. So you must call addIndex before opening the table, or set MaxIndexesCount to at least 3 in order to be able to add an index with the table active. (you can't close the table to add new index or you'll lost data).

There is no way to drop a single index, and there is no way to change the index fields on the fly, you must call clearIndexes. And to call clearIndexes, before you must close the dataset, and so loose data. Am I right?

So, if I want to process the dataset in different orders with TbufferDataset, before opening the dataset I must set MaxIndexesCount to all the orders I'm going to use.

Am I right? Or I have missed something?

By the way, What does the index option "ixNonMaintained" mean?

Correct.

Correct.

Yes.

So, if I want to process the dataset in different orders with TbufferDataset, before opening the dataset I must set MaxIndexesCount to all the orders I'm going to use.

Yes.

Am I right? Or I have missed something?

You are right.

Probably we should at least now offer the possibility to delete indexes, and I am not sure the MaxIndexesCount is still needed.

By the way, What does the index option "ixNonMaintained" mean?

I don't think it is used in FPC, I think it's a Delphi compatibility option.

Michael.

At least it is used.

If you create the indexes before opening the dataset, MaxIndexesCount is automatically updated to hold your indexes plus the two default indexes.

But in an active dataset, you can't call addindex more than MaxIndexesCount, and being aware that two slots are used by two default indexes. You may get "The maximum amount of indexes is reached."

By the way. I got "The maximum amount of indexes is reached." and I went mad trying to find where I was creating two indexes. Was I calling addindex twices before and I wasn't aware?. Finally I looked at the code and when you call addindex, it creates two indexes: 'DEFAULT_ORDER' and ''. What are those two phantom indexes for? I couldn't find any documentation about

Default order is the index used for the records as they are read from the file or from the query.

The other one is for when you set the indexfields property directly, I think.

Michael.

Michael Van Canneyt wrote

The other one is for when you set the indexfields property directly, I think.

That solves the my sorting problem!!

No need to create indexes or dropping indexes or changing fields of an index. Just leave IndexName blank and play with IndexFieldNames (that in fact changes the fields of index ''). and if you set  IndexFieldNames to blank it restores the original table order.

That make of TbufDataset a useful component, not a nightmare. It needs desperately documentation.

How can I help with doc?

Op 13-04-2020 om 13:59 schreef Santiago A. via lazarus:

Roflol. I'm happy to hear that.

And, yes, it desperately needs documentation.

How can I help with doc?

There is the official-style documentation, you can retrieve it from svn here: https://svn.freepascal.org/svn/fpcdocs/trunk

You can send patches to the bug-tracker. Normally a class is only included into the official documentation, once it is fully documented. You can send patches to the bug-tracker. Questions to the fpc-devel mailinglist. And Lazarus has a plugin to edit these kind of documentation-files. (Look for fpdoc, I don't know the exact name of the Lazarus plugin)

But, if I where you, I would start in the Wiki. Because people need background-information regarding the TBufDataset.

I've written this before, but I think it is hard to find, so here again. Some background-information, which explains the problems you had.

You can freely modify, edit, improve it. And maybe start some documentation with it. Or maybe decide that it is too much of a background-story. That's also fine.

The indices of TBufDataset are not just lists with references to the data. They contain the data itself! There are multiple descendants of TBufIndex, and each of them has their own way of storing data.

This is because I never could decide upon how to store the data: in a linked list of records, or an array (one contagious block) of records. Or any variants thereof.

In the end, only the TDoubleLinkedBufIndex made it into production. (Nowadays there is a unidirectional variant, but this is irrelevant for now)

When a TBufDataset stores it data through TDoubleLinkedBufIndex, each record in memory starts with a header. This header has a link to the prior and next records, *for all different indexes*. So when there are two indices, the default one and one spare, for each and every record you know what the prior (and next) record is in both indexes. (When you look at the code: the header consists of a list of TBufRecLinkItem-records)

The spare (hidden) index is there to make it easy to order the dataset based on a list of fieldnames, as you already discovered.

It is impossible to add a new index once the data is stored in memory, because this would mean that the memory of all records has to be allocated with space for one more index. While this might sound doable, it is not in practice, because all kinds of mechanisms demand that the dataset-data in memory stays at the same position (pointer).

So for each index, memory has to be allocated with the size two pointers for every record for each index.

This is why you have to provide the amount of indexes you want to use, before the data gets read. (+2, one for the main index, and one hidden)

The hidden index is rebuild each time the IndexFieldNames is changed.

Note that the double-linked buffers have advantages, but also some disadvantages. Adding and even inserting a record is easy. Finding where to insert a record in a sorted index, is not. You always have to loop through the records to find one particular record. This also holds for the lookup-functions of the TBufDataset.

And there is the issue of the RecNo property. If you want to know what the current record-number is, TBufDataset has to start with the first record in the index, and loop through all records until it reaches the current record. Maybe that the current record-number is cached nowadays. I do know that the record-count is cached. But what is important to remember: do *not* use the recordcount propery of a TBufDataset. It's Evil. Do not make your logic depend on it.

So, that's it for now. I hope some of you found it interesting.

(Oh, and note that TDataset itself also buffers the data, so a TBufDataset has *two* buffers, holding the same data. The TDataset buffer is fairly small, though. The default is 10 records, iirc. But when you attach a TDataSource to a TDataset, the TDataSource tells the TDataset how many records it when on of the
TDataSources need 200 records, TDataset will cache 200 records. This could be used by a grid-view, for example, wich shows 200 records at a time. Also note that in a uni-directional dataset, this buffer build in TDataset itself is not uni-directional. That's why you can see data in a data-grid bound to a unidirectional TDataset.)

Regards,

Joost.