I looked through the contributed units in the http://freepascal.org site and did not see anything that could take an array or some other data structure and write it to disk. Also the library would need to do the opposite, read from disk into a data structure.

I figure before I try and re-invent the wheel I would ask if there is any good libraries/unit(s) like that already. Perhaps something simmilar to what Python has on it's Picle library.

Is the Objects unit what I am looking for?
In this url
http://coleweb.dc.fi.udc.es/docencia/edi/freepascal/doc/user/node10.html

I foud this description.

objects
This unit provides the base object for standard Turbo Pascal objects. It also implements File and Memory stream objects, as well as sorted and non-sorted collections, and string streams.

> File and Memory stream objects

The application I am trying to do is a database.
I want to try some concepts for doing Analytics/OLAP work and performance will be a concern so I figure FPC would be a good tool for the job.

From reading this URL

http://www.learn-programming.za.net/programming_pascal_learn11.html

I see one can write records to a file. Would that be a feasible approach?

Not really.

Indeed. That's why it exists :-)

You can also try the Classes unit if you use object oriented programming. Each component can write itself to stream.

Michael.

Michael Van Canneyt wrote

You can also try the Classes unit if you use object oriented programming. Each component can write itself to stream.

Played a little bit with the Classes unit.
Don't really see any benefit over justin writing records directly. Am I missing something?

Also when reading back, on both cases(class or directly using records) it seems I need to use fixed length records. At least based on the examples I have seen so far. Is there a way to save variable length records and then read them back?

Normally you write the length as a prefix before the record, typically as some kind of integer type (e.g. LongWord or TsSize). When you read the record later you read the length prefix first, then read the number of bytes that the prefix specified. As a rough and ready example:

	in_file := FpOpen('input_data.dat', O_RdOnly);
	.
	{ Other stuff goes here. }
	.
	bytes_read := FpRead(in_file, record_len, sizeof(record_len));
	bytes_read := FpRead(in_file, buf_area, record_len);

Incidentally, from your first message in this thread, you said you were writing an OLAP application. You might care to look at PostgreSQL as a database manager. It does rather nice OLAP functionality, straight out of the box -- and it's free.

Regards,
Dave [RLU#314465]
====================================================== (David W Noon)
======================================================

David W Noon wrote

Incidentally, from your first message in this thread, you said you were writing an OLAP application. You might care to look at PostgreSQL as a database manager. It does rather nice OLAP functionality, straight out of the box -- and it's free.

Somewhat offtopic...
Im a full time Postgresql DBA. Have used postgresql for 5+ years and it doesn't do so well with the very uneven data distribution the dataset I have.

This is exactly why I am doing this work. I am primarily trying to experiment with some designs as proof of concepts. PostgreSQL is a general purpose DB wich does great most of the time, but I think it does not do well with very unevenly distributed data in the analytics environment at my job.

Have only been in this new job for about 4 months, but so far it seems by far this is the one dataset that has given me the biggest challenge to get it to work well with Postgresql.