-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I suspect I already know the answer to this, but is anyone aware of an *embedded* database for Lazarus and Linux, similar to NexusDB for Delphi and Windows for example (http://www.nexusdb.com)? The need is to be able to distribute a database application as a single program, no installation of MySQL or similar can be assumed.

Thanks,

Brian.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkwYuxMACgkQDLwN6j3kblbgfQCfeEzsFQ3R0s1EgAkS+MyoRuZ1 GQ4AoMt3BN8QvVPh2wS4ZO0HkfCJT4ph
=THc6
-----END PGP SIGNATURE-----

Something like Firebird Embedded?

http://wiki.lazarus.freepascal.org/fb_embedded2_Laz

Later on you can always switch to the standalone FB server.

Kind regards,
Birger Jansen

-----Oorspronkelijk bericht-----
Van: brian [mailto:]
Verzonden: woensdag 16 juni 2010 13:53
Aan: Lazarus mailing list
Onderwerp: [Lazarus] Embedded database for Lazarus/Linux

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I suspect I already know the answer to this, but is anyone aware of an *embedded* database for Lazarus and Linux, similar to NexusDB for Delphi and Windows for example (http://www.nexusdb.com)? The need is to be able to distribute a database application as a single program, no installation of MySQL or similar can be assumed.

Thanks,

Brian.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkwYuxMACgkQDLwN6j3kblbgfQCfeEzsFQ3R0s1EgAkS+MyoRuZ1 GQ4AoMt3BN8QvVPh2wS4ZO0HkfCJT4ph
=THc6
-----END PGP SIGNATURE-----

Try sqlite. It's completely free and only requires one dll (afaik).

Henry

Birger Jansen wrote

Something like Firebird Embedded? http://wiki.lazarus.freepascal.org/fb_embedded2_Laz

It seems not. Quoting from the first line of the introduction....

"Currently, fb_embedded2.pas works only on Windows."

As per my post, I already know the answer if I have to use Windows. I'm looking for something which works under ***Linux***.

Brian.

Well, sqlite does at least have a Linux distribution! Thanks - I'll do a bit of digging.

Brian.

Just to be clear, Firebird works just fine on Linux. It is only this Pascal unit that wraps the library that is restricted to Windows.

i suspect that something like sqlite is what you seek... i see it used in a lot of stuff, these days... apps like thunderbird and firefox are the first that come to mind ;)

I don't know where you read it, but it's definitely false. or outdated. firebird embedded works on linux, windows, mac.

I've been using it since years; version 2.0 and 2.1. Works very well with FPC, and unlike sqlite, it offers the possibility to work with full-blown client/server architecture without change in program code.

I can definitely recommend it.

Michael.

brian wrote

As per my post, I already know the answer if I have to use Windows. I'm looking for something which works under ***Linux***.

I don't know where you get your information, but embedded firebird works perfectly on Linux and Windows. We (our company) have used it for a few years now with FPC.

I big advantage of Embedded Firebird over Sqlite is that with Embedded Firebird, you an upscale to a full RDBMS without code change. A very positive feature indeed.

En/na Michael Van Canneyt ha escrit:

I don't know if it's outdated but overall I think this is a fair comparison:

http://www.sqlite.org/cvstrac/wiki?p=SqliteVersusFirebird

In my case the deciding factor was the last one:

"Concurrency

SQLite allows multiple programs to be connected to the same database simultaneously. The embedded version of Firebird does not. If you run Firebird in client/server mode, it allows concurrent access with fine-grain locking. But in embedded mode, only one program can connect to the database at a time."

Bye

Luca Olivetti wrote

SQLite allows multiple programs to be connected to the same database simultaneously. The embedded version of Firebird does not. If you run

Well, that sounds like a rather silly argument. I thought the whole idea of "embedded databases" is to get one application to access one database at one time - more convenience too than a flat-file or XML file for data storage. Embedded Databases being similar to the idea of MS-Access (*.mdb files) - a "desktop database". If you want a multi-user concurrency system, then you upscale to a full fledged RDBMS system, or the "big daddy" servers like Firebird, Oracle, PostgreSQL, MS-SQL-Server etc.

Say I've got a couple of desktop programs that want to share data/settings, why on earth would I want to install a fully fledged database server? Even jet databases (mdb) support locking.

Henry

Henry Vermaak wrote

Say I've got a couple of desktop programs that want to share data/settings, why on earth would I want to install a fully fledged database server?

Because it was designed for concurrency and multi-user support. BTW: The full fledged Firebird RDBMS is a whopping 12MB install - smaller than most desktop applications. So I would hardly say it's an effort to install - and Firebird is known for its "deploy and forget" installation style. No need for tweaking or other setups (unlike MS-SQL-Server, Oracle, DB2)

Even jet databases (mdb) support locking.

Yeah, and very crude at that - using a little external file (*.ldb) with records. I can't even quote the amount of issues we have with that in our applications. Windows freeze or application crash and the *.ldb is left there blocking the program and more often than not the mdb file ends up being corrupt too. Microsoft documentation clearly states it's for single user access only - for good reason.

I'm pretty sure even Embedded Firebird database will work better in a multi-application concurrency system than MS-Access. Anyway, the point is, that the full fledged Firebird is easier, quicker and less effort to install that most applications. So if you want concurrency, use the right tool for the job. For 99% of desktop applications that simply want to store data in a single user database instead of flat-file or XML, the Embedded Firebird will do just fine.

sqlite was designed for concurrency and still it's embedded.

It's not about disk space. It's about having a database server running all the time. Imagine everyone had your attitude, I'd have several useless database servers running on my computer. No thanks.

Yes, but read this forum-thread. (This applies to the used dataset offcourse)

http://www.lazarus.freepascal.org/index.php/topic,9504.0.html

Joost.

So firebird _does_ support accessing the same database from different host programs in version 2.5. This version has a release candidate, so presumably a release won't be long. Good to know.

Henry

Follow the link which Birger sent me (above). The introduction is right near the top of the Wiki article.

But anyway, if you say it works, OK, I'll give FB embedded a try.

Brian.

El 17/06/2010 9:39, Luca Olivetti escribi?:

If you write a program manages its own data, configurations etc, you can read sequential files, text files etc. But if the data is very complex and there is a lot of data, instead of reinventing the wheel, you can use a embedded DB, like SQLlite or others. That is what embedded RB were designed for, not to share data.

SQLite's concurrency is just fancy feature, not a very useful feature. Concurrency in SQLite is pretty simple. Only a process can write simultaneously, and it locks the whole file and has problems with threads: Check http://www.sqlite.org/faq.html (5) (6). If you want a program (a workstation) that manages data, and the rest of programs read data, SQLite concurrency may be good for you. But even in this case, one process is updating and several processes are reading, you can run into problems: http://old.nabble.com/Memory-Usage-td23154686.html.

But if you plan to deploy several programs sharing the same data as peers, updating and reading, SQLite is not the solution, you must use a real RDBMS. You shouldn't rely on operating system file locking systems and clients doing the right thing. You need a single process that controls the file and schedules users' petitions, that is, a RDBMS.

But if you think that data will be seldom updated, SQLite may do job. Just be aware that SQLite is not scalable for concurrency. Use the right tool, for the right task.

Santiago A.

I had a look.

I would not use the unit in that article.
The sqldb (and dynibase60 unit) supports embedded firebird out of the box.

The only difference is that you must distribute a single dll which must be located next to your binary. Since you must distribute some extra firebird files anyway (the messages, collations and whatnot), this won't hurt.

Michael.

i'm (still) trying to figure out why one would have an embedded database with multiple embedded apps attempting to talk to it... concurrently or otherwise... i would think that any kind of concurrent multiple access needs would be better served by a (dedicated?) database server...

i still find it amazing, though, that in today's world, so many opt to use a sql database of some kind even for smallish projects where a nice simple self-designed binary data file would suffice... in many cases, it is akin to swatting a fly with a hydrogen bomb ;)

i tend to agree with you and in your case, i would be wondering why each has its own database server or why they even need one if it is as useless as you say... why can't they query to see if there is a database server already installed and simply create their tables as needed in the existing database??

kinda makes total sense to me but it must be too logical otherwise it would already have been being done...

I still find it amazing, in today's world, that so many opt to re-invent the wheel. Why would you design your own buggy binary data file that only your program can read, when you can use a safe, tested, fast embedded database? I hardly call a 500k shared library a hydrogen bomb.

Because personal computers don't normally run database servers? I don't have any software on my pc that requires a database server to be installed. Do you?

i'm not speaking of wheels or reinventing... but since you bring it up, i'm sure i'm not the only one who would rather reinvent and know what their code is doing rather than use some library code (MFC for instance) that's buggy as hades and has all kinds of sneak pathways in it... sure, the hackers and spammers love code like that but...

and what can you do to fix library code? without the sources and a deep understanding of what it does and how it does it, well... i think you know where that's drifting off to ;)

Why would you design your own buggy binary data file that only your program can read,

what makes a binary (or flat or XML) data file buggy?? that doesn't make sense... did i step on a nerve or something??

is there something wrong with me supplying the format of the data file to those who want/need to read it? i don't think so... in fact, i recall a time when developer's kits were pretty much exactly that... header files, record structures, flow charts, possibly even some documentation... in fact, i still use developer's kits for numerous apps that i work with...

and why would someone want to use some sql data base format where they are required to use a special interface (sql) and method of access (database server) to manipulate their data when they can do it faster and easier in their own storage formats??

besides, XML is a good example as well since everyone and their mother seem to see it as a huge step forward for some reason... i see it as being more complicated than other available options but then again, i see how so many things are really nothing new and how, deep in their heart, they are still the same old methods and procedures that have been used since these machines first came into being...

when you can use a safe, tested, fast embedded database? I hardly call a 500k shared library a hydrogen bomb.

then you missed my point... that point being that a database is not necessary in many cases where they are used... speaking of embedded, what does a refrigerator, microwave oven, convection oven, coffee maker or toaster oven need with a database? for that matter, what does a security camera, laser measure tool, or television need with a database? i mean, seriously...

but they (storage formats) are "neat" and "cool" and one can say "my apps use this!" or "my apps use that!" so i guess it helps to raise one's perspective of one's or one's app's abilities... i see the same thing coming from both sides of the fence, so no one is in the clear, really ;)

then again, one can also shave another 500k off of their application's footprint if they don't have to use that 500k shared library code that could be buggy and full of sneak paths thru it that the hackers just love to find ;)

what's the real difference between the two? nothing other than the software load, really...

I don't have any software on my pc that requires a database server to be installed. Do you?

i'm sure there is software installed on my workstations that require the use of a database server, yes... am i aware of what those packages are? likely not since most people don't go digging behind the scenes to see what is really going on behind the pretty facade...

i find it rather interesting that you cannot simply copy the bookmarks.html file from firefox any more to back up or share your bookmarks... why? because they're stored in a sqlite database, now... before, bookmarks.html was just that... your bookmarks stored in XML format in a html /text/ file that was easily editable and sharable with any editor... hell, even edlin can handle it :lol: so... why the change to a database? to make it harder to get at the data? to make it more proprietary?

do i have database servers at my location? you betcha i do... several apps i use and services i offer use the database servers for their needs... one of those apps is a forum package... another is a recipe database... then there's a photo gallery and even several personal task management type apps...

Henry Vermaak wrote

re-invent the wheel. Why would you design your own buggy binary data file that only your program can read,

Last time I checked, writing a record to a binary file was pretty damn easy. I think I learned that in my first year of Turbo Pascal programming back in high school (many, many years ago). I can't see why that would be buggy either.

waldo kitty wrote

besides, XML is a good example as well since everyone and their mother seem to see it as a huge step forward for some reason...

I can't agree more on this point. Yes, XML has it's uses, but replacing a INI file with a XML file simply to store some application settings like "recently opened files", or editor color settings etc is pretty ridiculous. And yes, Lazarus IDE is a culprit in this too. I mean, INI is a dead simple file format which FPC and Delphi (and just about any other language) has very good support for reading/writing INI files. So why use XML which is much more complex and much harder to work with. Again Lazarus IDE being an example here. There are multiple XML reading/writing solutions (inside FCL and LCL and IDE), all with varying degrees of features and how to use them.

You lost me on the kitchen appliances though. ;-)

There is a very simple rule of thumb I use.

If it is 'flat' info, I use .ini files. They are easy to use, human editable, and inifiles is a small unit.

If the info is somehow hierarchical (multi-level), I use XML.

Michael.

I agree completely that storing an application's settings in human readable format is the best. It's obviously limited in terms of speed, though. Using a database for this is overkill.

Henry

I've seen horrendous bugs, overflow, corruption. What happens if your structures change? Can you do relations? Let's not compare apples with pears.

Fair enough, and that makes sense. Looking at all the xml files that Lazarus IDE generates inside ~/.lazarus/ directory, I found 15 of them. Only one (packagefiles.xml) I could see that really benefits being in XML, all the others would have been much simpler in INI files.

And looking at it again, even the packagefile.xml could easily be presented as an INI file where data is store using CSV format for the data section of an element - each xml element in the xml file is exactly the same structure over and over anyway. It's easier to read INI files and maybe having to parse a CSV data section than work with XML.

But hey, we are way off topic again. ;-)

On Thu, 17 Jun 2010 12:35:14 -0400
waldo kitty wrote:

Don't you think you are overreacting?
Sure, not everything that needs to store data needs a db but sometimes it is just convenient to use one.

and what can you do to fix library code?

Well, for sqlite for example download the code and fix it. If you use open source libs there should be no problem. Bug reports may help, too. Of course it is easier to fix your own code, but I am not able to code everything I need myself and therefor have to use libs.

and why would someone want to use some sql data base format where they are required to use a special interface (sql) and method of access (database server) to manipulate their data when they can do it faster and easier in their own storage formats??

Sometimes it is just easier to use a db (standard format) instead of coming up with and mantaining a GOOD format on your own.

besides, XML is a good example as well since everyone and their mother seem to see it as a huge step forward for some reason... i see it as being more complicated than other available options

XML is as complicated as you make it. All languages I know provide good support for working with XML/DOM and it is a human readable format, which makes it very convenient to use.

speaking of embedded, what does a refrigerator, microwave oven, convection oven, coffee maker or toaster oven need with a database? for that matter, what does a security camera, laser measure tool, or television need with a database? i mean, seriously...

Store data, what else. A laser measure tool may group measurements by object or place or something else and store location, time, measured data, error marging, etc. in the db. It may even be easier to bring the data from one db (the tool internal one) to another (which is used in the firm using the tool), than from a flat data file where you may be forced to write an exporter/importer to get the data in your db. Does it absolutely need a db?
No, but maybe the lifes of the developers and users get easier by using one.

then again, one can also shave another 500k off of their application's footprint if they don't have to use that 500k shared library code that could be buggy and full of sneak paths thru it that the hackers just love to find ;)

Your own code could be buggy as well or are you the only developer coding without bugs? You sound like a db is the root of all evil, full of buggy code and every cracker in the world just wants to break your embedded db. I highly doubt that this is the case. And btw the nineties called and want their 500K back. Remember, we are talking about desktop apps on current generation hardware. I don't even care if an app on my mobile uses 500K more or less, even less so on my desktop machine. If you really have problems with that 500K, I take it for granted that you don't run any Java programs on your computer. Perhaps you should switch from Pascal to Assembler to make sure that you have absolute control over every register and waste no single bit or CPU cycle.

NOM, just my 2 cents and now I am getting off your lawn. ;) R.

Not to mention speed. I've written code for hash table indexed files, and it isn't pretty.

Henry

Also see http://www.sqlite.org/testing.html for how serious they are about testing the code.

I would imagine that the project would have to be pretty smallish to not meet the justification to use something FB emedded given all the extra features you get since even smallish apps I've personally written sooner or later end up having more complex data persistence requirements to one degree or another.

The need for just a couple of INI file settings aside of course.

Nice to be able to set a couple of configuration settings and have a full blown FB server should the need arise. If it doesn't, the only downsize is that I'm stuck with a very versatile storage mechanism.

Sounds like a win/win to me :-)

damn! you had pascal as a course in high school? ya lucky barstard :P :)

Graeme Geldenhuys wrote

You lost me on the kitchen appliances though. ;-)

as examples of embedded systems... some of which i know do have databases for various reasons... in numerous cases, there ends up being repetition in filling each database (consider recipes on the fridge, stove and microwave) which would be much simpler with a central database server that all of the devices talk to...

yes, there are some extremely poor implementations out there...

What happens if your structures change?

since you've both the old and the new, you simply process the existing data file with the old structures, convert it to vars in the new structures and then save it to a new data file in the new format... conversion programs like this are pretty bog simple but they can be tedious depending on the number of vars you have to work with... one of the sets i work with has over 200 settings and options ;)

Can you do relations?

sure! you just do the same thing as in a database... you have your key here and it is linked in each related record over there... nothing more than a while loop walking thru the other data file(s) looking for a particular link key in each record... i was doing that before i had to become a dBase II/IV coder ;)

Let's not compare apples with pears.

exactly :)

Reimar Grabowski wrote

NOM, just my 2 cents and now I am getting off your lawn. ;)

we thank you for your support! :)

seriously... it brings good discourse to the table :)

Exactly, what a joke. If you used a database you wouldn't have to do anything like this, just update the table definition. No conversion apps (what will a user make of this), no extra code in your app for this.

Haha, you are looping through every record of all your files? This is getting hilarious.

You seem to be slightly confusing app settings and app data. Storing app settings (like options) in a database is definitely overkill (imho), but storing app data in an embedded database is a very good idea.

how? it is the same thing that a database does when you update its structures...

If you used a database you wouldn't have to do anything like this, just update the table definition. No conversion apps (what will a user make of this), no extra code in your app for this.

what will the user think? who says they even have to know about it? i dare say that it is easily done during the start-up of the application ;)

you seem to forget that this is the same basic thing that a database server does... there's no way around it... someone/thing has to iterate thru all the records to locate the matches... sure, maybe this is done via an index file... same difference since both sql and pascal can use indexes ;)

ok, ok, ok... so i understand using databases... i've been using them for years... since before sql was ever thought of, actually ;) it is much like using a FOSSIL driver to support comms on multiple types of systems and OS'... just like a FOSSIL allows the coder to concentrate on the mechanics of the app they are working on instead of having to also delve into the comms stuff, using a database (and server of some sort) enables a coder to concentrate on the app and not worry about how to handle the storage and retrieval of data... simply make the call with the info stuffed into the call and go on about the rest of the app...

You seem to be slightly confusing app settings and app data. Storing app settings (like options) in a database is definitely overkill (imho), but storing app data in an embedded database is a very good idea.

i'm not confusing anything... i've been using databases since before i found dBase II ;) someone else made the comment about storing app settings in a database, FWIW... deity knows that i've seen such many times over the years... but hey! it works... they use a table (they were called 'databases' back then and you had several of them) called settings or some such and there ya go... new versions may carry new options in that table... as you say above, change the structures and off ya go... employees, inventory, shipping, receiving and transactions are distributed throughout other tables ;)

Henry Vermaak wrote

Not to mention speed. ?I've written code for hash table indexed files, and it isn't pretty.

And amazingly everybody managed just for for 15+ years (in the OS/2, DOS, Win3.11 era) before RDBMS became popular on desktop systems. If you can't program something as simply as a binary storage system for some application data, then maybe you (any developer - not specifically targeted at you) should have a change in career.

waldo kitty wrote

damn! you had pascal as a course in high school? ya lucky barstard :P :)

It was as an extra subject, and was actually after school where I had to attend the classes in another school, because my own school didn't offer it. A lame setup, but programming was fun, so I made due with the crazy arrangement. :-)

En/na Graeme Geldenhuys ha escrit:

I suppose it depends on the use-case, doesn't it?
In *my* use-case that was a determining factor.

I thought the whole idea of "embedded databases" is to get one application to access one database at one time - more convenience too than a flat-file or XML file for data storage.

Again, in my use-case, there's a main application, running 24/7 unattended, that accesses the database, then there are some ancillary applications (running on the same machine, as the same user) that, from time to time, needs to access the same database (for parametrization, reporting, etc.).
And since it's an embedded application (even if it's running on an pc) I didn't want the hassle to set-up a database server.

My system isn't multi-user, high-concurrency and sqlite fits the bill. Firebird embedded didn't.

Bye

i'm (still) trying to figure out why one would have an embedded database with multiple embedded apps attempting to talk to it... concurrently or otherwise... i would think that any kind of concurrent multiple access needs would be better served by a (dedicated?) database server...

ACK. It's easy to have multiple apps accessing the same data base (of whatever kind), e.g. multiple Lazarus IDE instances...

i still find it amazing, though, that in today's world, so many opt to use a sql database of some kind even for smallish projects where a nice simple self-designed binary data file would suffice... in many cases, it is akin to swatting a fly with a hydrogen bomb ;)

What are the alternatives? If you already had to update a binary file format, for required extensions, you'll know about the problems with the conversion from the old to the new format, and with the synchronization of the load and store procedures. A database should do all that automatically, whenever the common model is updated.

XML files are no real alternative, they typically lack a "TOC" with all available or mandatory entries, that have to be specified and respected in application code. Of course there exist *means* to describe the XML content structure, but that feature is based on very complex descriptions.

The Windows Registry is another (poor) attempt to provide structured data storage, with only structured acces methods, but no description of the structure itself.

That's why "traditional" (relational) databases, with a number of named tables and entries, IMO offer both a simple structure definition, while leaving much room for common or special (SQL) statements for data retrieval and storage.

In so far a XML solution will more resemble the hydrogen bomb, compared to a simple DB solution. A database system is not necessarily a big piece of code, as shown by the embedded solutions, and the SQL query features can be applied as another layer on top of any concrete DB model.

DoDi

En/na waldo kitty ha escrit:

My "embedded" equipment is an automated store servicing a production line, so it has to keep track of what's inside it and to what's coming down the line to service it.
Granted, it doesn't need that many records (1000 more or less[*]), but why do I have to reinvent a database when sqlite already offers it and it's fully ACID.
Besides, call me lazy, but the databases components are extremely convenient and saves you from writing a lot of code when you want to present and edit your data in a tabular form.
In fact, at times I used an sqlite database just as a configuration file for the sheer convenience of using the database components.

[*]I won't make names, but the company where I deploy it would use an oracle database for that.....

Bye

En/na Santiago A. ha escrit:

SQLite's concurrency is just fancy feature, not a very useful feature.

It has been really useful to me.

Concurrency in SQLite is pretty simple. Only a process can write simultaneously, and it locks the whole file and has problems with threads: Check http://www.sqlite.org/faq.html (5) (6).

Checked that already at the time and took care of the problems in my application, that has been working fine, 24/7, for the last 2 years.

Again, in *my* use-case that was a determining feature.

Bye

En/na Joost van der Sluis ha escrit:

Yes, but read this forum-thread. (This applies to the used dataset offcourse) http://www.lazarus.freepascal.org/index.php/topic,9504.0.html

Exactly.
I have no such problems using zeos (for the gui applications) with a suitable busy handler, plus raw sql queries for the main application.

Bye

waldo kitty wrote

> Exactly, what a joke. how? it is the same thing that a database does when you update its structures...

No, a database file format doesn't change the whole time. You would have to adapt your file reading and writing code every time you change the format. When you use an embedded database, you won't.

what will the user think? who says they even have to know about it? i dare say that it is easily done during the start-up of the application ;)

You still have to write code for this, where when you're using a database, you have to write none.

Nonsense. Databases use very sophisticated algorithms. It will take you years to try and implement that yourself. I have databases with millions of records that I need to query _very_ quickly. My own b-tree algorithm didn't cope too well, but why would I re-invent the wheel.

ok, ok, ok... so i understand using databases... i've been using them for

You obviously don't if you think you can implement a similar system on a rainy friday afternoon after you've been to the pub.

Anyway, I'm through with this discussion, signal to noise is just too poor.

Not every program needs this. It's a specific use case. Even word processing formats are now based on xml that's compressed. If you don't need the speed, stick with human readable formats.

you can't program something as simply as a binary storage system for some application data, then maybe you (any developer - not specifically targeted at you) should have a change in career.

_Everyone_ can roll their own, and we all have. But it's not going to be fast, reliable and robust. Some people will just never learn, maybe _they_ need a change of career.

The first rdbms was released in 1968. Even dbase is now 30 years old.

Please re-read my original message (quoted for your convenience). I know when RDBMS were developed, but I clearly stated "became popular on desktop systems" meaning: used by desktop applications. Not everybody has a Mainframe at home - now or back in the 80's. Mainframes being the original systems that ran RDBMS.

Desktop systems ran perfectly with binary files as storage for many years. Pascal was at its high point and very popular for reading such files. I remember back in the days of Fidonet, when message forums were flooded with *.pas files containing structure information text file documentation for reading various binary storage files. Binary files were very easy to use, and upgraded to newer formats just as easily as your current application having to update the table structure in a database (normally done at application startup or via an upgrade tool).

But please note, my argument here is not against using a database for data storage (I actually don't know how this thread got twisted to that topic). Embedded systems have there place, but as soon as you have multiple clients or users accessing a database file, I think it more appropriate for a full RDBMS installation. Firebird being one of the top choices simply because it is the easiest full RDBMS to deploy
- a mere 10MB install and no admin knowledge or maintenance required afterwards - unlike MySQL, MS-SQL-Server, DB2, Oracle, PostgreSQL etc.

As for the debate about which format to use for simple application settings. INI being the obvious choice and de facto standard for years on multiple systems. Why anybody would want to use XML or a SQL database table for that just boggles the mind. And I fully agree with the other poster about Firefox and its Bookmarks change. I also wanted to restore my bookmarks.html file the other day on a newly installed system, but now Firefox uses SQLite for such simple storage, so the solution wasn't immediately obvious. It was dead simple a few versions back - simply copy the old bookmarks.html file over the new one. Now Firefox fell for the classic over-engineering mistake, and they had to extend their UI to include a Import function so you can import from a backup file, or from a selected HTML file or from a SQLite database - a lot more complex than it used to be.

Anyway, I replied to the original poster of this thread relating to an embedded database option for Lazarus, so I consider this thread closed. The original poster can now experiment with Firebird or SQLite and choose one based on his current and future needs and requirements.

i don't think they'd even heard of teaching anything computer related in high school when i attended at the end of the '70's beginning of the '80's... by then i had already taught myself BASIC from the TRS-80 manuals and was moving into ASM and pascal... some ASM stuff coming from IBM Technical Reference Manuals, IIRC... i've also got pages and pages from a compaq (i think) manual that covers ASM as well...

what a difference 30 years makes :)

ummm... where did i ever make such a comment or statement? you are reading something that i never implied or intended :? :(

Forgive me my naivity (I never used a database yet) but just as an example:
If you want to read a complete directory branch
(with thousands of files and directories) every 5
minutes in a loop and want to store this information together with other information (like date/time
of first appearance) at the end of each loop so
that it's available even after a reboot, would
you use a database for this?
If yes, how would the tree structure be defined
in the database?

I just did this using a simple binary file and
it was quite easy to do and it's speed is amazing. If my data changes, I simply delete the file
and start from cratch.
I also wrote programs that read multiple versions
of such binary files and I found it quite easy to maintain. When using a database instead of a binary file I would need to feed the data from it into the data structures of my program in the same way anyway.

If I see the delays that applictions like Thunderbird produce when simply moving an email to another folder (the hard disk spins for a minute or so) then
I am happy that I never was forced to use a database. Nearly all (PC) programs that I know and which use a database for relatively simple data are awfully slow with anoying delays in unexpected situations.

Of course, for huge data amounts of data you cannot avoid using a database but I would prefer if programmers think about performance a bit and not use databases for every small junk.

Forgive me my naivity (I never used a database yet)

I'm also not a DB power user, but...

Depends ;-)

In your example...

I just did this using a simple binary file and it was quite easy to do and it's speed is amazing. If my data changes, I simply delete the file and start from cratch.

... I'd not bother with storing the become
extremely important when you can *not* simply rebuild your data from scratch.

If I see the delays that applictions like Thunderbird produce when simply moving an email to another folder (the hard disk spins for a minute or so) then I am happy that I never was forced to use a database.

That's because TB does *not* use a database, at least not standard DB that is designed for fast transaction processing. Instead TB holds all information in one file per folder, so that moving a piece from one folder to another one means to rewrite two such (huge) datafiles.

You see how important the right use of the right storage model is in practice?

DoDi

J?rgen Hestermann wrote

but I would prefer if programmers think about performance a bit and not use databases for every small junk.

+1

Hardware speed increases + Lazy programmer faster programs

yup! even if the database is a binary design of one's own work ;)

this is true and i don't dispute it... one of the points is that using a database server and such removes that level of detail from you, the coder, and you no longer have to worry about it... on the one hand i can understand and agree with this... on the other hand, though, i see where new folk coming up are not getting the full experience and teachings that they should have... especially if binary data files is only one day in their programming course and they move to using a database and spend much more time on it... i feel that this gives a false sense and idea that using one's own binary files and working out one's own access routines is a BadThing<tm> when it is not...

XML files are no real alternative, they typically lack a "TOC" with all available or mandatory entries, that have to be specified and respected in application code. Of course there exist *means* to describe the XML content structure, but that feature is based on very complex descriptions.

yeah... i've looked at XML but haven't truly seen a need for it... i understand what folk are explaining about it but i've just not seen a real need for it...

The Windows Registry is another (poor) attempt to provide structured data storage, with only structured acces methods, but no description of the structure itself.

you found that, too, eh? :)

That's why "traditional" (relational) databases, with a number of named tables and entries, IMO offer both a simple structure definition, while leaving much room for common or special (SQL) statements for data retrieval and storage.

as in the case of XML, it has taken me some time to get wrapped around SQL... i can understand why it was created but its goal wasn't fully achieved and just like before SQL came along, there are still several differences between them... so nothing has truly changed...

In so far a XML solution will more resemble the hydrogen bomb, compared to a simple DB solution. A database system is not necessarily a big piece of code, as shown by the embedded solutions, and the SQL query features can be applied as another layer on top of any concrete DB model.

this is true... it just seems that many things are over-engineered for various reasons... one being as i depict above which can easily be expanded to cover more than just the binary data file example given... there are other reasons, too but these are what i have seen and been able to actually "touch" as to being /the/ "problem" (at it were)...

J?rgen Hestermann wrote

but I would prefer if programmers think about performance a bit and not use databases for every small junk.

+100000000000(can't put enough zeros)00000000000000

imagine how fast our apps would be today if they were developed with the same attention to compactness and speed as they were 10 years ago... they'd be even faster if the attention of 20 years ago was used...

sadly, though, i see many new coders refusing to look at the older stuff because "it is older than i am" or "older then 5 years" or similar... but the truly funny thing is that, in the end, their result is the same as that which they refused to look at... in some cases, with the same bugs...

one extremely hilarious aspect of working with newer coders is the expressions on their faces when you show them 20 year old code running rings around their code with the same test data and in the same environment... the best part is when they realize that the 20 year old code is where they were actually headed with their own code but their refusal to look to the past for education hampered them...

another funny is seeing the looks on people faces when they realize that no matter how new the "technology" is, it is still the same old thing... HTML and XML are still plain old ascii text files ;)

How did you come to the conclusion that databases == slow? In my case, the database _improved_ the speed of my program due to the clever indexing. Never mind the advantages of crash-recovery, extensibility, etc.

Henry Vermaak wrote

How did you come to the conclusion that databases == slow?

I guess with databases it depends on how well managed the database is, how well the database tables are designed, and how skilled the developer is at writing SQL. I remember when I started at our company (which I worked at some years earlier too), the first thing I did was look where the program was slow. One report took 2-3 minutes to run. I looked at the code and simply said: WTF! 10 minutes of refactoring the code (undoing their mess) and rewrote the SQL statements... I got that initial 2-3 minute report down to 4 seconds.

So in summary, using a database doesn't instantly mean faster! Binary files or Databases - you still need a skilled developer to make it work.

The latest Mozilla Thunderbird uses SQLite databases for just about everything and constantly I see the statusbar saying "re-indexing XXX". As mentioned earlier, even for managing something as trivial as bookmarks. Luckily my Quad Core just powers through all that re-indexing that now occurs, so I don't notice it that much.

Your last statement is 100% correct though. Unfortunately not everybody seems capable of making the right choice. :)

The re-indexing is for searching the messages. I don't know how this is implemented, but I know that there were some teething problems with this, which are fixed in the upcoming version. Only the initial indexing should take long, but it's essential for us who has come to rely on google mail's searching capabilities.

Henry

indexes are another huge factor for having efficient databases... i can't count the number of times that i've created an index on a set of tables and the efficiency has gone up several magnitudes compared to what it used to be...

kinda makes ya go "hummm..." :)

So in summary, using a database doesn't instantly mean faster! Binary files or Databases - you still need a skilled developer to make it work.

absolutely!

funny that you mention that because i was just thinking about how it was possible that so many may not be using local apps for their mail... in my case, i have several dozen email addresses and all of them are serviced by my local thunderbird... yes, my gmail, my yahoo, my excite, my hotmail, etc, etc, etc are all pulled (via pop3) from those servers into my thunderbird... i don't worry about those services loosing my mail any more... if it does get lost, it is my fault ;)

FWIW: email is another of those things that is supposedly "new technology" that isn't... it is still all plain text but now it carries "tags" in similar manner to html and such... even binary files are (still!) encoded into plain ascii text for traveling thru the mail networks... that's quite funny, in itself because fidonet, what a huge majority of people used before the internet came along, has allowed binary attachments to messages "forever"... yes, i do know that UUCP did the same thing, as well... i used to carry UUCP mail here for my BBS users... these days, that's all done with SMTP with the associated loss of technological capabilities (attaching a pure raw binary to a message) ;)

On Fri, 18 Jun 2010 18:44:21 -0400, waldo kitty wrote about Re: [Lazarus] Embedded database for Lazarus/Linux:

Reminds me of a database I was designing about 10 years ago on an IBM mainframe. One query was taking 18-20 minutes to run. I added another index to one of the tables -- it had 3 or 4 already -- and the query dropped to 12 milliseconds. That is some speed up!

Hi!

I don't know what others do, but I simply copy/backup my complete profile directory of Firefox (that with the alpha-numeric name). That works perfectly well and I also have my complete history, open tabs and Add Ons with me on the new system. :D

I'm aware that this might pose a problem if my profile becomes corrupted, but should this happen I'll find a way to solve this. ;)

Regards,
Sven

Hi!

What do you mean with "description of the structure"?

Nevertheless the Registry is not that bad if you think about it: it's an data storage system that is provided by the operating system itself and thus available to all applications running on it.
Also - as the system itself uses it, too - it provides many information about the system and allows for finegrained configuration of your OS. Another nice thing are permissions: You can - at least on NT+ systems - set the permissions for single keys (not values though) for specific users and groups. Try that with INI files (where the Registry originated from) or the simpler SQL databases. (I must admit though that I yet have to use that feature ^^)
If your application has the correct priviledges you can also load and unload private Hive-files (the Registry is saved in such files). You can use that for example to make your application portable: when your application is started it loads a private Hive into the Registry, uses the Registry methods to modify/read the data and then unloads it again when the application exits. (another feature I haven't tried/tested yet :( ) This might also work with applications from other developers... you just need to provide the Hive before the application is started, the application writes to the loaded Hive and after it terminates you unload the Hive again... Tada! Portable Registry-using application and less mess in the Registry :D (I should follow this idea sometime...)

To sum this up: The Windows Registry (and the NT architecture/design itself - excluding the Win32 subsystem :P ) is not as bad as most people see it.

Regards,
Sven

2010/6/19 Sven Barth :

[OT]
I simply need a password and a secret phrase to sync bookmarks, passwords, history, tabs, configurations. Also between many pc :) Mozilla Weave (now Firefox sync)
[/OT]

Sven Barth wrote

Nevertheless the Registry is not that bad if you think about it: it's an data storage system that is provided by the operating system itself and thus available to all applications running on it.

Our mileage clearly varies a lot. The registry is a mess. Fully of crap, often corrupt, easy to corrupt and lots of "uninstalled application" information is left there to rot ever though the app is not on the system any more.

Another nice thing are permissions: You can - at least on NT+ systems - set the permissions for single keys (not values though) for specific users and groups.

Yeah, this has been around in *nix systems for 30+ years using text config files and user/group/world permissions. :-)

If your application has the correct priviledges you can also load and unload private Hive-files (the Registry is saved in such files). You can use that for example to make your application portable:

Umm, using a ini file stored in a global location for system wide setup, but also allow the application to see if a ini file is in the same directory as the executable also makes it instantly "portable" (like running from flash drives). Been doing this for years with Windows and Linux software - always using INI files.

yup! exactly what i'm speaking of... i've got genealogy and recipe databases that i added an index to each of... the coders had been going on and on about being unable to speed them up and this and that and "just put it on faster hardware"... thankee but no... "FixYerShit<tm> and oh BTW: i added this index for a major speed increase on my 400mhz dual CPU server"... i think they're still arguing about what it's gonna take to make it go and i've yet to see a new install carrying that index i created... so when i install it, i wait to see what folk say and if they complain about the speed, i go create that index, pick up my $$$ and don't hear anything from them about speed problems any more ;)

Hello Lazarus-List,

Friday, June 18, 2010, 6:39:07 PM, you wrote:

JH> Of course, for huge data amounts of data you cannot avoid JH> using a database but I would prefer if programmers think JH> about performance a bit and not use databases for every JH> small junk.

Its funny to see how many people fight in favor or against databases (embedded in this case) without without taking in account the purpose.

Your approach about directory tree is fine, well is fine if your memory is infinite which you can assume it usually. I had the same problem, the directory traverse and I'm using an embedded database, and yes, SQLite because two processes work with database in parallel, one is a realtime browser about gathered information and the other is the gatherer itself. Why database ? easy, I do not know how much memory will be available when running the program, and I do not know in advance how much data I must gather. I can not put my data in memory and explore/update it because the software can gather 2-4 MB of data or in the worst case a 10% of hard disks space, and in some computers there are several terabytes of data storage. Ah! I can not install a full fledged database server.

Is this an special case ? Yes, it is, but each one is an special case.

Forefox bookmarks in SQLite seems overkill, but does anybody browse looking for the reason of such change ?

--quote-
It stores both the bookmarks and the history as the database with the help of SQLite. This is the reason why Firefox 3 is able to add tags to the bookmarks and the awesomebar is able to list the suggestions as per most visited links. This also enable Firefox 3 to handle these requests very fast. SQLite database bookmark system was added since Firefox 3 Alpha 5
--quote-

The other side effect is that now the private browsing could be achieved simply by creating the database in memory. Same layout, same everything, except database creation. Also if you delete your private data, only one file must be erased and no one will be left behind (all or nothing).

One problem, one solution. One with databases and one without.

Hi!

Caution: I'm going very off topic here, so if anyone expects to read something about embedded databases for Lazarus/Linux, you might ignore this message ;)

I never said that it's perfect. I only wanted to stress that the system "Registry" itself is much more powerful than many people/developers are aware of.
The crap that is lurking around is mainly because of crappy applications, their deinstallers or because of backwards compatibility (from Microsoft's side). But the first two points also apply to file based configurations. I don't think, that Lazarus' uninstaller will remove my configs that I'm setting with "primary-config-path" (I also don't think that it will remove my settings in their default location... :S ). So if I'd never install Lazarus again and I'd never delete those files, I'd also have crap lurking around taking up disk space and wasting time for "search applications" (indexers, anti virus, etc.). Also the performance of the Registry when being full of crap is different from Windows 9x to Windows NT (the secnod one being superior). The Registry from NT is not the same as the Registry from 9x, but most people think of the Registry as the crap it was back in the 9x days... (Global) Config files are also easy to corrupt, so nothing is gained here. The main benefit of the Registry is that it is provided by the OS with a documented/consistent API and the OS itself keeps sure that the Registry is secured/intact/backed up as good as possible.

But what if I want to give Sven full access to a key, Graeme the rights to read and write to the key, read only access for all other FPC developers and no access rights for every one else? (this example might seem constructed, but on multi user systems you might need this) You can't really do that with the usual user/group/world permissions... (you might be able to do that with POSIX ACLs though)
For example I've blocked access to any webbrowser on the family's computer for my brother by only adding his account to the Deny list of the application (and of course his account being a "normal user"). Windows NT supports such ACLs on nearly everything the system provides: Files, Registry keys, kernel objects (yes, NT is object oriented) like Semaphores, Pipes, Sections, Devices... NT itself is much more powerful than most people are aware of... if Microsoft hadn't screwed the Win32 subsystem the way they did (plus the fact of always being Administrator from XP on), we'd have a very relieable system there (and with ReactOS we also have a very compatible open source variant of it :D ). Don't get me wrong here though: I'm not a Microsoft-fanboy. I only appreciate the way they designed the NT architecture and I realise the power that is sleeping in there. Thus I started the NativeNT port of FPC, so that I can somewhen release the full power of such an operating system. :) (yes, my todo list is also very impressive :P )

Yes, that is also a way to do it (and I also do it this way ^^). But the Registry nevertheless allows for things like that and that should be kept in mind when designing the data storage backend for an application - especially if the application is Windows only (for what reason ever).
Also (as said) it might be very useful to write a simple application which can make applications portable that rely on the Registry...

Regards,
Sven

Even funnier when they see 20 year old code on 20 year old computers running circles around their code on modern computers. I have seen it, yes I have.

hehehe, yup! you're not the only one...

newer is better? oh yeah? how? :lol:

El 18/06/2010 20:18, waldo kitty escribi?:

It is a problem of coding time vs other resources. It is a problem of reuse vs specialized routines.

If you don't write things from scratch, you begin to use libraries and the program gets bloated. Even when you write parts form scratch, sometimes you say "I will extend this procedure so it could be used also for...". When you extend a procedure you add IFs to control different cases, so you are penalizing performance a little. Whenever you, as programmer, make the decision "I don't want to worry about these details when I'm coding, I will let the software takes care of those nasty details", you are relieving of work, and charging that work on the software. Instead of making decisions in code time, you let the software make decisions dynamically in runtime. That has some impact on performance, perhaps only a little, perhaps negligible, but accumulative.

A simple case: How many times we use a TStrings when an array of strings would be enough?. How many times in FCL and LCL is TStrings used?. TStrings is a good convenience that is why we used it instead of declaring an array of string of fixed size even when we know the size in advance.

In the old days, adding an IF in each iteration was expensive in performance. Now 1000 IFs per iteration have no impact. Unfortunately, every body does it, so everybody uses 1000 IFs per iteration and finally we have 100.000 IFs that have some impact.

It is difficult to balance between coding time and performance. But comparing with code written 20 year ago is unfair. It was more simple, now we demand a lot to a simple program. And in the old days many things were written almost from scratch. Nevertheless, perhaps we have gone too far and we should be a little more careful.

Santiago A.

Careful guys... You are starting to sound like old farts :)

"You kids, get off my lawn!"

Growing old is hard to do, but its inevitable. If you have kids, you'd know how hard it is to tell a teenager anything. You have to let them think they are discovering something you already know, and then they'll get it thinking it was their idea. Identity building & ego is far more important to a younger developer that your great code. Its a strange trait of us humans. If only we were machines....

They'll work it out eventually. By the time they do, they'll be telling their kids the same thing. And the circle of life continues...

Myles

Well spoken :-)

Maieutics is a very old technique, still worth to study a bit (wikipedia). The essence (here) is to lead people gently into the right(?) direction, based on own experience, and then make them provide their own solutions. Once such a solution came out, one can check it for compatibility with known problems, so that it can result in another or improved version.

Microsoft BTW insisted in making all the well known errors themselves again, in the design of Windows, Office etc. We would have much less trouble with compatibility to legacy code, establishing poor interfaces or data structures, when people would have a closer look at preceding work of others, before starting to do it their own way (out of thin air). OTOH such knowlege can block new ideas, once one sucked up old useful models.

But younger people have similar problems, in convincing older people that things can be done better now, in different ways from the old ones. Habits are cobwebs first, then cables :-(

DoDi