Hi list,

There are some bugs open for sdfdataset, e.g.
22894 Sdfdataset: empty file with FirstLineAsSchema reports Recordcount 1 instead of 0
22882 SDFDataset .AllowMultiLine does not support multiline import

I've written a test suite to check sdfdataset behaviour. An older version of this is already in FPC trunk in the database tests (tcsdfdata.pp).
The test set can be donwnloaded here:
https://bitbucket.org/reiniero/fpc_laz_patch_playground/downloads

Currently, taazz on the forum is improving sdfdataset against this test set: http://lazarus.freepascal.org/index.php/topic,18224.0.html

If there are any sdfdataset users out there:
1. I'd very much appreciate additional test cases for tcsdfdata.pp. Especially if you think there are issues with current sdfdataset, please don't hesitate to send an tcsdfdata.pp with more tests.

2. Presumably sdfdataset is used to exchange data with Delphi systems using tstrings.delimitedtext. Therefore I'm stressing Delphi compatibility in my tests. Do you think this is justified?

3. Taazz has an improved version of sdfdataset available at http://lazarus.freepascal.org/index.php/topic,18224.msg103232.html#msg103232 My test suite contains test results for this code; an additional test is satisfied, but there are still failing tests.
Do you think this version is good enough? Comments here or on the forum would be welcome.

Thanks very much,
Reinier aka BigChimp

No, definitely not. The two are totally unrelated.

Michael.

I'm very confused here. To say the least.

So remarks like these from sdfdataset:
14/Ago/01 Version 2.00 (Orlando Arrocha)
Delimiter property added to TSdfDataSet. No more dependency on CommaText
methodology -- choose your own delimiter.

means sdfdataset just happened to use commatext but the format does not need to be sdf?
(SDF => see
http://docwiki.embarcadero.com/Libraries/en/System.Classes.TStrings.CommaText)

And the name sdfdataset does not mean the dataset actually needs to use sdf format data?

Then as far as I'm concerned, I'll withdraw the TestInputOurFormattests from
packages\fcl-db\tests\tcsdfdata.pp
as that doesn't make sense.

Further, I propose removing multiline support for sdfdataset as 22882 proves sdfdataset cannot read multiline fields (it can write them). Multiline fields are possible with Delphi sdf (see bug 19610), but as sdfdataset need not be compatible, this can be stripped out to avoid "half a functionality".

Finally, I'll post on the forum that sdf compatibility is not one of the goals of sdfdataset.

Is there some defined on-disk format that sdfdataset should be following?

Thanks,
Reinier

Define SDF ?

(SDF => see http://docwiki.embarcadero.com/Libraries/en/System.Classes.TStrings.CommaText)

That is not a usable definition.

And the name sdfdataset does not mean the dataset actually needs to use sdf format data?

That depends on how you define SDF data.
There are 76 definitions of SDF. Several of which are a data format.

The one I know is a fixed-length format (Standard Data Format), so totally different from what Borland/Embarcadero/whatever knows.

Finally, I'll post on the forum that sdf compatibility is not one of the goals of sdfdataset. Is there some defined on-disk format that sdfdataset should be following?

As I understood it, it is either fixed length or CSV. CSV as in http://tools.ietf.org/html/rfc4180

Michael.

That is in contradiction with the existing implementation as well as the following comments following comments in SDFData.pp

14/Ago/01 Version 2.00 (Orlando Arrocha)
John Dung Nguyen showed me how to make this compatible with C-Builder

and

Delimiter property added to TSdfDataSet. No more dependency on CommaText
methodology -- choose your own delimiter.

Is fpc promoting version 77 of the sdf definition?

Ludo

You are erroneously assuming I actually read the comments.

I added TSDFDataset to FPC because it added an implementation of CSV and fixed format data, because this is how I had used them. That's all I cared about. I had never heard of SDF as a data format. At the time, I probably assumed SDF were the initials of the author or so.

Michael.

No specs, no docs, only code and comments to go by.

That is indeed another approach that doesn't require specs nor docs. Unfortunately only available to those that have the keys to kingdom :)

Ludo

Well, I think that trying to let TSDFDataset conform to certain "SDF" specs is trying to do some unjustified backfitting.

IMHO, it should do fixed format and CSV as indicated by the RFC I posted. All the rest is nice, but not required from my perspective.

Michael.

The only problem I have with TSDFDataset is the following issue related to more strict CPU targets: http://bugs.freepascal.org/view.php?id=22964

Regards,
Sven

Sorry... wrong bug... *blush*

I meant this: http://bugs.freepascal.org/view.php?id=17871

Regards,
Sven

People can only go by what they see (they can't read your mind when you committed sdfdataset).
There is:
- the unit name ("sdfdataset")
- the comments
- the code
- the output
All of this makes it in my opinion more likely to be SDF than CSV.

CSV is not SDF.
See the test results from the Delphi test I posted to the list earlier. For a description of what SDF is (based on Delphi help as well as the tests), see
http://wiki.lazarus.freepascal.org/SDF

If you understand it is CSV, why not document it, even if only in a readme:

In contradiction to what its name may apply, Sdfdataset reads and writes CSV data according to RFC4180, not the Delphi SDF format.

PS: I'll leave writing a test set to test against RFC4180 for people who care.

If conforming to the format that is specified in all existing evidence is wrong but conforming to some format that is never mentioned at all is right, I give up.

IMO this is no way to run any serious development effort: don't expect people to read minds about what was intended if that is not documented, *especially* in cases like this where any knowledgeable person's obvious conclusion is the one you don't want them to make.

Also, first accepting patches that explicitly aim for sdf/delimitedtext compatibility (bug 17285,22213) and then stating the opposite is an excellent incentive for me to drop everything that's sdfdataset related.

IMHO, it should do fixed format and CSV as indicated by the RFC I posted. All the rest is nice, but not required from my perspective.

It's not required anymore from my perspective either. I just wanted to fix sdfdataset so that it does what it says on the tin.

I'd suggest:
1. adding a readme as indicated in my other mail so that users and developers do not fall into the same trap
2. documenting similar unwritten assumptions in other relevant units as well. Not doing so is a great way to discourage contributors
3. dealing with issues 22894, 22882 as you see fit

Reinier

Oh and
4. considering removing the TestInputOurFormattests test from packages\fcl-db\tests\tcsdfdata.pp

Thanks, Sven,

Perhaps the modified testsdf.pp could be added to the db test suite (packages\fcl-db\test), but I suppose some writelns need to be replaced by conditional functions (the writelns can be handy when running some tests).
I suppose the db test suite doesn't run at all on wince?

Excuse me, but I'm not spending any more effort on sdfdataset tests.

Thanks,
Reinier

AS I understand, there are in sdfdata.pp:

TFixedFormatDataSet - for SDF = "(*S*tandard *D*ata *F*ormat) A simple file format that uses fixed length fields."

TSdfDataSet - for variable field length, where fields are delimited by Delimiter (and probably quoted) ... we can call it "CSV" and here we can follow http://tools.ietf.org/html/rfc4180 (with exception, that field delimiter can be user defined - .other than comma)

It will be good also document in begining of sdfdata.pp what standard is followed ;-)
L.

I never tested it, so I can't tell. For the normal FPC test suite there is the possiblity to "cross run" it using either SSH or an emulator... and for WinCE there is the following description in the readme:

=== text begin ===

Example cross testing of target arm-wince
-----------------------------------------
//arm-wince example:
see FPCTRUNK\DEMO\WINCE\TESTEMU\ for additionally required tools Connect your device via ActiveSync and execute:

make TEST_FPC=ppcrossarm TEST_CPU_TARGET=arm TEST_OS_TARGET=wince \ TEST_OPT="-XParm-wince- -WC -Xs" \
EMULATOR=MyDisc:\My\Path\to\wcetemu.exe

Tests will be performed in the \fpctests folder on the device.

=== text end ===

[though I don't know in what state the test emulator is...]

Regards,
Sven

Like I said, I had never heard of SDF.

Look, for me, this is not a requirement set in stone, I am quite flexible.

You guys asked what the original idea was, and I explained some of the history behind the dataset.

If the desire is to have a TSDFDataset (whatever the specs) and a separate CSV dataset, fine, we'll make it so, this is really not a problem.

I don't want anyone to feel that he has wasted his time :-)

Michael.

Hoho, there is no trap :-)

2. documenting similar unwritten assumptions in other relevant units as well. Not doing so is a great way to discourage contributors

That is definitely not the intention. See my other mail.

I was frankly surprised by the strong responses I got.

Any assumptions I made were mine, and definitely not the law, I just gave them as 'historical background', because that is how I perceived the original question :-)

Michael.

<snip>

> All of this makes it in my opinion more likely to be SDF than CSV. Like I said, I had never heard of SDF.

I understand (and appreciate) your reasoning when you committed it. Having a CSV dataset is very handy.

Before I understood the full horror of what SDF really means, especially as implemented by Delphi (see the nasty behaviour when you add spaces after a quoted text and before a delimiter) I thought SDF was really some form of CSV, too. (Well I suppose it is, it's just not RFC4180 CSV unless you e.g. decide to quote everything. IIRC then it actually complies with RFC4180).

Look, for me, this is not a requirement set in stone, I am quite flexible.

So am I... see below.

You guys asked what the original idea was, and I explained some of the history behind the dataset.

Not really, I asked what the specs were ;)

If the desire is to have a TSDFDataset (whatever the specs) and a separate CSV dataset, fine, we'll make it so, this is really not a problem.

Actually, if you are happy with an SDFDataset that complies with RFC4180 CSV (but not SDF), I would vastly prefer that over the atrocity that is SDF - as long as this requirement/behaviour is quite clearly specified (e.g. in a readme).

If you think that any changes to sdfdataset to make it RFC4180 compliant are acceptable to current users, fine by me ;)

The csvdocument [1] code has a very nice record-by-record CSV parser and writer with test set that aims to be RFC4180; this could be used for interoperability testing or even, if evidence by test set failures, used instead of parts of the current code.

I don't want anyone to feel that he has wasted his time :-)

Well, getting clarity about what the on disk format must be is worth it even though I can't really say I enjoyed this particular episode :) Even better, as I said, if everybody is happy with RFC4180 CSV, I vastly prefer that format to SDF and would consider my time well spent.

I'm sure tatamata from the forum will appreciate it too (AFAIR, his ZMSQL is based on sdfdataset).

I'm just not interested right now in implementing either a test set or the changes myself, but perhaps somebody else is.

Michael.

Thanks,
Reinier
[1] http://wiki.lazarus.freepascal.org/CsvDocument

Well, not intentional, but it sure is confusing.

Ok, that explains some of the disconnect then.
I've responded to your other mail ;)

Friends? ;)

I can relate to that, being confused myself :-)

I have never thought otherwise :-)

Michael.

No problem :-)

If you think that any changes to sdfdataset to make it RFC4180 compliant are acceptable to current users, fine by me ;)

I have not seen any evidence that it has ever been used other than for CSV, but I may be wrong. So I suspect that this should be acceptable.

I'm fine with that.

Well, in the not-too-far future, I'll spend time on improving the DB testsuite.

Michael.

On 25-9-2012 11:05,

Well, in the not-too-far future, I'll spend time on improving the DB testsuite.

In that not-too-far future, you might want to have a look at the following patches ;)
22970 [Patch] Database test suite: expanded readme 22972 Firebird EXECUTE BLOCK RETURNS test
22976 [Patch] Database test suite: temporary table test: add support for Firebird

thanks,
Reinier

used the SDF dataset were only to load a CSV into a TMemDataset or similar for further computing.
I found it simpler than CSVDocument (which anyway I discovered later) because I can setup everything in the form designer and I only have to change che filename I simple call to
copyfromdataset will then copy the real data to the working dataset. Finally, when everything is done, I use a TCSVExporter (also completely filled in at design time and bound to the working dataset) by calling its Execute method.

Very selfishly, If you drop the fixed length support, I won't notice it; while if you rename the TSDFDataset to a TCSVDataset, then I have to tweak & rebuild a few bit-tons of code.

R#

It seems that gmane has started mangling the mailing list address (the above mail was sent to rather than to ). The result is that such mails are held for moderator approval (not including the list's address amongst the recipents is a spammy property). Please either ask gmane to fix this, or manually change the to-address when replying via gmane to the proper mailing list address (if that is possible).

Thanks,

Jonas
FPC mailing lists admin

and

22377 [Patch] Opening a non-empty dataset doesn't return BOF=true (at least the test case part)
23008 [Patch] fcl-db DBtestframework Firebird embedded support

As far as sdfdataset is concerned,
this one by John Kozikopoulos
22980 [FPC] Database Components
is currently assigned to Joost van der Sluis; you might want to have a ,
Reinier