Hi,

I made a unit to work with files in CSV (comma separated values) format and would like to submit it to Lazarus CCR. What is the best way to proceed?

I made a unit to work with files in CSV (comma separated values) format and would like to submit it to Lazarus CCR.

Have you seen the built-in CSV support in TStrings?

DoDi

Or the TSDFDataset class in sdfdata ?

Michael.

See
http://wiki.lazarus.freepascal.org/Code_Conversion_Guide#Creating_a_Code_Release_Page and further.

Vincent

That's interesting. Where is it? I am unable to find it it Classes unit.

2010/1/22 Michael Van Canneyt :

> Have you seen the built-in CSV support in TStrings? Or the TSDFDataset class in sdfdata ?

Or the two different implementations in tiOPF project. :)

Anyway, to get your code in the CCR, I normally email Vincent the code in private.

Thought I must say, I don't really see the point of the CCR. Many of those projects need to be maintained, and just having releases without revision control systems is pretty stupid in this day and age.

One good example of this is DCPCrypt. I'm now "unofficially" maintaining DCPCrypt because we use it in our company software, but I simply don't have the time to keep making releases after every little change for the CCR site. I even get patches emailed to me in private. So what's on the CCR site is 99% of the time outdated. Each of those downloadable components/projects should really have each their own repository. I'm considering moving DCPCrypt to GitHub - that way at least everybody will be able to get the lastest changes as they happen. GitHub also allows you to download a tarball of the latest revisions of each branch without the need to explicitly create releases or having Git installed on your system.

2010/1/23 Alexander Klenin :

> > Have you seen the built-in CSV support in TStrings? That's interesting. Where is it? I am unable to find it it Classes unit.

I also didn't know about this.

strictdelimiter and delimiter to comma?

Have you seen the built-in CSV support in TStrings?

Yes, if I understand the documentation right, it can be used for basic per-line csv parsing only (using DelimitedText). It also requires to quote all characters below some code point, including spaces, that makes it impossible to read csv files saved from OO Calc or MSO Excel. If I'm wrong I would appreciate if someone point out my mistake.

Or the TSDFDataset class in sdfdata ? Or the two different implementations in tiOPF project. :)

No, I haven't seen either of those. A quick search shows TSDFDataset has field length limit. This approach can greatly improve performance on large datasets, but it is not very convenient for me, because I use CSV to exchange data with Calc/Excel and cannot expect field length to be limited, or at least I don't know the limit.

As of tiOPF CSV implementation, I'm afraid it is already too late :) It might have been a good idea to ask for implementations on the mailing list before reinventing the wheel, but such basic formats are quicker to reimplement than to thoroughly search the web for.

Anyway, to get your code in the CCR, I normally email Vincent the code in private.

That's what I've done.

Thought I must say, I don't really see the point of the CCR.

Well, I thought of making it a separate project, but decided to start with CCR. First, it is still too small piece of code to be a project of its own. Then I believe CCR is the first place to go when searching for components, so it will be easier to find it there. Once the code needs to be updated quicker than CCR allows, it can always be moved to sf.net/github/whatever, but I don't feel such a need for this project at the moment.

Hi,

I think there is need for an OPTIMIZED code for splitting and joining. TStringList.DelimitedText is good when speed is not very important. See:
http:/msg18548.html

I didn't look at tiOPF project's implementation so I don't know how optimized it is.

"CSV" is little misleading because the separator can be some other than comma. Text itself often has commas and thus the text between separators must be quoted. TAB is a much better separator because it seldom occurs in text.

Splitting can be slow. It creates and copies strings and allocates small chunks of memory.
There should be variations for those functions. Sometimes you want to support quoted stings, sometimes not.
Sometimes you want TStrings, sometimes an array of string (which allows more optimization).

Regards,
Juha Manninen

No, the D2006 extension "strictdelimiter" fixes this, and FPC has it too.

I've added kind of an How To topic to TStrings, and also filed an bug report about StrictDelimiters. Currently StrictDelimiters is False when using CommaText. When True, DelimitedText will be read ignoring any but the defined Delimiter (hopefully unquoting if required), and will be written without any quotes.

CSV files have to be processed in two steps, one for entire lines (e.g. ReadLn or LoadFromFile), and another one for splitting a line into the CSV fields (using CommaText or DelimitedText).

BTW, we'll have to replace at least parts of the RTL/FCL (FPC) help, which is not very precise or informative right now.

DoDi

Splitting can be slow. It creates and copies strings and allocates small chunks of memory.

No way around that, unless you use PChars and zero terminated strings, which are slow in any further processing.

There should be variations for those functions. Sometimes you want to support quoted stings, sometimes not.

See StrictDelimiter.

Sometimes you want TStrings, sometimes an array of string (which allows more optimization).

TStrings allows for the same optimization, when you use [Set]Capacity instead of SetLength. Once configured for the expected field count, the TStrings object can be reused like an array.

DoDi

Graeme Geldenhuys wrote

Anyway, to get your code in the CCR, I normally email Vincent the code in private. Thought I must say, I don't really see the point of the CCR.

I guess that is why you missed
http://wiki.lazarus.freepascal.org/OnGuard ;-)

If you want to to maintian the Lazarus/FPC version of any unmaintained port, you are welcome. Please read
http://wiki.lazarus.freepascal.org/Using_the_Lazarus-ccr_SVN_repository and send me your SF account.

If you think you can maintain DCPCrypt best by using GIT, then move it into a git repo. I am willing to add git repos on Lazarus CCR, if you want them. But you may be more comfortable with an a git repo administered by you. As maintainer, you decide.

Vincent

Graeme Geldenhuys wrote

One good example of this is DCPCrypt.

TBH, I am glad it is in Lazarus-CCR or at least mentioned on Lazarus-CCR, instead of lying on barko's hard disk, even though it didn't seem to be maintained anymore.

Vincent

Hans-Peter Diettrich wrote

> If I'm wrong I would appreciate if someone point out my mistake. No, the D2006 extension "strictdelimiter" fixes this, and FPC has it

too.

Thanks, I have not known this.

I've added kind of an How To topic to TStrings

Great, it would be very useful to cannot find it
at the moment, would you please provide a link?

BTW, the unit I mentioned is finally on Lazarus CCR. http://wiki.lazarus.freepascal.org/CsvDocument
Thanks for the help.

2010/1/24 Vladimir Zhirov :

As of tiOPF CSV implementation, I'm afraid it is already too late :)

In case you were interested. Here is the documentation of the unit I wrote. It is stand-alone, so doesn't require ://opensoft.homeip.net/tiopf/core/titokenlibrary/index.html

It can be used for more than just CSV, and can handle a certain amount of errors found in CSV files. It can also be strict reporting on exactly which line and column a syntax error occurred.

Well, I thought of making it a separate project, but decided to start with CCR. First, it is still too small piece of code to be a project of

You are probably correct, if the "project" is only a single unit. But some code found in CCR (like DCPCrypt) is large enough to justify a separate project I think.

Sorry, it's complicated because my update has been rejected by the FPC maintainer, so that it is available neither from online nor offline help. For use with the FPDoc Editor I've started to add FCL and RTL directories to the docs/xml/ directory, but didn't add content yet. Currently I'm trying to prepare build options for offline help, but this will also take some more time.

DoDi

Thanks, I'll have a look.

Hans-Peter Diettrich wrote

For use with the FPDoc Editor I've started to add FCL and RTL directories to the docs/xml/ directory, but didn't add content yet. Currently I'm trying to prepare build options for offline help, but this will also take some more time.

Adding it to Lazarus xml docs is definenely a good idea. Maybe it worth adding also to the wiki as a code example?

Vincent Snijders wrote

TBH, I am glad it is in Lazarus-CCR or at least mentioned on Lazarus-CCR, instead of lying on barko's hard disk, even though it didn't seem to be maintained anymore.

A can't agree more, and that is exactly why I want to share my changes too. I'll make a plan during the week to share my DCPCrypt repository and email you a new DCPCrypt release that actually supports 64-bit FPC. The previous release just made it compilable, but not usable. :-(

Regards,
- Graeme -

I want to prevent duplicate texts in the OH and wiki. The wiki can contain an extended version, covering several topics in depth, while IMO the OH should be more compact, with SeeAlso and Examples references to other places.

But you're right, the wiki can be updated by everybody, so that after some time it can reflect the needs and expectations of multiple users, not only the viewpoint of an single contributor.

Do you have an idea how to add such topic pages, so that they can be found easily?

DoDi

I'm sorry for not responding for quite some time.
I'd suggest adding it to

http://wiki.lazarus.freepascal.org/Components_and_Code_examples#File_formats

that is where I added CsvDocument to. This way, everybody looking for csv parsing solution can immediately see both ones. Then, we could cross-link these articles via "See Also" section.