Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2007-11

CSV via PCRE

10 messages · 7 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

S. Fisher

Fields are separated by commas, but if a field is surrounded by double quotes it can contain commas---in fact, can contain any byte whatsoever; double quotes (") within the field must be doubled, just as single quotes within a Pascal string are doubled.

All we need in order to parse a csv record is a single regex and a few trimmings:

'\G\s*"([^"]*(?:""[^"]*)*)"\s*,|\G([^,"]*),'

The part to the left of the pipe is for fields that are enclosed in quotes; the part to the right is for ordinary fields.

{$mode objfpc}
uses pcre, strutils;

{

Regex PCRE options
Some of these things can be triggered by "(?x)" at the start of the regex, where x is a lowercase letter. In particular, "(?i)" forces case-blind matching, "(?s)" forces dot to match newline, "(?x)" forces whitespace to be ignored, and "(?m)" forces ^ and $ to match around newlines, not just at the beginning/end of the string. (\A and \Z match the beginning and end of string respectively independent of this flag, so you can get the effect of PCRE_ANCHORED by wrapping your pattern with them.)

}

type
  string_list = array of ansiString;
procedure parse_csv( s: ansiString;  var ary: string_list );
const
  field_pat : iRegex = nil;
var
  captures: iMatch;
  str, field: ansistring;
  where, truelength: longint;
begin
  // Has the regular expression engine been created?
  if field_pat = nil then
    field_pat := regexCreate(
      '\G\s*"([^"]*(?:""[^"]*)*)"\s*,|\G([^,"]*),'  );
  str := s + ',';
  setlength( ary, 32 );
  truelength := 0;
  where := 0;
  while where < length( str ) do
  begin
    captures := field_pat.match( str, where );
    if captures.groups[1].index > -1 then
      // Field was surrounded by quotes.
      field := AnsiReplaceStr(captures.groups[1].value, '""', '"' )
    else
      field := captures.groups[2].value;
    if truelength >= length( ary ) then
      setlength( ary, truelength + 32 );
    ary[ truelength ] := field;
    inc( truelength );
    where := captures.groups[0].index + captures.groups[0].length;
  end;
  setlength( ary, truelength );
end;
var
  fields: string_list;
  i: word;
begin
  parse_csv(
    ' "Make one'#13#10'or two lines" ,brick,"""Look at that,"" he said"',
    fields );
  for i := 0 to high(fields) do
  begin
    writeln( '  Field ', i+1, ' ---------------------' );
    writeln( fields[i] );
  end;
end.

__________________________________________________ Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Reply by email

Graeme Geldenhuys
to S. Fisher

OK, while we are busy with show-and-tell... Then have a look at my token library implementation.

http://tinyurl.com/395vgp

* It's based on a Infinite State Machine.
* No external units required.
* Allows multiple separators (user selectable) between tokens.
* Allows for user selectable seperator characters.
* Does line number and position error reporting in case the CSV file is not well formatted.
* Only parses the string once, so if you request multiple tokens, it takes no performance it.
* I also think it is much easier to understand and extend than than than single regex which looks more to me like your are cursing! ;-)

Sample Usage:

tokenizer := TTokens.Create(FieldSpecLine, ', ', '"', '"', '\',
                    tsMultipleSeparatorsBetweenTokens);
try
  lField := tokenizer.Token(2);
  lAnotherField := tokenizer.Token(4);
finally
  tokenizer.Free;
end;

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Micha Nelissen
to Graeme Geldenhuys
Graeme Geldenhuys wrote
OK, while we are busy with show-and-tell... Then have a look at my token library implementation.

You've implemented some kind of 'cut'. But grep is also very useful (and more often used in a shell, at least by me).

Micha

Reply by email

S. Fisher
to Graeme Geldenhuys: OK, while we are busy with show-and-tell... Then have a look at my token library…
--- Graeme Geldenhuys wrote 16 lines
OK, while we are busy with show-and-tell... Then have a look at my token library implementation. http://tinyurl.com/395vgp Sample Usage: tokenizer := TTokens.Create(FieldSpecLine, ', ', '"', '"', '\', tsMultipleSeparatorsBetweenTokens); try lField := tokenizer.Token(2); lAnotherField := tokenizer.Token(4); finally tokenizer.Free; end;

That's not a working sample. It has no CSV record to parse.

Give a working program that we can run with no modifications whatsoever; parse an actual CSV record; print every field in the record. That's what my sample did.

__________________________________________________ Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Reply by email

Graeme Geldenhuys
to S. Fisher
S. Fisher wrote 6 lines
That's not a working sample. It has no CSV record to parse. Give a working program that we can run with no modifications whatsoever; parse an actual CSV record; print every field in the record. That's what my sample did.

The code shown in the url below works just fine. Also the usage sample is all you need to use the tokenizer. Just replace the FieldSpecLine variable with the content from a CSV file and you are good to go. I use it as-is in my production code. Plus that unit has been well unit tested as part of the hourly tests run by the tiOPF project.

http://tinyurl.com/395vgp

But if you insist, I can give you a full running application. My point was that regular expression are normally a nightmare to debug and maintain. Plus not everybody knows them (syntax wise), so it make it hard for other developers to maintain that code. Implementing something like a CSV parser in a more Object Pascal fashion goes more with the style of the language (easy to read and understand) and is much easier to maintain by others if needed.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

S. Fisher
to Graeme Geldenhuys
--- Graeme Geldenhuys wrote 7 lines
The code shown in the url below works just fine. Also the usage sample is all you need to use the tokenizer. Just replace the FieldSpecLine variable with the content from a CSV file and you are good to go. I use it as-is in my production code. Plus that unit has been well unit tested as part of the hourly tests run by the tiOPF project. http://tinyurl.com/395vgp

It won't compile.

Free Pascal Compiler version 2.2.0 [2007/09/09] for i386 Copyright (c) 1993-2007 by Florian Klaempfl
Target OS: Win32 for i386
Compiling tiTokenLibrary.pp
tiTokenLibrary.pp(26,2) Fatal: Can't open include file "tiDefines.inc"

But if you insist, I can give you a full running application. My point was that regular expression are normally a nightmare to debug and maintain.

I don't think so.

Plus not everybody knows them (syntax wise),

Not everyone knows how to tie his shoelaces.

Regular expressions are used by vi and emacs; in fact, any editor that doesn't let you do a regex search is a joke. (Even some microsoft applications understand regexes.) So everyone who programs should learn regular expressions.

Regular expressions are also used by grep, and by the languages awk, Ruby, Perl, etc. Every programmer should know how to use at least one of these languages. Those who don't should perhaps be lumped together with COBOL programmers.

Therefore, I honestly believe that anyone who doesn't understand regular expressions has a huge, glaring, inexcusable hole in his computer knowledge. Ignorance of regular expressions is like not knowing how to tie one's shoelaces: it's not something to boast about.

The fact that a single regular expression takes as much time to read and understand as many lines of Pascal isn't a problem, because the regex does as much work as many lines of Pascal. Don't worry about that; you'll spend more time writing them than reading them.

Regular expressions are no more against the spirit of Pascal than associative arrays (hash tables) or any other feature that is added by using a unit.

__________________________________________________ Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Reply by email

Bee
to S. Fisher
Not everyone knows how to tie his shoelaces.

One who doesn't tie his shoelaces doesn't mean he doesn't know how to do it. ;)

Regular expressions are no more against the spirit of Pascal than associative arrays (hash tables) or any other feature that is added by using a unit.

http://z505.com/cgi-bin/qkcont/qkcont.cgi?p=Regex%20vs%20Maintainability%20Graph

IMO, it's a nice read. Personally, I agree with many of points in this article.

Of course regex is important and very useful. But it has its own advantages and disadvantages, like everything else. There are situations or conditions where some programmers prefer to use his own custom but readable and maintainable parsing code over regex cryptic syntax.

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Reply by email

Marco van de Voort
to S. Fisher: It won't compile. Free Pascal Compiler version 2.2.0 [2007/09/09] for i386 Copyright (c)…
Show quoted text 9 lines
> But if you insist, I can give you a full running application. My > point was that regular expression are normally a nightmare to debug > and maintain. I don't think so. > Plus not everybody knows them (syntax wise), Not everyone knows how to tie his shoelaces.

Not everybody realises that adding a interpreter to a compiled program to parse a fixed constant is somewhat odd.

(rest skipped, to ridiculous to spend time on)

Reply by email

Florian Klaempfl
to S. Fisher: It won't compile. Free Pascal Compiler version 2.2.0 [2007/09/09] for i386 Copyright (c)…
Regular expressions are used by vi and emacs; in fact, any editor that doesn't let you do a regex search is a joke. (Even some microsoft applications understand regexes.) So everyone who programs should learn regular expressions.

I can write regexs and decypher them, I even being guilty for having written the regexpr unit in fpc :) but I don't use them though I'am coding professionally and as a hobby and even writing compilers ;)

Reply by email

Jeff Wormsley
to S. Fisher: It won't compile. Free Pascal Compiler version 2.2.0 [2007/09/09] for i386 Copyright (c)…
S. Fisher wrote 14 lines
Regular expressions are used by vi and emacs; in fact, any editor that doesn't let you do a regex search is a joke. (Even some microsoft applications understand regexes.) So everyone who programs should learn regular expressions. Regular expressions are also used by grep, and by the languages awk, Ruby, Perl, etc. Every programmer should know how to use at least one of these languages. Those who don't should perhaps be lumped together with COBOL programmers. Therefore, I honestly believe that anyone who doesn't understand regular expressions has a huge, glaring, inexcusable hole in his computer knowledge. Ignorance of regular expressions is like not knowing how to tie one's shoelaces: it's not something to boast about.

I have been programming professionally for 18 years, in roughly 10 different languages, including assembler. I have never once written a program with vi or emacs, nor have I ever programmed in awk, Ruby or Perl. Or COBOL, for that matter. Does that make me somehow unprofessional?

'\G\s*"([^"]*(?:""[^"]*)*)"\s*,|\G([^,"]*),'

Claiming a line of gibberish like that given above is either readable or maintainable is the real joke. Adding all the pages of code to parse gibberish like that above, to be able to handle 40,000 different possible text formats, when all you need is to handle simple CSV files, is also a joke.

Jeff.

I haven't smoked for 1 year, 2 months and 3 weeks, saving $2,035.90 and not smoking 13,572.71 cigarettes.

Reply by email