Hi Group,

How should the TStringList behave when

Delimiter := ' ';
QuoteChar := '"';

?

On my

Lazarus 0.9.28.2 r22279 FPC 2.2.4 i386-win32-win32/win64

when reading a line like this

Sep 3 16:27:32 router_ip src="internal IP:46987" dst="external_ip:64880" msg="Traffic Log" note="Traffic Log" devID="0019CBDF94A3" cat="Traffic Log" duration=300 send=168 rcvd=48 dir="LAN:WAN" protoID=17 proto="others" trans="Normal"

{the above should be one line}

I will obviously get the date split in 3 different Strings[], !but! it also splits the 'msg="Traffic' and 'Log"' - is this correct ?

Kind Regards,
Lukasz

2010/9/7 Lukasz Sokol :

Maybe StrictDelimiter is true?
http://www.freepascal.org/docs-html/rtl/classes/tstrings.strictdelimiter.html

Vincent

No, after forcing it to false, it still behaves the same (splits 'msg="Traffic' and 'Log"').

I know this is possibly a corner case, when Delimiter = ' '; but there seems something wrong with the logic in SetDelimitedText... ? (stringl.inc) It fails on things like 'msg="Traffic Log"'... because it does not understand that a string can be composed of plain and quoted part and should then be treated as one:

	// next string is not quoted
       j:=i;
       while (j<=length(AValue)) and
             (Ord(AValue[j])>Ord(' ')) and
             (AValue[j]FDelimiter) do inc(j);
       Add( Copy(AValue,i,j-i));
       i:=j;

maybe could become

  while (j<=length(AValue)) do
    begin
      // we might find QuoteChar in string that begins unquoted
      if (AValue[j]FDelimiter) then inc(j)
	else
      if (AValue[j]=FQuoteChar) then
        begin
          repeat
            inc(j);	// carry on until you find another quote, regardless of what is quoted
	  until (AValue[j]=FQuoteChar) or (j=length(AValue));
        end
      else
      // do we really have to have a special case for ' ' ? for backward compatibility maybe
      // but then be it lower priority than quotechar and delimiter
      if (Ord(AValue[j])>Ord(' ')) then inc(j)
	else break;
    end;

This should be enough to support 'name="some value"' separated by spaces... and also "some name"="some value" too... - comments ?

(the if...then...else...if maybe isn't optimal, but shows the idea)

HTH,
Lukasz

W dniu 2010-09-08 12:03, Lukasz Sokol pisze:

I think TStringList expects at this situation items like "msg=Traffic Log". What you need is extended list with support to Keyword=Value for example having property Values[Item : string] : string returning left side of equation. Something I used in KOL but not sure if it can cope with " quote char.

Bogus?aw

Yes. They are not quoted, since the first char after a separator is not a quotechar. Only when the first char after a separator aggregate is a quotechar, quotes are processed.

Afaik TStringlist doesn't support stuff like this.

But plain TStrings (hence TStringList too) already has the
Names[index], Value['Name'], ValueOfIndex[Index] properties;

And IIRC Delphi supports '"Name with spaces"="Value with spaces","Other Name"="Other Value"' format on plain TStringList... CSV IOW.

And

Sure. But it also reacts exactly the same to this test.

In short, this is a combination of TStringList features taht is not supported.

I know this is possibly a corner case, when Delimiter = ' '; but there seems something wrong with the logic in SetDelimitedText... ?

That was my opinion as well, when I had a look at the code some time ago. Either we copy the Delphi code, for compatibility, or we provide our own exact (and meaningful) definition of the behaviour.

[...]
Your code looks good to me (snipped it for lost formatting). But there may exist another corner case, when the quoted text contains line breaks. IMO these should be treated as part of the quoted string, i.e. quote chars should have precedence over *all* other characters. This may require special code, when otherwise the text is processed line-by-line.

Alternatively we could use an enumerated type for the parse mode, so that more than only StrictDelimiter=True/False can be distinguished.

DoDi

The Delphi compatible behaviour must be followed, and is currently implemented. Delphi and FPC give the same output with the testdata provided. Thus, no change is required.

For a good understanding: TStrings is not a substitute for a CSV kind of format. Any attempt at using it for that is doomed to fail. Use a proper CSV format handling class if you need CSV. For example, the sdfdataset unit delivered with FPC contains one such class.

Michael.

OK, so I stand corrected :/

For a good understanding: TStrings is not a substitute for a CSV kind of format. Any attempt at using it for that is doomed to fail. Use a proper CSV format handling class if you need CSV. For example, the sdfdataset unit delivered with FPC contains one such class.

Well it works for 'simplest' case(s); I'll have a look at sdfdataset (time allowing).

You may also want to have a look at CSVDocument class (http://wiki.freepascal.org/CsvDocument). I tried to handle all corner cases when implementing it, including line breaks within quoted text. It is likely to be somewhat slower than sdfdataset though, since it uses variable-length string values instead of fixed-length.

Just to be complete:
the sdfdataset unit contains a variable-length component as well.

Michael.

Lukasz Sokol wrote

> > For a good understanding: TStrings is not a substitute for a CSV kind of format. Any attempt at using it for that is doomed to fail. Use a proper CSV format handling class if you need CSV. For example, the sdfdataset unit delivered with FPC contains one such class. > Well it works for 'simplest' case(s); I'll have a look at sdfdataset (time allowing).

I have written a CSV parser for tiOPF 2 project as well. It can be used outside tiOPF as well, because there isn't actually any tiOPF dependency. The unit is called tiTokenLibrary.pas in the Core directory - tiOPF is hosted on SourceForge.net

Documentation for tiTokenLibrary.pas is available here:

http://opensoft.homeip.net/tiopf/core/titokenlibrary/index.html

Michael Van Canneyt wrote

Just to be complete: the sdfdataset unit contains a variable-length component as well.

Yes. I was not quite clear, I messed fixed/variable and limited/unlimited length concepts. TSdfDataSet does have variable field length but it still requires you to know maximum field length in advance. This approach improves performance greatly but is not very convenient in cases where data is less predictable.