This never happened before when I generated fpdoc help for tiOPF. I tried now with the latest fpdoc in Trunk (2.5.1) and I get the following errors. I then tried with fpdoc from latest 2.4.1 branch and the same thing. The last time I generated fpdoc help for the tiOPF project was 2009-05-28 and then it worked fine.

Looking at the actual code, I cannot see what the problem is. FPC 2.4.1 and FPC 2.5.1 compiles the tiOPF code without such errors.

---------[ fpdoc errors ]------------------
../Source/Core/tiVisitorDB.pas(160,16): Expected ";" at token "Identifier deprecated"
../Source/Core/tiUtils.pas(1371,14): Expected ";" at token "of" ../Source/Core/tiStreams.pas(384,5): Syntax error at token "label"

----------------------------

Here are the actual code snippets. I marked the lines fpdoc reports the errors on.

------------[ tiVisitorDB.pas ]-------------

  // alias class will be removed soon.
  TVisOwnedQrySelect = class(TtiVisitorSelect)
  end deprecated;        // <<-- error reported here

--------------------------------------------

------------[ tiUtils.pas ]-------------
function tiGetFileSize(AValue : string): longInt;
var f: file of Byte;     // <<-- error reported here
begin
  AssignFile(f, AValue);
  Reset(f);
  result := FileSize(f);
  closeFile(f);
end;
--------------------------------------------
------------[ tiStreams.pas ]-------------
function MimeEncodeString (const s: AnsiString): AnsiString;
label                     // <<-- error reported here
  NothingToDo;
var
  l: Cardinal;
begin
  if Pointer(s) = nil then
    goto NothingToDo;
  l := Cardinal (Pointer (Cardinal (Pointer (s)) - 4)^);
  SetLength (Result, (l + 2) div 3 * 4);
  if Pointer (Result) = nil then
    goto NothingToDo;
  MimeEncode (Pointer (s)^, l, Pointer (Result)^);
  Exit; //==>
  NothingToDo:
  Result := '';
end;
--------------------------------------------

Regards,
- Graeme -

The problem is in supporting the 'depecated' and other modifiers. I have added support for that, but apparently, it breaks parsing in other cases (the implementation is rather messy).

Please add bug reports for this, using small code samples.

Also, it is not necessary to report that the compiler compiles it. I assume this is the case anyway.

Michael.

In our previous episode, Michael Van Canneyt said:

1) I would particularly be interested if there is a difference between 2.4.1 (say a week old maximally) and 2.5.1 (likewise). Before my holiday I committed a fix or two.

2) there are more patches/issues in mantis (16340-16347)

Marco van de Voort het geskryf:

1) I would particularly be interested if there is a difference between 2.4.1 (say a week old maximally) and 2.5.1 (likewise). Before my holiday I committed a fix or two.

Marco, you are off the hook. :-)

Damn I love the 'git bisect' command. ;-) Anyway the problem is sub-divided. The errors for 'label' and 'xxx of byte' syntax is introduced in r14938. Sorry Michael, that the big patch from Mattias. Now you will understand why I *hate* such big single commits. Where do you start looking for the problem?? You should ask Mattias for a refund. ;-)

------------------------
Author: Michael Van Canneyt 2010-02-26 11:56:35
Committer: Michael Van Canneyt 2010-02-26 11:56:35 Parent: c8193a4a625fd40bc9c1792c212079261425f94c (* translation updates from kms (Mantis 15849))
Child: 5fcb15f64c3bbcada2c64dd0afa37c508ea37407 (* Fix to work with new version of Pascal scanner)
Branches: fpdoc_ipf, master, remotes/origin/master Follows:
Precedes:

* Complete parsing of Pascal Sources. Contracted work by Mattias Gaertner

git-svn-id:
3ad0048d-3df7-0310-abae-a5850022a9f2
------------------------

The 'deprecated' syntax issue has existed as far back as 2009-05-31. I'll do a 'git bisect' now for that issue and see if it did actually work correctly or not for the tiOPF generated docs back in 2009-05.

Regards,
- Graeme -

Marco van de Voort het geskryf:

1) I would particularly be interested if there is a difference between 2.4.1 (say a week old maximally) and 2.5.1 (likewise). Before my holiday I committed a fix or two.

The 'deprecated' issue hasn't worked as far back as 2009-04 so I take it I can safely assume that fcl-passrc never did have support for the 'deprecated' modifier. fpdoc still generated documentation output even with the unknown modifier.

Regards,
- Graeme -

Michael Van Canneyt het geskryf:

Please add bug reports for this, using small code samples.

'label' and 'xxx of byte' syntax problems reported as: http://bugs.freepascal.org/view.php?id=16476

The 'deprecated' issue has been reported as an FCL issue (due to fcl-passrc): http://bugs.freepascal.org/view.php?id=16477

Regards,
- Graeme -

I can send you my 49 smaller patches. But I doubt that would help. AFAIK deprecated and label were not supported, so I did not break that.
As I wrote on the fpc mailing list:
I extended the fcl parser to parse the FCL code, which does not use all the FPC features (tons of modifiers, special compiler directives, nested classes) and not all the Delphi "bugs" like ommitting semicolons. It took some effort to extend the fcl parser without breaking compatibility and having extendable code.
Pre- and post-modifiers make parsers ugly and complicated, because there is no single start/terminal token anymore and because some modifiers can be identifiers, requiring bigger lookaheads. The fcl parser design requires to know the node type before start parsing the node. This makes implementing lookahead somewhat more difficult (compared to the codetools). That's why I added some more context information. But its parser design is still not the best for pre/postmodifiers. OTOH you get a nice class hierarchy: When the parser is changed affected code is spotted easier than with the codetools.

In short:
It is difficult to add pre/postmodifiers to the fcl parser, because of its design. But its design is good for tools like fpdoc. (Note: the fcl parser was designed by Sebastian Guenther).

Mattias

This is correct. I did (attempt to) add the support for it.

Tell me about it :/

100% correct, and the 'breakage' is entirely my fault; Parsers are not my strongest point (in fact, I hate them, but what can I do...).

Michael.

Mattias Gaertner wrote

I can send you my 49 smaller patches. But I doubt that would help.

Personally, I would have preferred that because it's easier to debug a problem afterwards when things doesn't work. You have smaller commits to work through. Having a 150K patch means I (and probably others that didn't work on that patch) have no clue where to start looking for the problem.

AFAIK deprecated and label were not supported, so I did not break that.

The tiOPF code I tried to document with fpdoc did NOT fail with the 'label' or 'xxx of byte' syntax. Only after your commit did the errors of that syntax appear.

That is why I split the bug report in two. The 'deprecated' issues was always there, but the others wasn't.

I wasn't trying to be rude and point blame (I had a wink in my earlier reply). I'm simply stating that two syntax bugs slipped though your testing (obviously because those syntax wasn't used in FCL. But they are bugs because they worked before.

Michael Van Canneyt wrote

This is correct. I did (attempt to) add the support for it.

Either way using the same tiOPF code, with commit r14937 fpdoc did not raise any error with the usage of 'label', but r14938 and onwards does. If fcl-passrc parser 'label' correctly before I don't know - I'm simply stating that it did not raise an error.

I still say the syntax for modifiers stink! I vote for making them keywords so they can't be used as identifiers. And I vote that the syntax for modifiers get fixed. Sometimes you need a semi-colon before the modifier and sometimes you don't - that's an anomaly, not a syntax rule.

Well, I doubt that the patches would help here. I rewrote some parts multiple times.

Probably because the former parser only parsed the interface. Maybe fpdoc now tries to parse the implementation too? That's not needed, is it?

It was not rude.
I just took the opportunity to write about the stupid pre/postmodifiers introduced by Delphi.

Mattias

It should stop at the implementation keyword ?

Michael.

Yes.
I think implementation needs good comments. The local types and variables don't need fpdoc docs, do they?
If someone wants to use fpdoc for implementation, maybe it should be made optional?

Mattias

For fpdoc only! For other applications using fcl-passrc you might want to parse all source code. So there must be some switch introduced to say how much to parse.

Mattias Gaertner wrote

Well, I doubt that the patches would help here. I rewrote some parts multiple times.

'git bisect' would still have narrowed down the commit with the change causing the problem. Smaller commits = easier to resolve a problem introduced.

Probably because the former parser only parsed the interface.

Good point! That is probably the case.

Maybe fpdoc now tries to parse the implementation too? That's not needed, is it?

For fpdoc, no.

I just took the opportunity to write about the stupid pre/postmodifiers introduced by Delphi.

I don't like the modifier syntax at all. :(

In our previous episode, Mattias Gaertner said:

Mattias, what did you use for testing? makeskel/fpdoc?

I've searched for a test that simply dumped the parsetree (e.g. to check if ^g is really transformed to something like a char/string literal), but haven't found anything.

No. I wrote a small program to parse every fcl unit. Maybe I find today some time to create a patch to make fpdoc parse only to implementation.

I've searched for a test that simply dumped the parsetree (e.g. to check if ^g is really transformed to something like a char/string literal), but haven't found anything.

AFAIR ^g is/was not supported, as this is a context sensitive token. Two tokens in types, one in the others.
The codetools do not support it too.

Mattias

BTW,
The parser parses expression as whole and only stores the whole expression as string. This is done because in order to parse an expression you need to resolve identifiers, because operators are context sensitive. A simple parser can not do that. If the parser does not resolve identifiers it can only create symbol structures which would be as complicated as parsing the expression. In this sense the fcl parser already supports ^g constants.

Mattias

In our previous episode, Mattias Gaertner said:

I don't fully understand this. Afaik ^ in a constant expression is not related to ^ in a type expression?