Unfortunately I managed to supply the EasyImports.patch twice, please drop Mantis #21168 as a dupe of #21167. Sorry for the inconvenience :-(

Starting with this patch I suggest further improvements of fpdoc. Except for the RTL docs, which require a couple of special compiler options for every input file, a package project can be shrunk essentially to the specification of these options:
--package the package name
--imports a list of related packages
--input-dir containing all source files
--descr-dir containing all XML files
--output-dir where the output files are created
--format the document format

Until now every --format requires special --output and --import values, what IMO is not really required. The backend can create an standard output filename, given only an output directory, e.g. <output-dir>/<package-name>.chm. When the content files reside in that directory, the content and import specifications can be created by the backend as well, e.g. <output-dir>/<package-name>.xct. In addition to the supplied patch, a new --imports option can accept a list of packages to import, e.g. --imports=rtl,fcl.

The already existing --input-dir option should be extended by compiler options, applied to every source file. Several solutions are possible, e.g. for the FCL with an explicit file list an additional --input-options option could specify the common compiler options. For packages with a simpler source directory structure it would be sufficient to specify e.g. --input-dir="../lcl,-Fi../lcl/include" or, also applicable to the FCL, --inputs="@input.lst,-Fi../../include". Which solution looks the best to you?

These improvements will allow to reduce e.g. the command to build the LCL documentation to:

fpdoc --package=lcl --inputs=../lcl,-Fi../lcl/include --descr-dir=xml/lcl --imports=rtl,fcl --output-dir=/wherever --format=html

For building e.g. CHM docs only the --format options has to be changed as required.

I can supply another patch, if you like, covering above updates.

DoDi

Hans-Peter Diettrich wrote

Unfortunately I managed to supply the EasyImports.patch twice, please drop Mantis #21168 as a dupe of #21167. Sorry for the inconvenience :-(

Thank you. I will look at it.

As I said before:

Neither --input-dir nor --descr-dir will be put in the fpdoc implementation.

I don't see the problem of having to specify all files that are part of the project explicitly. Make does it, Lazarus does it, Delphi does it, Visual Studio does it.
All require you to specify exactly which files should be there when compiling, and rightly so.

For the --input-dir there is an extra reason: the order of files is important. Just like in the compiler, which must compile dependent units first, fpdoc should first document dependent units. It currently does not know how to do this by itself.

So, a patch providing these 2 options will not be applied.

Michael.

As I said before: Neither --input-dir nor --descr-dir will be put in the fpdoc implementation.

What are you talking about? They are already in fpdoc :-)

I don't see the problem of having to specify all files that are part of the project explicitly. Make does it, Lazarus does it, Delphi does it, Visual Studio does it. All require you to specify exactly which files should be there when compiling, and rightly so.

Millions of flies ...? ;-)

Look e.g. at the generation of the LCL docs, which I already used as the test case for my fpdoc updates.

Again, what are you talking about? FPDoc doesn't require a special order of input files, neither source nor documentation files :-)

DoDi

In our previous episode, Hans-Peter Diettrich said:

It does :-)

Contrary to the compiler the fpdoc files can refer freely to each other. Is the above "should first document dependent units" your preference or is there really a reason?
What if I need an alphabetical or logical order (e.g. the most important at top)?

Mattias

In our previous episode, Mattias Gaertner said:

The parsing order and the order in the overview pages should then be decoupled (e.g. by a sorting event that is configurable)

The parsing order is mandatory even. I still have to research the consequences for latex. The latex backend currently doesn't seem to suffer from order problems which is probably means it is vulnerable to duplicate identifiers.

The html backend however isn't free from this, but has the limitation that it only suffers from duplicate identifiers wrt already parsed units and .xct modules.

It does, see the explanation of Marco.

Which is why I again removed the input-dir option (but this is not yet committed).

Michael.

> Again, what are you talking about? FPDoc doesn't require a special > order of input files, neither source nor documentation files :-) It does, see the explanation of Marco.

Then I wonder how I ever could generate the documentation for a single unit?

The LCL documentation build sorts the units alphabetically, and this also seems to work.

DoDi

Of course it produces documentation, but cross-links can and will fail because the correct target is not found.

For cross-links to be correct, the order must be correct.

See the following bug

http://bugs.freepascal.org/view.php?id=19144

Meanwhile I have a simple example using 2 units floating around somewhere. It took me lots of time to find it, but now I know where the cause is. Solving it is another matter entirely.

Michael.

In our previous episode, Michael Van Canneyt said:

Meanwhile I have a simple example using 2 units floating around somewhere. It took me lots of time to find it, but now I know where the cause is. Solving it is another matter entirely.

I've been thinking about it, and following the compilers way of working is the only solution.

If you want to resolve identifiers like a compiler, you have to build scopes like a compiler, and do that in the right order.

Otherwise funny things will happen with duplicate identifiers.

I know.
The problem will get only worse with nested type declarations, which are next on my todo list.

This whole mess is why I am writing lots of FPCUnit test cases and using a structured approach. IMHO the only way to do this properly.

But this takes time, of course.

Michael.

I still don't understand this requirement, and how documentation writers should establish the correct order :-(

Which is why I again removed the input-dir option (but this is not yet committed).

I see no need for removing this feature. A user can sort the files later, in the project files, when fpdoc gave hints on what should be improved.

In the meantime I implemented the improved --input and --input-dir options in the FPDocManager. They require no modification of the fpdoc units, but can be added there as well. Essentially the file and file list specifiers are extended, so that common compiler options can be appended to the (automatically generated or explicitly given) file list.

I'm not sure yet how to implement the --output-dir option, it affects the --output, --content and --import values, and should be implemented in fpdoc itself. For now I try to implement it in the FPDocManager, and leave it to you to eventually add it to fpdoc as well.

DoDi

Linear documents IMO should follow their own rules, which make the docs readable. A simple concatenation of element descriptions is a poor man's solution, far away from e.g. the Delphi(7) PDFs. I'd suggest a skeleton, containing the document structure and verbose parts, and include instructions for the XML elements, overviews and tables, created e.g. for class members.

The html backend however isn't free from this, but has the limitation that it only suffers from duplicate identifiers wrt already parsed units and .xct modules.

Isn't this a matter of properly linking identifiers, declarations and descriptions to each other? What detailed information does the PParser need about identifiers declared somewhere else?

DoDi

Simple.

The same order as the compiler uses: it is 100% determined by the uses clause.

Why is this so surprising ? How do you think the lazarus codetools manage this ? Exactly the same way. It's not black magic, but very deterministic.

Michael.

What's wrong with the current #package.unit.identifier... qualification? [Apart from the Delphi/.NET aberration of dotted unit names, which should be avoided in FPC :-]

Otherwise funny things will happen with duplicate identifiers.

Is this a problem of the implementation parser, or does it also apply to the interface sections?

Until now I only found problems with overloaded proedures or methods, which are documented in only one element - nasty but liveable.

Quite a different problem are e.g. platform specific identifiers, for which the XML files have no special provisions - the same descriptions are used for every combination of --ostarget and --cputarget values. Also liveable when the docs mention all eventual essential differences, and fpdoc always ignores the descriptions of elements without declarations. But it becomes very nasty when the fpdoc commandlines contain hard-coded platform-specific include directories, because then the user will not see help on identifiers of his actual platform :-(

In such cases I also can understand that the parser is fooled by search pathes for multiple platforms at the same time. Is this problem addressed (and cured?) by the macro expansion in mkfpdocproj?

DoDi

> Otherwise funny things will happen with duplicate identifiers. I know. The problem will get only worse with nested type declarations, which are next on my todo list.

One more problem introduced by uneducated Delphi developers. Must FPC really follow all these aberrations, which only bind development capacity, without any adequate improvement?

I wonder what a chaos will manifest, as soon as somebody tries to implement the new Delphi (XE2) namespaces in FPC :-(

DoDi

Please explain "cross-links".

Can you give an example of what can go wrong, what's the wrong result, and what fpdoc errors are (or are not) reported then?

DoDi

Only because YOU think they are aberrations this must not be the opinion of others. Also they have proven rather useful in context of generics.

I wonder what a chaos will manifest, as soon as somebody tries to implement the new Delphi (XE2) namespaces in FPC :-(

Ehm... AFAIK the only difference from XE2 namespaces to pre-XE2 namespaces is that you can select a default namespace (and pre-XE2 namespaces are already implemented in FPC).

Renaming the RTL units so that they use namespaces is a different topic though and was not yet decided upon.

Regards,
Sven

How is that related to documentation and the order of input files?

When every module can have a different order of the used units, how should this be reflected in an fpdoc commandline or project?

Why is this so surprising ? How do you think the lazarus codetools manage this ? Exactly the same way. It's not black magic, but very deterministic.

Please don't blame the users, and don't remove useful functionality like --input-dir, when the bug resides in the documentation generator, and cannot be cured by ordering the input files :-(

DoDi

If you refer to a type in a unit, then the parsing engine tries to resolve the type. If it finds the type, then it creates a straight reference to this type. if it does not, it creates a TPasUnresolvedTypeRef instance.

From this you can see that the order in which the parsing engine parses units matters:

if the order is correct, then you will not have any TPasUnresolvedTypeRef instances, because the engine will be able to find all types. If the order is not correct, then there will be lots of TPasUnresolvedTypeRef instances.

Now, when the HTML engine constructs its pages, there are links that it must resolve by itself (parent classes, type references in all kinds of places etc). If a TPasUnresolvedTypeRef is encountered, it will not know how to make the reference. It can (and does) try to guess, but often the guess fails.

When every module can have a different order of the used units, how should this be reflected in an fpdoc commandline or project?

It's a simple tree.

I am not blaming anyone for anything.

But if I tell you that order matters, please be so kind to assume that I know what I'm talking about: I spent far more time on fpdoc and all related units than anyone else on these lists.

As for the presence of --input-dir, we'll see. I understand why you want it. God knows I spent enough time mailing about it.

Michael.

What's the question for this answer? ;-)

IMO we're discussing the third case, when multiple declarations might apply. Then the parser has to search the units in their order within the uses clause, which is unrelated to the order of input files.

Why not try to resolve that reference later, e.g. after construction of the content table?

> When every module can have a different order of the used units, how > should this be reflected in an fpdoc commandline or project? It's a simple tree.

So what? IMO you didn't understand the problem :-(

Well, maybe you don't see the wood for trees any more? ;-)

As for the presence of --input-dir, we'll see. I understand why you want it. God knows I spent enough time mailing about it.

Perhaps it helps to look closer at the questions, before you give answers to questions never asked?

DoDi

The question you asked:

"How is that related to documentation and the order of input files?"

Michael.

In our previous episode, Michael Van Canneyt said:

The question you asked: "How is that related to documentation and the order of input files?"

I've the feeling that Dodi assumes all references are fully qualified and not just the inter-package ones?

I'm not going to guess at what other people assume.

I know what is wrong/missing in fpdoc, I know what needs to be done to fix it. I am working on it.

Michael.

No. I'm familiar with parsers in theory and practice, but not with the implementation details of the pparser and fpdoc.

I know about the use of unqualified names (and links) in source code, and that these must be looked up in an declaration tree. Then I only can assume that such a tree is built from the currently parsed source module, but references to other unites (in "uses") are not implemented properly. As you already pointed out, the search order depends on the order of the units in the uses clause, while the parser currently seems to use the order of fpdoc input files and imports? These (technical) aspects never has been described in all the answers, instead everybody proceeded immediately to explain solutions for never mentioned implementation flaws :-(

My only point is the impact on the order of input files. When two source modules have the following uses clauses:

   A: uses X, Y;
   B: uses Y, X;

then the order of X and Y in the fpdoc inputs can not satisfy both requirements at the same time. That's all.

As you already pointed out, this requires that the search order depends on the order in the uses clauses, but lacking information about the current implementation I cannot help you out with more practical suggestions.

DoDi

Well, you have the sources, you can look up the actual implementation. Then you would see that:

* The fpdoc engine keeps a list of units to parse. It keeps all symbols from all units it parses in a large tree.
* The passrc parser parses 1 unit at a time.
* If the parser encounters a uses clause, it simply notes the dependency; no more, no less. In particular: it does not start parsing the unit in the uses clause if it was not yet parsed.
* If the parser needs to look up a symbol, it searches for symbols in the fpdoc list, in all the units for which dependencies were noted.

From this it follows that the order of parsing the units matters:

the symbol must be there in order for the parser to find it.

To avoid having to specify the correct order, there are now 2 possible solutions:
- We do 2 passes.
One to establish parsing order (it can stop after the uses clause), the second to actually parse.
- We parse recursively. But only if the unit is in the input list, otherwise you end up parsing all of the FPC sources.

Michael.

Well, you have the sources, you can look up the actual implementation. Then you would see that:

Please don't confuse syntax and semantics. Reading source code reveals only *what* is done, but not *why* it's done, and what are the consequences.

Lazy variation:
Stop when a used unit was not already parsed, postpone the blocked unit in the parse list, and move the used unit in front of it - kind of bubble sort.

- We parse recursively. But only if the unit is in the input list, otherwise you end up parsing all of the FPC sources.

That's how FPC works, except for the input list.

Thanks for the enlightenment, now I also see the impact of the input list order :-)

DoDi

Michael Van Canneyt wrote

But if I tell you that order matters, please be so kind to assume that I know what I'm talking about:

Michael, maybe this "difference of opinion" can be resolved if you post your example of the 2 units the reproduces the error (like you mentioned in a earlier post). I looked at the bug report you mentioned, and no simple/small example was given that reproduces the problems that order is important. And I agree with your comments in that bug report - I too, will not going to look at LCL docs/mess either.

I know I once hit this problems a year or two ago (and I believe I mentioned it to you at the time), but to this day, can't reproduce it again. I also looked at the fpdoc code at one point, and from that code I couldn't see a problem, and assumed all was fixed.

So I am very curious myself to see and example which reproduces the

Regards,
  - Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://fpgui.sourceforge.net