I just implemented and option to create an XML project from the commandline arguments. This should allow to create projects from the scripts and Makefiles, used to build the documentation for the standard libraries. See Mantis #20769.

Unfortunately there remain some problems, e.g. the FPC documentation cannot be created on Windows, what is one reason for the extended project option, but doesn't allow me to debug this feature with real life projects (RTL...).

I'd like to discuss some aspects of the current implementation, as well as this and further fpdoc extensions. PM acceptable.

DoDi

Hans-Peter Diettrich wrote

I just implemented and option to create an XML project from the commandline arguments. This should allow to create projects from the scripts and Makefiles, used to build the documentation for the standard libraries. See Mantis #20769.

I had a look, I will apply the patch when I find a free moment.

2 small remarks:

1. When coding changes, please try to observe the style used.

for instance
   if .. then begin
   end
is simply not done, this is not C. Begin is always on a new line.

2. Try to separate things. You coded a change in package selection handling, please don't mix this with the change for writing a package file. It makes it more difficult to debug when bugs arise.

Unfortunately there remain some problems, e.g. the FPC documentation cannot be created on Windows, what is one reason for the extended project option, but doesn't allow me to debug this feature with real life projects (RTL...).

Why not ?

I'd like to discuss some aspects of the current implementation, as well as this and further fpdoc extensions. PM acceptable.

Feel free to discuss.

Michael.

Unfortunately there remain some problems, e.g. the FPC documentation cannot be created on Windows, what is one reason for the extended project option, but doesn't allow me to debug this feature with real life projects (RTL...).

A "make -n rtl.chk > test.txt" succeeded, at least. Now I suspect some Windows or RTL commandline limitations, which seem to truncate the long command lines created by the scripts (4890 chars for rtl.chk).

Now I'll try to implement such a dry-run option for fpdoc, too, that accepts an command file like created by make -n. This extension, and more features, could become distinct projects, but for that purpose the fpdoc.pp file should be splitted into the bare program Run part, and everything else moved into a new unit. Such a split could be accompanied by a Lazarus package, created from the fpdoc worker code, that then could be used immediately in LazDE or further GUI applications.

Another extension should allow to debug and update fpdoc projects, which currently don't run on Windows. The observed fatal errors can result from bad XML structure, or from missing include files.

The latter problem, together with the commandline length, restricts the use of fpdoc to Linux systems, where everything seems to work right now. But is this a reasonable restriction, which we can tell all FPC users?

DoDi

This is a problem of windows, yes.

Try working with the FCL, there are less units, and thus a corresponding less long command-line.

Please elaborate ? Last I looked, everything is already split in separate units. The main program file just collects command-line options, and then passes on everything to the actual workers. What more do you want to see split out ?

Another extension should allow to debug and update fpdoc projects, which currently don't run on Windows. The observed fatal errors can result from bad XML structure, or from missing include files.

What do you mean with "currently don't run on windows" ?

Michael.

Well, I don't like your style as well :-(
I could stop myself before removing all those empty lines, that make it hard to review your code...

2. Try to separate things. You coded a change in package selection handling, please don't mix this with the change for writing a package file. It makes it more difficult to debug when bugs arise.

How can I create patches for different topics, when the whole application only works after all required changes have been implemented together?

I understand that you'll have problems with (my) patches, of course. If you have *practical* suggestions, I'll try to match your expectations better.

I could e.g. enclose the various topics by conditional compilation, but I'm not sure whether this will be really helpful.

See my parallel mail, I suspect general problems with long command lines on Windows.

This suggests to me to use 2 file lists, for the descriptions and *also* for the input (source) files. But IMO the list of description files could be dropped entirely, when fpdoc simply searches for an XML file for every given source unit, in the current or a given documentation directory. It's useless to enumerate the XML files, when documentation is created only for the given input files.

The missing include files problem is harder to solve, because the scripts have to create different compiler options for every platform. This leads me to the question, how "portable" the make and other scripts should be, so that their output can be used on multiple platforms.

Perhaps the following settings:

const
   DefOSTarget    = {$I %FPCTARGETOS%};
   DefCPUTarget   = {$I %FPCTARGETCPU%};

should be replaced by the (already existing) commandline options, e.g. --ostarget, so that the given include directoris match the expectations of the script writers, and parsing the source files does not fail with missing include files. Dunno whether that helps, I did not yet dig into these details.

What you could do right now: split the fpdoc program file, so that further programs can be created for specific tasks, like creating an fpdoc project instead of creating output. You can do this from the unpatched version, so that nothing will be broken by that update.

This would allow to move most of the changes into derived classes, where the affected methods can be introduced or overridden, instead of modifications to the original classes and methods. Then it would also be possible to use the fpdoc worker code in other applications, like LazDE or build_lcl_docs. In detail I want to implement a GUI for the creation and maintenance of fpdoc projects, that allows for comfortable project debugging.

DoDi

I'm not a huge fan of the style used in the compiler, but am I complaining? I'm even trying to concentrate as good as possible to adhere to the style used there, so that the changes I want to see merged are indeed merged.

You and I are not the "masters" of these projects, so we can not decide which style to use. And a consistent style throughout a project (e.g. the compiler or fpdoc) is important. "When in Rome do as the Romans do"

Regards,
Sven

You could have done so, and the result would have been that your patch would maybe not have been accepted.

Always try to mimick the style used in code you modify. The FPC team sticks to this rule ourselves when we modify other people's code, and we ask the same from you. (as you well know)

Selection of the package should be separate from that IMHO, if not, there is an error in your logic.

Then you misunderstand how fpdoc reads files.

1. It's perfectly possible for a single XML file to contain documentation for several units. Practically, it means 1 XMl file can contain many <package> and <module> nodes. That I decided to use 1 XML file per unit is pure coincidence. Vice versa, the documentation for 1 unit can be split over 2 XML files. fpdoc will merge them.

Since there is no 1-1 relation, the description files must be enumerated.

2. The input file can contain a limited number of compiler options '-DSomething'. Therefor, it cannot be used to determine the filename (unless you start parsing it)

I cannot answer this question.

It won't, see above.

Since the project definition already is in a separate file, I fail to see what can be moved more ? Please be more specific.

Currently, the fpdoc.pp source file only contains code which is useful in a command-line enironment: it is a TCustomApplication class descendant which collects command-line arguments and passes them on to the objects that do the actua work. It contains no other logic.

All code doing the actual work is in the other units, ready for re-use by a GUI environment, or any other command-line tool you care to create.

If there are things you don't understand, feel free to ask questions, and I will answer them as best as I can.

Michael.

.
.

The current limit of the default MS Windows shell is 8 kB according to information found on Microsoft pages (among others). Are those 4890 characters mentioned above Unicode (UTF-16) characters (thus breaking the limit)? Anyway, it might be possible to use a different command shell not imposing such a limit, but that obviously requires installation of additional software and thus may not be a completely general solution.

Tomas

Hans-Peter Diettrich wrote

A "make -n rtl.chk > test.txt" succeeded, at least. Now I suspect some Windows or RTL commandline limitations, which seem to truncate the long command lines created by the scripts (4890 chars for rtl.chk).

Does this limitation apply when using TProcess under Windows as well? Truncating a command line.

If not, why not remove the Makefile idea completely, and simply create a console application that calls the appropriate applications with the correct parameters. I have removed all make files and scripts from some of our projects (and for the generation of fpGUI or LCL docs into INF or HTML), and replaced them with simple command line programs that can be compiled from: fpc somecoolutitility.pas

You then have TProcess, the whole RTL and FCL at your disposal. MUCH more powerful than scripts or makefiles.

Am 29.11.2011 12:18, schrieb

But the 32k characters limit of CreateProcess is reached harder than the 8k or 2k limits of the command line...

Regards,
Sven

Yes.

http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx

Specifies a limit of 32 Kb.

Linux has a limit of 5000 arguments (or thereabouts).

Exactly the ideas behind the switch to fpmake and fppkg.

Michael.

In our previous episode, said:

IMHO we should get rid of the need for extremely long commandlines where possible. No immediate rush, but a new solution must be better in this, otherwise it is just waiting for the next special case.

Concerning fpdoc, I just didn't get around to converting the docs to use a project file. It's on my todo list.

For compilation, you know we are toying with the idea to let fpmake create a build file and let the compiler compile that.

Michael.

I'm not a huge fan of the style used in the compiler, but am I complaining?

Did I?

I'm even trying to concentrate as good as possible to adhere to the style used there,

Where's the code formatter, that would eliminate all such discussions?

so that the changes I want to see merged are indeed merged.

Small patches should not cause any problems, there it's also easily possible to respect the style of the source code. Larger or experimental patches, as the one for fpdoc, usually deserve further work before they fit into the existing codebase. It was more a sketch of what could be done, with debug code that has to be removed again when everything works as expected. But I suppose that it will be a long way until all involved developers only agree in how the new features should be implemented.

DoDi

In this case I'd search the error in the existing logic. Projects are a recent extension to fpdoc, which has not even been tested thorougly. That's why there are many places in the code, which deserve a reconsideration of the current model and procedures.

This IMO is not very practical, not supported e.g. by LazDE. In contrast it were more practical to *split* documentation, so that e.g. large classes can be documented in their own file. I never tested whether fpdoc could deal with such files as well...

Since there is no 1-1 relation, the description files must be enumerated.

In either case an enumeration of the XML files in the documentation directories would be sufficient. Fpdoc already ignores all documentation, for which no source files are given. When memory usage is of concern, the excess modules can be deleted in the DOM tree.

2. The input file can contain a limited number of compiler options '-DSomething'. Therefor, it cannot be used to determine the filename (unless you start parsing it)

I'm not sure whether a description file is required *before* an input file is being or has been parsed. It also looks somewhat expensive to me, when *all* description files have to be loaded before the unit specific documentation can be created, because fpdoc has to search *all* description files for an matching module name.

IMO it were much more handy when the names of the description file(s) would become an attribute of every input file, so that they can be loaded on demand. When no such XML attribute exists, as is, a fallback to the current search procedure can be used. For the current commandline usage the description file could be added to the input specification, perhaps as a new option -D<descr.xml>.

Consider the implementation of a stand-alone tool, that takes a commandline and creates an fpdoc project from it - this is what I just tried. When both the application class definition *and* its use reside in the program file, its impossible to select a derived application class, that performs an specific task or extends the functionality.

All code doing the actual work is in the other units, ready for re-use by a GUI environment, or any other command-line tool you care to create.

This is not true - see what e.g. TFPDocApplication.CreateDocumentation does.

begin
   With TFPDocAplication.Create(Nil) do
     try
       Run;
     finally
       Free;
     end;
end.

When this were the *only* code in the fpdoc program file, another project could create a derived class, with a possibly specialized Run method, without touching the declaration or implementation of TFPDocApplication.

DoDi

Argh :-(

Try working with the FCL, there are less units, and thus a corresponding less long command-line.

Certainly I could use other (smaller) packages, what also would speed up debugging, but this would not eliminate any problem that Windows users have with creating local documentation right now.

BTW, "make fcl.chk" works here :-)

But what about the many unknown targets? Do they result from the inability to create (or use?) the RTL content file?

I also cannot create LCL documentation, it fails with an unhandled exception. The reason may be the same as for the RTL documentation, I did not yet test it right now.

DoDi

There exists another problem, resulting in a commandline containing --input='../rtl/win32/classes.pp
Obviously the single quote (') should be a double quote, at least for the Windows shell.

After fixing this manually a "make rtl.chk" works fine :-)

DoDi

I ran into my own problem when writing a program in freepascal that also should compile in Delphi (don't ask)
Logically you would expect $mode delphi to enforce delphi compatibility at the sourcecode level.
But this is not the case if f.e. C style operators are used or specified, or macro's.
I accidentally used the former in my code.
Shouldn't it be better if compatibility is enforced both ways?

I am sure this would have been discussed before.
Shouldn't the compiler at least generate a hint or possibly a warning, when non-delphi compatible syntax is used in delphi mode? Or with a bit more logic, shouldn't those options be excluded by $mode delphi and -Mdelphi?

Or if this is inconvenient something like $mode strictdelphi?

Thaddy

Possibly, but it would also require RTL changes (the FPC system unit contains many types, constants, variables and functions that do not exist in Delphi -- and for programmers the difference between the language and the system unit is often not very clear) and possibly also blocking certain packages (which can't be compiled with Delphi). It would be a lot of work even if you would limit it to only the compiler (the use of which would be questionable), and additionally only few people are interested in such functionality.

For backwards compatibility reasons, such changes would have to go into a separate mode switch. Patches for that could be accepted, but again, without also changing the RTL and packages I'm not sure how much it would help you in reaching your goal.

Jonas

PS: please do not reply to existing messages when starting a new topic, it messes up threading both in mail clients and in web archives.

Partly, yes.

Well, it's a good test case for a project file.

Michael.

That would conflict with the GUI TApplication instance, so I really don't see the point of this exercise.

Michael.

Then another error in the logic exists: CreateDocumentation should be a method of TFPDocProject, not of TFPDocAplication.

DoDi

Hans-Peter Diettrich wrote

> I'm even trying to concentrate as good as possible to adhere to the > style used there, Where's the code formatter, that would eliminate all such discussions?

You could try the Jedi Code Formatter included with Lazarus. I've never tested it myself, because it is the first thing I disable when configuring a new Lazarus :D

Regards,
Sven

Sven Barth wrote

You could try the Jedi Code Formatter included with Lazarus. I've never tested it myself, because it is the first thing I disable when

That copy of JCF is bugy. Rather use the original version available for download on SourceForge, from the JCF project. Also with the same configuration file, JCF from SourceForge JCF project works, but when using it with JCF included with Lazarus it generates wrote (incorrectly styled) code. Somewhere inconsistencies crept into the Lazarus version.

Why ? TFPDocProject is just storage of data.
It has no dependencies on the rest of fpdoc, and this is by design.

Michael.

Where else should the CreateDocumentation method reside, so that it is available in other projects? As long as it resides in TFPDocAplication, which again resides in the fpdoc program file, only the fpdoc program can create documentation - what makes the remaining fpdoc units useless in other projects.

DoDi

I do not think it should be available in other projects. I can perfectly think of a system where all descriptions are in a database, and then the whole CreateDocumentation method is useless. In fact, such a system existed for a short time, in a web front-end, which created a single page only.

again resides in the fpdoc program file, only the fpdoc program can create documentation - what makes the remaining fpdoc units useless in other projects.

No, it just requires them to copy about 50 lines from fpdoc.pp and possibly adapt these for their particular use-case. There is no problem with that.

I see no point in abstracting out a single routine. First of all since this routine is useless without knowledge of all the rest of the routines and setup. Doubly so because later on there'll be inevitably questions to adapt it to some particular use-case for which it wasn't designed in the first place. I think it is better to copy this routine; The person wanting to do something with it will need to study the sources anyway to understand the system.

Feel free to disagree, though.

Michael.

This is what I would worry about as well, if I used free pascal or lazarus.

Having a strict delphi mode, would bring free pascal one step closer, to using it instead of Delphi.

There are other issues which would then remain though like:

Lazarus Forms
Lazarus Packages ?

And perhaps slightly less good debugger ?

Also unicode troubles could remain.

Strict delphi mode would be a good step in the right direction ;) =D

Bye,
Skybuck.