Hi,

Currently fpdoc supports a hybrid HTML syntax for documentation content. I use fpdoc a lot, and over time and noticed a few problems:

* Document writers think it's full HTML, so they add other HTML tags inside the content, because they are used to using fpdoc to generate HTML output. HTML is definitely not the only output format used with fpdoc. We have PDF, PS, IPF, XML, TXT, MAN etc.. Those embedded HTML tags now causes problems in the non-HTML output generated.
* HTML or the hybrid-HTML used by fpdoc uses the same tag brackets as XML, so you can't easily insert a < or > or a & character inside your documentation content.
* The hybrid-HTML tags are very verbose
* Nested content inside lists are difficult to get right, sometimes impossible (I had this issue before, but can't remember the exact example. I'll have to give it some thought again).

Here is a simple example.

--------------------
< p >Paragraph one.< /p >
< p >< b >Note< /b >< br >
Paragraph two goes here.
< /p >
< p >Paragraph three goes here.< /p >
--------------------

Just look at the amount of tags for something so simple. We don't write emails like that, or any other text for that matter, so why must we burden the documentation writers to do so.

Why couldn't the above be something like this... a simplified "light" markup. --------------------
Paragraph one.

[b]Note[/b]
Paragraph two goes here.

Paragraph three goes here.
--------------------

* Tags use the square brackets, so as not to interfere with XML tags.
* Hard line endings, are considered the beginning of a paragraph.

Or use something as simple and popular as Creole or WikiText. --------------------
Paragraph one.

*Note*\\
Paragraph two goes here.

Paragraph three goes here.
--------------------

* A blank line denotes a end of paragraph.
* The \\ denotes a forced line break.
* The * before and after a word denotes bold text.

Everybody seems to know AsciiDoc, WikiText, Creole syntax. They are all very simple, very near plain text, not very verbose and easy to parse. They don't interfere with XML tags, and doesn't make the documentation writer think he/she may use full HTML tags in the documentation content (the syntax is clearly different).

I have already written a simple Object Pascal based Wiki that parses the syntax of the square bracket example, and started a parser for the Creole syntax.

Even so, I lately think using AsciiDoc style syntax is still the best option for documentation. It is pretty much the same syntax as you would use to write a README.txt document or a text email. The learning curve is near a flat line - everybody can do it without even thinking about it. Just looking at this email alone, it is pretty much how you would write a AsciiDoc document. :)

So would the FPC team consider such a change in fpdoc? Obviously I will do most of the work, as I am suggesting the change. If you guys don't agree, I'll probably still do the work by forking fpdoc (which I don't really want to do), to fulfill my own needs and some of the open source projects I work with (fpGUI and tiOPF use fpdoc a lot). Mixing HTML tags inside XML or the documentation content is just not the right way to go. fpdoc is great, I simply want to make it even better.

References
==========
AsciiDoc: http://www.methods.co.nz/asciidoc/
Creole: http://www.wikicreole.org/wiki/Creole1.0 My OP wiki: http://opensoft.homeip.net/wiki/wiki.cgi?p=Main-Page

Regards,
- Graeme -

+1
Although I don't know how attributes like /var should look like. Can you create a full list of supported tags?

It should be possible to still read old xml pages. Maybe add an "dataencoding" tag at the beginning of a fpdoc format? And we need a converter.
I don't think a mix of different formats in one file is needed.

Mattias

Graeme Geldenhuys wrote

Hi, Currently fpdoc supports a hybrid HTML syntax for documentation content. I use fpdoc a lot, and over time and noticed a few problems:

[snip]

I think this is largely a matter of opinion, but I will not go into that.

One thing which is not clear from your explanation:

Do you want to drop the XML entirely, or just replace the content of the <short> and <descr> nodes with asciidoc ?

fpdoc is great, I simply want to make it even better.

I am absolutely not against introducing this, I even welcome any attempt to make fpdoc better.

But there are some "rules" to adhere to.

a) The asciidoc parsing (or whatever you call it) engine *must* be in the FCL as a separate package (I need to be able to maintain it in case you don't). Preferably with testsuite, although I don't insist on that. b) It must exist alongside the current format (obviously), and the two must be mixable on a per-file basis. (that is, one format per file, but 2 files may have a different format)
c) all current 'extra' constructs must somehow be supported. (printshort, img, link etc.)
d) Documentation of the format in fpdoc.tex (in LaTeX)

I think most of these rules speak for themselves :)

I suggest --ascii-descr=filename as a commandline option, so the fpdoc engine can link together the plain xml and asciidoc engines.

And of course I do not intend converting the current documentation to asciidoc :-)

Michael.

In our previous episode, Michael Van Canneyt said:

> fpdoc is great, I simply want to make it even better. I am absolutely not against introducing this, I even welcome any attempt to make fpdoc better.

I think losing full layouting capabilities (even though sometimes problematic) would be an huge loss. Specially because now we are also think about using fpdoc for normal helpfiles.

However, Graeme does have a point that it is overkill for many cases, and a simpler option would be ok.

b) It must exist alongside the current format (obviously), and the two must be mixable on a per-file basis. (that is, one format per file, but 2 files may have a different format)

Ok, you made that point already.

c) all current 'extra' constructs must somehow be supported. (printshort, img, link etc.)

And alink. (which I still have to define :-) It is a link to an abstract tag. It's my next chm package todo point (currently finishing multiple window support).

Somewhere else, the abstract tag then generates a list of links that point to that abstract tag. These links are indexed, so that these indexes can be summed up over multiple helpfiles (e.g. make a lemma "stringroutines" that reaches over all stringroutines).

IIRC Graeme said IPF also supported this, and for html we can just generate add the linkdata to .xct, and generate static pages from that. Less flexible (only links to lower pkgs work), but already a start.

Op 2010-07-09 09:32, Mattias Gaertner het geskryf:

+1 Although I don't know how attributes like /var should look like. Can you create a full list of supported tags?

For which syntax? The square bracket one, Creole, WikiText, AsciiDoc?

It should be possible to still read old xml pages. Maybe add an "dataencoding" tag at the beginning of a fpdoc format? And we need a converter. I don't think a mix of different formats in one file is needed.

Yes, I was planning a converter for the existing XML files. I also agree that syntax must not be mixed in the same file.

Regards,
- Graeme -

Zitat von Graeme Geldenhuys :

A list of fpdoc xml tags that needs to be supported. Michael already started a list of all tags in the short and descr fields.

Mattias

Op 2010-07-09 09:38, Michael Van Canneyt het geskryf:

Do you want to drop the XML entirely, or just replace the content of the <short> and <descr> nodes with asciidoc ?

Sorry, I see I wasn't clear in my original post. The XML file can stay, it's just the syntax used for the documentation content in the <short> and <descr> that should change. It's the help content syntax I have a issue with, not the usage of the XML file itself.

a) The asciidoc parsing (or whatever you call it) engine *must* be in the FCL

Not a problem.

as a separate package (I need to be able to maintain it in case you don't). Preferably with testsuite, although I don't insist on that.

You should know me by now. Test suites are a given. I didn't know you can program in any other way than TDD (Test Driven Development). ;-)

b) It must exist alongside the current format (obviously), and the two must

This shouldn't be you suggested, a parameter telling fpdoc what syntax is used in the description is a good solution.

c) all current 'extra' constructs must somehow be supported. (printshort, img, link etc.)

whatever "light" markup syntax we use, if that syntax doesn't support something like <printshort>, then we simply make our own syntax symbol for it. All the syntax I mentioned support img and link symbols already.

d) Documentation of the format in fpdoc.tex (in LaTeX)

Again, not a problem.

And of course I do not intend converting the current documentation to asciidoc :-)

I wasn't suggesting that, but I do intend to write a converter from the old format to new format. You can decide if you want to switch or not. Obviously, I'll do lots of testing on this with the existing fpGUI and tiOPF documentation, as I'd much prefer a light markup syntax for those projects.

Regards,
- Graeme -

Op 2010-07-09 09:51, Marco van de Voort het geskryf:

I think losing full layouting capabilities (even though sometimes problematic) would be an huge loss.

Could you give an example? Do you mean you hard-code HTML tags inside the <descr> and <short> contents? This is exactly what I am trying to prevent.

The documentation should have it's own markup syntax, and the backend (HTML, IPF, LaTeX etc) should translate that markup to its native syntax and formatting.

Specially because now we are also think about using fpdoc for normal helpfiles.

I thought about that as well, and fpdoc wasn't designed for that. My thinking was that whatever new syntax (for the help content) is used, the parser for that could be reused in a new "general application help" tool, which also supports various output formats. There could possibly be some overlap (common code) between that new documentation tool and fpdoc, but that should rather become a task of refactoring, once both tools exist.

Another since step forward will be a WYSIWIG editor for the help content - completely hiding the markup syntax from the help author, but that fall outside the scope of FPC.

Regards,
- Graeme -

Op 2010-07-09 10:47, Mattias Gärtner het geskryf:

A list of fpdoc xml tags that needs to be supported. Michael already started a list of all tags in the short and descr fields.

Page 23 of fpdoc.pdf (FPDoc Users Guide).

Tag Description
------------------------------------------
b Format bold
caption Specify table caption
code Syntax highlight code
dd definition data
dl definition list
dt Definition term
i format italics
img include image
li list element
link Cross-reference
ol numbered list
p paragraph
pre Preformatted text
remark remark paragraph
table Table
td Table cell
th Table header
tr Table row
u format underlined
ul bulleted list
var format as variable
------------------------------------------------

The Creole and AsciiDoc websites show their syntax for most of the tags listed above. What is not covered (I think <var> being one of them), we can make our own markup syntax for.

Now we only need to come to a consensus of what "light" markup we want to use for fpdoc's new syntax? I already mentioned here that I favour AsciiDoc, as it is the closest to standard email or plain text format - lowest learning curve.

Regards,
- Graeme -

In our previous episode, Graeme Geldenhuys said:

Yes.

The documentation should have it's own markup syntax, and the backend (HTML, IPF, LaTeX etc) should translate that markup to its native syntax and formatting.

It could be an option for people that want everything to be wholly portable. It should not be forced. There should be a passthrough option for people that only use backend (be it latex or html or whatever), so that these don't get forced to use some politically portable lowest common denomitor subset.

Maybe this should be typed though (iow you need to specify what the contents is), becuase that can help the parser that substitutes the "own" tags.

> Specially because now we are also think > about using fpdoc for normal helpfiles. I thought about that as well, and fpdoc wasn't designed for that.

Michael said he saw some possibilities. (since it can add arbitrary topics)

There has been nothing yet decided. Michael saw some possibilities for lazdoc/fpdoc, nothing more.

But whatever solution is chosen, I would allow people to use the full possibiltiies of the backend, and import/reuse fragments they already have in the backend language.

Another since step forward will be a WYSIWIG editor for the help content - completely hiding the markup syntax from the help author, but that fall outside the scope of FPC.

More trouble then it is worth IMHO.

Op 2010-07-09 11:11, Marco van de Voort het geskryf:

It could be an option for people that want everything to be wholly portable. It should not be forced. There should be a passthrough option for people

True, I guess we could add some "don't parse me" markup similar to a <pre> tag, which will be copied as-is into the backend output. I actually think AsciiDoc already has something like this. AsciiDoc also supports a <whatever backend>.conf file, which can be configure per user or project, to tweak the backend output to their liking.

AsciiDoc also supports backend or parse hints in some markup. for example, in the Source Code example shown in the Cheat Sheet, the specify the programming language, so the backend could correctly syntax highlight the sample source code.

eg:
.Optional Title
[source,perl]
----
# *Source* block
# Use: highlight code listings
# (require `source-highlight`)
use DBI;
my $dbh = DBI->connect('...',$u,$p)

    or die "connect: $dbh->errstr";

----

AsciiDoc Cheat Sheet:
http://powerman.name/doc/asciidoc

I guess something like that could be added to the pass-through markup tag to tell the parser to which backend the pass-through content belongs too.

> I thought about that as well, and fpdoc wasn't designed for that. Michael said he saw some possibilities. (since it can add arbitrary topics)

The major obstacle as I see it, is that fpdoc cannot generate documentation without the *.pas files. General application help will not be associated with source code files. But I guess at this point Michael knows the inner workings of fpdoc better that I.

Regards,
- Graeme -

In that case, you'll still have problems with < and > (and possibly other) characters, unless you plan to put everything in CDATA sections.

> a) The asciidoc parsing (or whatever you call it) engine *must* be in the FCL Not a problem.

With modified LGPL license ?

[snip]

Again, not a problem.

In that case, I expect your patch next week ;-)

Well, I don't see the point in learning yet another syntax (however simple

Michael.

Op 2010-07-09 11:11, Marco van de Voort het geskryf:

It could be an option for people that want everything to be wholly portable. It should not be forced. There should be a passthrough option for people

Ah, here is the passthrough option in AsciiDoc - also from the Cheat Sheet page posted in my previous message.

++++
*Passthrough* Block
<p>
Use: backend-specific markup like
<table border="1">
<tr><td>1<td>2
</table>
++++

But like you said, maybe extend it to give hints as to which backend supports this passthrough content.

Regards,
- Graeme -

Michael Van Canneyt wrote

Well, I don't see the point in learning yet another syntax (however simple

If the time it takes to learn the new syntax is less than the time saved modifying and writing documentation then it's worthwhile ;-).

Micha

This is correct, but the solution is about 10 lines of code; instead of using the pascal file to generate the list of pages, I just need to loop over all package/module <topic> tags; that's it.

Michael.

I considered that. There is no gain. You'll have to check the asciidoc syntax just like you must check XML/HTML syntax.
Just look at the formatting errors in the wiki pages, with their supposedly 'simple syntax'...

And if there was a WYSIWYG editor, the whole point would be moot...

Michael.

In our previous episode, Michael Van Canneyt said:

Won't WYSIWYG with multiple backend (and thus renderers) become a mess? Most of them have the habit of producing quite dirty code (with empty and very nested tags), and not all backends would react the same on this.

Despite my dislike of the limitations of wiki, the solution they have (edit with tags, have an example rendering in a memo below) is easier and more practical I think.

Op 2010-07-09 11:35, Michael Van Canneyt het geskryf:

And if there was a WYSIWYG editor, the whole point would be moot...

I'm been thinking about a WYSIWIG editor for use with IPF and fpGUI for some time. The idea could be used with fpdoc syntax too. I am actually leaning towards the idea used in Umlet (very cool UML diagram tool). You have an editor window where you type your documentation/markup, and in realtime, you have a preview window of the output. This reducing the chances of wrong markup syntax to zero, and is easier to write that a full "rich edit type" component. Plus you can copy and paste parts of content to other parts without worrying about clipboard richtext formats etc. The markup is simply plain text.

Regards,
- Graeme -

Op 2010-07-09 11:46, Marco van de Voort het geskryf:

Despite my dislike of the limitations of wiki, the solution they have (edit with tags, have an example rendering in a memo below) is easier and more practical I think.

I haven't seen that in wiki's yet, but Umlet does something similar for UML diagrams. Edit with plain text and have a realtime preview of how it will look. This has the benefit that you have clean and correct markup (unlike WYSIWIG editor for HTML that writes the worst HTML code ever).

Regards,
- Graeme -

Op 2010-07-09 11:26, Michael Van Canneyt het geskryf:

In that case, you'll still have problems with < and > (and possibly other) characters, unless you plan to put everything in CDATA sections.

Yes, that would be required if we stay with the XML format, I don't see any other way around it, and I definitely wouldn't want to introduce escaped text inside the documentation content, just to get around the XML format issue. Again, no extra burden should be placed on the documentation writer
- simply writing documentation is already a burden enough. :)

Alternatively (which I believe is not desired - and wasn't my plan either), is to use a totally different file format than XML. I guess for the sake of covering all basis, what do others think of this? Personally, I don't see an issue with staying with XML and simply put all <descr> and <short> content inside a CDATA block. I assume the fcl-xml units know how to handle CDATA blocks?

Regards,
- Graeme -

Obviously...

And:
Not using XML would require a major rewrite of fpdoc - it is deeply rooted in XML, as you'll find out - and I am not very keen on this. So hold your horses there, please...

Michael.

I'd suggest the following:

1) XML stays :-)
2) The fpdoc is modified in a way that, after parsing the initial XML, every text node within <short> or <descr> elements is additionally parsed with another parser.
3) The "another parser" does its job, placing its results into a DOM fragment.
4) The original text node is replaced with the resulting subtree.

5) Choose the new format so it does not use '<' and '&' extensively, because these will have to be escaped.

This way, it will be possible to mix 'old' and 'new' syntax within a single document to any extent, and we won't need to touch the ,
Sergei

I don't quite see why we need to change the storage format? Most of the discussion is about issues applying to editing the content. Especially since a new content format together with the need of reading old files too, will always leead to the need of supporting 2 file formats => that IMHO is the worst solution.

You can have an editor that displays and edits the content in any format you like and validates it at the same time. But it saves it in the current format after all.

so you edit: "[b]key[/b] < 1"
and the editor saves: "<b>key</b> &lt; 1"

You need the translator anyway, for dealing with old files, so you may rather put the translator into the editor.
that allows to offer different editors for different likes of different people.
\bkey\B < 1
*key* < 1
true wysiwyg
....

Martin

Martin wrote

You can have an editor that displays and edits the content in any format you like and validates it at the same time. But it saves it in the current format after all.

The point is we don't have any fpdoc content editor that supports the tags currently listed in the fpdoc documentation. I also more often that not, edit the xml files directly, without any editor. Also not everybody uses Lazarus which has the FPDoc Editor built-in (though that editor also doesn't support rich text editing. I still have to type the extremely verbose syntax (as my simple example showed).

Simplifying the syntax will reduce a lot of all these issues, plus then anybody (even those not using Lazarus) can easily use fpdoc.

As I also mentioned, I have found situations where the current fpdoc content syntax (tags) doesn't support what I want to do. This was due to limitations of the <ul> and <ol> tags in fpdoc. I can't remember my exact example, it was a while back. A simple "near plain text" syntax should reduce such issues too.

In our previous episode, Sergei Gorelkin said:

If I want to edit with tags, it should be possible to do it straight away, and not force the wikiwakkawhatever syntax. This also allows to workaround bugs in the wiki<->html parsers.

I don't understand this, since if I follow these steps, the stored nodes still may contain the wiki syntax, and thus need backend changes?

Or am I wrong and are you suggesting encoding the xml/html to wiki as much as possible before editing, and decoding it to xml/html tags afterwards?

IOW what is the storage format?

should read
and the editor saves: "<b>key</b> &lt; 1"

You need the translator anyway, for dealing with old files, so you may rather put the translator into the editor. that allows to offer different editors for different likes of different people.

ACK

DoDi

In our previous episode, Graeme Geldenhuys said:

everybody uses Lazarus which has the FPDoc Editor built-in (though that editor also doesn't support rich text editing. I still have to type the extremely verbose syntax (as my simple example showed).

I don't think 10% tag overhead in an already short example counts as "extremely verbose", or am I missing something?

But anyway, if editing is all you want, then html2wiki() and wiki2html() would fix it?

The storage format may be called "XML with wiki-formatted textnodes". We keep XML elements for general structure, and optionally use wiki syntax for HTML formatting.

As an example, you type:

<element name='foo'>some *bold* text</element>

When loading it with fpdoc, first the xml parser parses this and creates an element with a textnode. Then it passes content of text node ('some *bold* text') to wiki-or-whatever parser. That parser, in turn, recognizes 'bold' and creates another DOM element:

some <b>bold</b> text

This stuff replaces the content of 'element' tag in the original XML. At this point, if you save the DOM tree, you'll lose all wiki markup and get pure XML.

Of course, you may edit with tags as much as you wish. If the original example already contains 'bold' tag:

<element name='foo'>some <b>bold</b> text</element>

then the wiki parser will be called 3 times (for 'some ', 'bold' and ' text'), but that will be effectively a no-op.

Regards,
Sergei