Hi,

In researching how to type Unicode characters on different platforms, I came across an interesting argument regarding Unicode characters and HTML. The argument might apply to fpdoc documentation (xml) files as well—hence the reason for this post.

With W3C embracing UTF-8 as the de facto standard for HTML pages, do we still need to escape characters like ampersand [' U+2019] to [&] etc. Unicode has been around for some time now, so surely all half-decent software should be able to read and display the actual character correctly by now (sensitive subject for FPC and Delphi at the moment), instead of having to bother with the escaped version.

How does this argument fit with XML which also uses UTF-8 as the de facto standard encoding. And seeing that fpdoc uses XML for the documentation files, can I use the actual Unicode characters in my fpdoc documentation, or must I still stick with the—what now seems to be outdated—escaped method?

BTW:
These are the characters I was interested in.
— (U+2014): emphasis dash
… (U+2026): horizontal ellipses
' (U+2019): right single quotation
" (U+201C): left double quotation
" (U+201D): right double quotation
― (U+2015): quotation dash (introducing quoted text)

Just for completeness, and the curious among you, here are some methods of entering the Unicode characters on different platforms:

Windows (Win2K and up I guess):
ALT+<hex number>

Linux (GTK2 only):
* Press Ctrl+Shift+U, then type the desired hexadecimal code. The benefit of this approach is that you can correct the hexadecimal code in typed wrongly. (I never knew this handy little trick!)
* The Compose key can also be used, but I don't know all the key combinations. The previous option always works.

No idea about the other platforms or desktop managers like KDE.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

Yes, since that is not related to encoding, but because HTML uses & as a control character.

Depends. Is & a steering character in all of XML, or only the xhtml like standards?

It's a special escape character (entity reference) in all XML. But only the &#0000;-style entities and &amp; are defined for plain XML. Entities like "&trade;" "&Uuml;" and the like are specific to the XHTML-DTD.

Vinzent.

I think only XHTML.

But what is fpdoc's xml files? Pure XML, XHTML or some custom/hybrid format? The layout of fpdoc's files seem XML, but the documentation content seems some hybrid HTML - hence the confusion with what is allowed!

Anybody know the rules of strict XML files and Unicode? Can I use Unicode characters as data in XML nodes? I would imagine I may because most well-formed XML files specify UTF-8 as the encoding type.

Also something I think has been resolved in recent versions, but in older 'makeskel' versions, it did not include the encoding in the generated .xml file. So what are we supposed to treat such files encoding as? Default to W3C standards and use assume UTF-8? LCL and fpGUI's fpdoc documentation (mostly) has no encoding specified in the .xml files. FPC's documentation specifies ISO8859-1 as the encoding type, though I found one file (dateutils.xml) it FPC docs that hasn't got an encoding (but my doc update is out of date).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Well, what's the DTD saying about it? ;)

Anybody know the rules of strict XML files and Unicode? Can I use Unicode characters as data in XML nodes?

As long as the encoding doesn't say otherwise, yes.

I would imagine I may because most well-formed XML files specify UTF-8 as the encoding type.

Yes, you can.

Also something I think has been resolved in recent versions, but in older 'makeskel' versions, it did not include the encoding in the generated .xml file. So what are we supposed to treat such files encoding as? Default to W3C standards and use assume UTF-8?

Errmm, yes?

Well, as long as the contents is English, it doesn't matter to much, UTF-8 is fully compatible to 7-bit ASCII. ;)

If you're unsure about the encoding, stick to the &#x0000; unicode entities, that way you can encode anything in 7-bit plain, portable ASCII.

Vinzent.

Errmm???

Then we have a problem.

I just tried that and it failed miserably!

Steps to reproduce:
1. Make sure the fpdoc's .xml file specifies encoding of UTF-8. [By the way I think this encoding gets ignored totally.]
2. Type Unicode characters in any format. Actual or escaped.
3. Generate HTML documentation with fpdoc

Problems:
1. The generated HTML always specifies encoding ISO8859-1! So why bother specifying the encoding in the .xml file??? Is the encoding in the xml file
actually used anywhere?
2. Actual and escaped Unicode characters end up being '??' garbage characters in the generated HTML.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

XML too. In XML, you *must* escape ampersand (U+0026) and less-than sign (U+003C). Also greater-than sign (U+003E) must be escaped if it is preceded by ']]' sequence. Additionally, in attribute values, quotes (U+0022) must be escaped if they are used as value delimiters (other option is to delimit values with apostrophes (U+0027)). Here I mean the XML file, not the DOM tree. You may freely use the mentioned characters in plaintext while manupulating DOM; the writer will escape them on output.

XHTML is XML with defined 'vocabulary' (DTD). These formats have no character-level differences.

W3C demands that XML file without encoding label should be treated as UTF-8 (unless it has an UTF-16 BOM, in which case it should be treated as UTF-16). Therefore UTF-8 labeling is optional.
In older times, makeskel used to write 'ISO8859-1' label, which btw is invalid (IANA recognized names are ISO-8859-1 and ISO_8859-1). Later, when the parser got more compliant, the labeling was removed. The parser has a workaround to understand the ISO8859-1 labeling. The XML writer always produces UTF-8 encoding and writes no label.

To summarize: Unicode can be used in fpdoc xml files. If the file has ISO8859-1 encoding label, it should be removed or replaced with UTF-8 label. The output stages of fpdoc may or may not have problems with Unicode - that requires additional research.

Sergei

No. The input is copied to the output.

You can specify --charset=UTF8 on the command line of fpdoc, that will set the encoding of the generated HTML page, but it does NO conversion whatsoever.

We could maybe have the engine take over the fpdoc specified charset to the generated HTML.

Michael.

On Thu, Aug 14, 2008 at 2:24 PM, Michael Van Canneyt

You can specify --charset=UTF8 on the command line of fpdoc, that will set

OK, but it should be --charset=UTF-8
Thanks.

the encoding of the generated HTML page, but it does NO conversion whatsoever.

Well this is not entirely true then! Something is screwing up my documentation content.

-------- xml file --------------
<?xml version="1.0" encoding="UTF-8"?>
<fpdoc-descriptions>
<package name="CoreLib">

<module name="gfx_UTF8utils">
<short></short>
<descr>Is this character: &#x2026; displayed correctly? </descr>

[....snip....]
------------------end---------------

U+2026 is the Horizontal Ellipses character. The following page has more details and shows the different encodings in HTML etc... http://www.fileformat.info/info/unicode/char/2026/index.htm

--------------------- HTML generated output --------------------------- Overview

Is this character: ? displayed correctly?
----------------end--------------------

Exactly as is! I view the HTML source and the '...' character is not encoded or anything. It became a actual questionmark character.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I forgot the actual HTML...

---------- generated HTML source ----------------------- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> <title>Reference for unit 'gfx_UTF8utils'</title>

[...snip...]

<h2>Overview</h2>
<p>Is this character: ? displayed correctly? </p>
</body>
</html>
--------------------------end----------------------------

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Sergei Gorelkin wrote

To summarize: Unicode can be used in fpdoc xml files. If the file has ISO8859-1 encoding label, it should be removed or replaced with UTF-8 label.

I'll assume this is all in theory then. :-)
See my previous reply. Even if I escaped a Unicode character as follows:

  …

it becomes a literal '?' question mark character in the generated HTML Source output.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Here is another example:

------------part of fpdoc xml file----------------------- <module name="gfx_UTF8utils">
<short></short>
<descr>Is this character: &lt;&#x2026;&gt; displayed correctly? </descr>
-----------------end------------------------

... and the generated html source with UTF-8 encoding ...

------------html source------------------
<h2>Overview</h2>
<p>Is this character: &lt;?&gt; displayed correctly? </p> </body>
</html>
----------------end-----------------------

NOTE:
The < and > characters went through fine, but the ellipses character did not.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

The Document Type Description? Well, I suppose, there is none...

Huh?

Steps to reproduce: 1. Make sure the fpdoc's .xml file specifies encoding of UTF-8. [By the way I think this encoding gets ignored totally.]

Yes, last time I checked, it did.

2. Actual and escaped Unicode characters end up being '??' garbage characters in the generated HTML.

So the entities are probably not resolved correctly. I suppose someone just connected all the text-nodes without bothering to resolve any contained entities. Things like &#x2462; should work regardless of the chosen encoding, as those are unicode-entities.

Vinzent.

I know what DTD means, I meant there is none I know of. :)

I tried encoding it as decimal and hexadecimal notation and it always ends up being '?'. So Michael's theory that the documentation content gets copied as-is is not quite true. My example using &lt;&#x2026;&gt; and becoming &lt;?&gt; in the .html file proves that somewhere fpdoc is doing something with the documentation content.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

Sounds like a browser font issue, rather than an encoding issue.

No, but that was my first thought as well. That is why I view the actual generated HTML file that fpdoc produced. It has a literal ? character in the .html file. I used Midnight Commander's editor and Gnomes gEdit to view the .html file.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Try again with a unicode entity below 256 (like &#xFC; for instance). If that works, the reason is probably there:

-- 8< -- snip -- rtl/inc/wstrings.inc --
procedure DefaultWide2AnsiMove(source:pwidechar;var dest:ansistring;len:SizeInt);
var
  i : SizeInt;
begin
  setlength(dest,len);
  for i:=1 to len do
    begin
      if word(source^)<256 then
        dest[i]:=char(word(source^))
      else
        dest[i]:='?';
      inc(source);
    end;
end;
-- 8< -- snip --

Vinzent.

It looks like the problem is in htmwrite.pp unit. It is AnsiString-based and therefore all Unicode gets simply stripped away and replaced by '?'. Maybe adding cwstring to fpdoc uses clause can make things better for Linux with UTF-8 locale.

Sergei

Indeed.

At the time when fpdoc was written, widestring support was not very complete nor stable, so parts of the fpdoc engine were reworked to ansistring, but this has never be undone. This is not an easy task, and requires some testing.

Michael.

That makes no sense because I used the escaped unicode character format, just like HTML does. So those characters (documentation content) should be copied as-is to the HTML. The following characters interpreted by a Web Browser will display a unicode character, but on their own (as-is), they are valid ASCII characters: &#x2026;

…

should be treated exactly the same as

<    or   >    or   &

when generating HTML output from fpdoc.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Datum: Thu, 14 Aug 2008 16:06:14 +0200
Von: "Graeme Geldenhuys"

That makes no sense because I used the escaped unicode character format, just like HTML does. So those characters (documentation content) should be copied as-is to the HTML.

I suspect those entities get parsed by the DOM-unit as entities (which is the right thing to do generally) and simply get lost in the transformation back to the byte stream (aka. AnsiString) then.

Yes. But the latter have a character code below 256 (even below 128, so they're plain 7-bit ASCII).

Vinzent.

I think you have a point with the DOM-unit parsing the documentation content. So we can safely say, the actual content is NOT copied as-is!

…

If the above was interpreted as-is (with the rest of the content), it would be 8 ascii characters all below 256 character code. No issues then!

So yes I think I agree with you. Somewhere the above is being parsed, then found that as a whole it's above 256 ascii code and simply replaced with a ? character. I simply found this confusing, because Michael is very versed with fpdoc, and when he said the content (not the XML tags) is copied as-is, I would not have envisioned any issues with Unicode escaped character.

Now we know better! :-)
I'll file a bug report in Mantis.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

It is parsed in the xmlreader. The DOM simple contains one single widechar, not these 8 chars. I suspect it is copied as is from the DOM to the output file.

Vincent

One test to do would be to add cwstrings to the fpdoc project and trying again, see what it does.

Michael.

Graeme Geldenhuys wrote

I'll file a bug report in Mantis.

Report as:
http://bugs.freepascal.org/view.php?id=11881

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Michael Van Canneyt wrote

> Now we know better! :-) One test to do would be to add cwstrings to the fpdoc project and trying again, see what it does.

We are not that lucky, I'm afraid. Now we have 3 question marks instead of 1. :-)

--------------HTML output from cwstrings enabled fpdoc--------------------- [...snip...]

<h2>Overview</h2>
<p>Is this character: &lt;???&gt; displayed correctly? </p> </body>
</html>
--------------------------end---------------------------

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Vincent Snijders wrote

It is parsed in the xmlreader. The DOM simple contains one single widechar, not these 8 chars. I suspect it is copied as is from the DOM to the output file.

Correct. The character (numeric) entitity references are expanded upon parsing and replaced by the corresponding WideChar. This corresponds to the W3C XML specification and is not a bug.
The 5 predefined symbolic entities (&lt; &gt; &amp; &apos; &quot;) are treated the same way and expanded immediately, too. Other symbolic entities are not expanded by default (although this can be enabled by setting Options.ExpandEntities to True).

Sergei

Fixed, with the help of Sergei Gorelkin. (bug id 11881)

Michael.