Hello

In revision 33401 of FPC subversion, a lot of fixes have been committed for Font handling in the PDF generator: e.g. Unicode fonts should now render correctly.

The test program generates the following PDF:

http://www.freepascal.org/~michael/test.pdf

Page 1 looks as follows on my system:

http://www.freepascal.org/~michael/pdftest.png

The output has been tested for correct rendering in the following applications:

Firefox native viewer (Linux)
Acrobat Reader 10 (Windows)
Foxit Reader (Windows)
Xpdf (FreeBSD)
Evince (FreeBSD)
The GIMP (FreeBSD)
Google Drive viewer
LibreOffice Draw (linux)
Okular (linux)

The file was also generated on various platforms to make sure all platforms handle the generating correctly (locales, code pages etc.).

If you find any more errors (let's hope not), please let me know.

Enjoy,

Michael.

Very good. Thanks.
Red line left margin has problem in Google chrome default PDF viewer, maybe it is Chrome problem but I think you should know.

Regards,
Ara

On Thu, Mar 31, 2016 at 11:02 AM, Michael Van Canneyt <

[...]

Great news! :-)

I'm going to re-test it ...

Finally the PDF was succefully generated on Windows! \o/

I have some observations:

. The demo print a lot of lines if the font is not found:

Exception at 0041F4B7: EFOpenError:
Unable to open file "fonts\FreeSans.ttf".
Exception at 0041F4B7: EFOpenError:
Unable to open file "fonts\FreeSans.ttf".
Exception at 0041F4B7: EFOpenError:
Unable to open file "fonts\FreeSans.ttf".
Exception at 0041F4B7: EFOpenError:
Unable to open file "fonts\FreeSans.ttf".
... infinitely ...

. The PDF was generated with some wrong chars (you can download it here: https://dl.dropboxusercontent.com/u/135304375/test.pdf):

Greek: Ge?? s?? ??s
Polish: Witaj swiecie
Portuguese: Ol? mundo
Russian: ???????????? ???
Vietnamese: Xin ch?o th? gi?i

In the previous fppdf version, the PDF was generated about ~650KB, after update it I get a PDF ~381KB. Is it OK?

The attached patch remove some compile hints/warnings.

That is logical, if it cannot find the file which it tries to embed ??

No, I have one of ~1Mb. But probably logical if the file is not there ?

It's not quite clear what to think of your report ?

For me

"PDF was succefully generated"

does not compute with

"Unable to open file "fonts\FreeSans.ttf".

:-)

Michael.

Please, don't do {%H-} etc:

procedure CreateTTFCIDSystemInfo(const {%H-}EmbeddedFontNum: integer;{%H-}FontDef: TFontDef);virtual;

Typecasts like this should also not be done:

+    FontDef.FDiffs := WideString(lFontDef.Diffs);
+    FontDef.FCharWidth := WideString(lFontDef.CharWidths);
-  Arr.AddIntArray(FontDef.FCharWidth);
+  Arr.AddIntArray(string(FontDef.FCharWidth));

We need to investigate why you think this typecast is needed.

The buffer initialization is good, though.

I will not apply this patch, this needs deeper investigation.

Michael.

I believe the loop is caused by the usage of TCustomApplication (the previous test app didn't use TCustomApplication).

And yes, the demo requires that TTF file otherwise the text output will not work.

> . The PDF was generated with some wrong chars (you can download it here: > https://dl.dropboxusercontent.com/u/135304375/test.pdf): That is logical, if it cannot find the file which it tries to embed ??

I had a look at your PDF. I did not extract the embedded FreeSans font you have embedded, so don't know the font file version, but based on its size (264KB) it is a very old FreeSans.ttf file. The one I tested with is 1.56MB in size.

If I must, I can extract your embedded FreeSans font to query it further and see the version and glyph count it defines.

> Greek: Ge?? s?? ??s > Polish: Witaj swiecie

This tells me your version of FreeSans.ttf doesn't have all the glyphs the demo uses.

Indeed. :)

Regards,
- Graeme -

Michael Van Canneyt wrote

- Arr.AddIntArray(FontDef.FCharWidth); + Arr.AddIntArray(string(FontDef.FCharWidth)); We need to investigate why you think this typecast is needed.

Plus the fact that I haven't complete all the code clean-up yet. As far as I can see all the TFontDef related code is legacy - related to when the pdf code still used *.fnt files, instead of querying *.ttf files directly. What does depend on the TFontDef structure can easily be replaced by finding the info directly from the TTF file.

The TTextDictionary class can be removed as well. That has been replaced by TTextMappingList & TTextMapping classes.

As you might know [Michael], my last concern was compiler hints. ;-)

Regards,
- Graeme -

A quick question (sorry I haven't tested): is word breaking supported and is it possible to get the target rect of multiline and singleline text?

Ondrej

Ondrej Pokorny wrote

A quick question (sorry I haven't tested): is word breaking supported and is it possible to get the target rect of multiline and singleline text?

It is possible to get a target rect of a single line; unit fpttf:

function TextWidth(AStr: string; APointSize: single): single;

Word breaking is left to the calling application.
A reporting tool is in the make (well, made, but not released yet) which will handle this.

Michael.

Great, thank you!

And is it possible the other way round? I.e. I know the target rect and I have to export a single line exactly into that rect? (The text has to be scaled to fill the rect.)

Why I ask: I need to print my PDFs directly from a Lazarus application. I have to render the page on a canvas (monitor for preview, printer for printing) and create a PDF from the same source. All three targets (monitor, printer, PDF) have to be exactly equal.
I achieved this in Delphi with Synopse PDF. Now I'd like to port my reporting solution to Lazarus.

Ondrej

I suspect this can be easily implemented.

Synopse pdf is the reason I started fppdf. Synopse is heavily Windows centric, but otherwise quite capable.

Fppdf is a young library (although it required many months of work) I am sure many improvements will be made as the reporting library develops.

A canvas descendant that renders to pdf is planned, but not very high priority.

Patches and suggestions are definitely welcome. The library is intended for production, so quality is important.

Michael.

TPDFPage has some basic primitives for drawing: lines, rects, ellipses and images. I don't see a way to add our own (except by patches to fpPDF of course) but it would be nice to have some mechanism for extension, for example if a TPdfPageClass=class of TPdfClass is defined, then there could be a TPdfPages.AddPage overloaded with an extra argument PageClass:TPdfPageClass, and this could be used for implementing additional primitives? what you hink?

Regards.

Jesus Reyes A.

Jesus Reyes A. wrote

TPDFPage has some basic primitives for drawing: lines, rects, ellipses and images. I don't see a way to add our own (except by patches to fpPDF of course)

I like the idea of giving developers the ability to extend TPDFPage with more drawing functions. Good news is, it is quite possible already, with one minor modifications - TPDFPages needs to surface a Add(APage: TPDFPage) procedure.

I just made the change locally and managed to add my own drawing function as follows:

type
  // page class with new custom drawing methods
  TMyPDFPage = class(TPDFPage)
  public
    procedure DrawRoundedRect(...);
  end;
  // Add your custom page as follows
  P := TMyPDFPage.Create(Doc);
  Doc.Pages.Add(P); // global Page Object list
  lSection.AddPage(P); // which Section our page belongs too
  // painting our custom page.  P is a TMyPDFPage type.
  P := TMyPDFPage(FDoc.Pages[5]); // get our page reference
  { call a custom draw function from our custom class }
  P.DrawRoundedRect(...);

I'll send the changes to Michael so he can update fcl-pdf.

Regards,
- Graeme -

En Thu, 31 Mar 2016 18:16:18 -0600, Graeme Geldenhuys escribi?:

Yes that would suffice, thanks.

Jesus Reyes A.

Michael Van Canneyt wrote

I suspect this can be easily implemented. Synopse pdf is the reason I started fppdf. Synopse is heavily Windows centric, but otherwise quite capable.

Yes, correct. I was who implemented exact drawing to Synopse PDF (i.e. rendering to exact rect) because the font sizes on Canvas and PDF were quite different. IIRC you compare the actual text width with the wanted text width and you scale the text box with a transformation. I also implemented underline text and did more enhancements, I don't remember exactly any more. It was years ago.

Now I stay before a choice to use Synopse PDF (and to have PDF export Windows-only) or create a new bindings to fppdf.

I have to take a 'll need a PDF export in the next months. If I decide for fppdf you can be sure that I'll make you busy with sending patches :)

Ondrej

Coordinate transformations are explicitly part of the PDF generator design.

Well, obviously I cannot make this choice for you. For me there was no choice: my software runs on headless linux servers. So maybe you will profit from my lack of choice :-)

No worries there, they will be very welcome :)

Michael.

On Thu, 31 Mar 2016 18:16:18 -0600, Graeme Geldenhuys wrote:

BTW, out of curiosity, in the sample there is code like this:

   lPt1 := P.Matrix.Transform(15, 120);
   P.WriteUTF8Text(lPt1.X, lPt1.Y, 'Languages: English: Hello, World!');
   lPt1 := P.Matrix.Transform(40, 130);
   P.WriteUTF8Text(lPt1.X, lPt1.Y, 'Greek: -F???? ??? ??????');
   lPt1 := P.Matrix.Transform(40, 140);
   P.WriteUTF8Text(lPt1.X, lPt1.Y, 'Polish: Witaj -B?wiecie');
   lPt1 := P.Matrix.Transform(40, 150);
   P.WriteUTF8Text(lPt1.X, lPt1.Y, 'Portuguese: Ol-A? mundo');

and then WriteUTF8Text and practically every other drawing function does something like this:

   p1.X := X;
   p1.Y := Y;
   DoUnitConversion(p1);

The question is why isn't the transformation matrix "active"?,

I mean the code above would be simply:

   p.SetTransformationMatrix(paramst o adjust the matrix) // or ....
   p.Matrix.SetTransformationMatrix(params to adjust the matrix);

(it seems currently is not necessary because the matrix is auto-adjusted) and then ...

   P.WriteUTF8Text(15, 120, 'Languages: English: Hello, World!');
   P.WriteUTF8Text(40, 130, 'Greek: -F???? ??? ??????');
   P.WriteUTF8Text(40, 140, 'Polish: Witaj -B?wiecie');
   P.WriteUTF8Text(40, 150, 'Portuguese: Ol-A? mundo');

Of course, before DoUnitConversion(p1); there should be a DoMatrixTransform(p1); etc.

One thought is that coordinates could be reused and in such case it might help but it doesn't seem to be a very strong reason, is interesting to know the real reason.

Jesus Reyes A.

That does indeed look much better. The origin of the external matrix usage is quite simple really... Initially there wasn't a matrix, and coordinates had to be calculated (PDF's origin point is bottom/left of the page) manually before passed to drawing routines. Introducing Matrix was the first step in cleaning up the code.

Your recommendation is the next step and already on my todo list. I'll make the change and adapt the demos. This will affect the fpReport code too (hence it wasn't done yet), but the changes really aren't that big. I'll make the change in this coming week.

Regards,
- Graeme -

On Thu, Mar 31, 2016 at 1:13 PM, Michael Van Canneyt <

wrote:

[...]

Oops, I meant: "PDF was succefully generated but with some issues". :-D

On Thu, Mar 31, 2016 at 1:26 PM, Graeme Geldenhuys <

wrote:

[...]

Hm... I'll try Arial instead of FreeSans.

On Thu, Mar 31, 2016 at 1:17 PM, Michael Van Canneyt <

wrote:

[...]

{$HINTS OFF} instead of it?

Because the FontDef.FCharWidth is a WideString.

- Arr.AddIntArray(FontDef.FCharWidth); + Arr.AddIntArray(string(FontDef.FCharWidth));

Because Arr.AddIntArray() is AnsiString (string) and FontDef.FCharWidth WideString.

We need to investigate why you think this typecast is needed.

It just stop hints/warnings from compiler.

The buffer initialization is good, though. I will not apply this patch, this needs deeper investigation.

It just remove unused variables and apply some cast fixes.

On Thu, Mar 31, 2016 at 1:13 PM, Michael Van Canneyt <

This attached patch definitely fixes this error on Windows. :-)

Oops, I forgot the test using the attached patch result:

https://dl.dropboxusercontent.com/u/135304375/test.pdf

It was generated on Windows 7 (using Arial instead of FreeSans).

Thanks for this awesome and promising library!

Why is this patch needed ? It should not be needed at all ?

In each case it will not be applied, since it assumes use of lazarus, and that is not acceptable in the FCL.

Michael.

silvioprog wrote

This attached patch definitely fixes this error on Windows. :-)

This was already fixed (plus other changes). I simply forgot to send Michael the email yesterday to update FPC.

Regards,
- Graeme -

No, fix the hint (if possible).

The correct thing to do is to make sure the types are compatible. What you do is hide the real issue.

But we will .

This change has been made and will be available in FPC repo shortly. Incidently, this drastically reduces the amount of application code and makes it much neater. Something I like. :)

Regards,
- Graeme -

Patch applied. Rev. 33428.

Michael.

As I "hinted" (excuse the pun) at before, any reference at TFontDef will be removed soon. It isn't needed any more.

Regards,
- Graeme -

Michael Van Canneyt wrote

Patch applied. Rev. 33428.

Thanks Michael.

Regards,
- Graeme -

This change has now been made.

Regards,
- Graeme -

On Wed, Apr 6, 2016 at 2:14 PM, Michael Van Canneyt wrote:
[...]

Why is this patch needed ? It should not be needed at all ?

Sorry, I sent a wrong patch, please consider this new one in attachment.

My patch just fix wrong chars in the generated PDF, eg, before the apply it, I got:

...
English: Hello, World!
Greek: Ge?? s?? ??s
Polish: Witaj swiecie
Portuguese: Ol? mundo
...

After apply it:

...
Languages: English: Hello, World!
Greek: ???? ??? ??????
Polish: Witaj ?wiecie
Portuguese: Ol? mundo
...

I needed to apply it even after update my FPC to rev. 33428.

In each case it will not be applied, since it assumes use of lazarus, and

that is not acceptable in the FCL.

Indeed.

ps. about the hints/warnings, see the compiler log below:

Compile Project, Target: testfppdf.exe: Success, Warnings: 3, Hints: 4
fppdf.pp(1049,28) Hint: Local variable "Buffer" does not seem to be
initialized
fppdf.pp(1028,3) Note: Local variable "I" not used
fppdf.pp(3194,31) Warning: Implicit string type conversion from
"AnsiString" to "WideString"
fppdf.pp(3195,35) Warning: Implicit string type conversion from
"AnsiString" to "WideString"
fppdf.pp(808,44) Hint: Parameter "EmbeddedFontNum" not used
fppdf.pp(808,70) Hint: Parameter "FontDef" not used
fppdf.pp(3308,37) Warning: Implicit string type conversion with potential
data loss from "WideString" to "AnsiString"

I see. I don't understand why this patch fixes it for you.

Because it means that somewhere a conversion happens that should not happen.

Maybe one of these string conversion warnings are the cause of the error. I suspect the last one, actually, line 3308.

The other ones will be fixed too, of course.

Michael.

;)

Thanks.

Jesus Reyes A.

En Wed, 06 Apr 2016 13:14:49 -0500, Michael Van Canneyt escribi?:

Here it fixes the problem too. So I did a small investigation and this is what I found:

The problem starts with this code:

procedure TPDFPage.AddTextToLookupLists(AText: UTF8String);
var
   str: UnicodeString;
begin
   if AText = '' then
     Exit;
   str := UTF8ToUTF16(AText);
   Document.Fonts[FFontIndex].AddTextToMappingList(str);
end;

AText (a CP_UTF8 tagged string) is passed away to UTF8ToUTF16(AText) which expects a mere and mundane ansistring (to be used later as a pchar), the assembler window shows at what point the conversion is attempted:

C:\ThePathTo\fpctrunk\packages\fcl-pdf\src\fppdf.pp:1583 str := UTF8ToUTF16(AText);
00435974 8b45fc mov -0x4(%ebp),%eax 00435977 8d4dc8 lea -0x38(%ebp),%ecx 0043597A 66ba0000 mov $0x0,%dx
0043597E e80d3dfdff call 0x409690 <fpc_ansistr_to_ansistr> 00435983 8b45c8 mov -0x38(%ebp),%eax 00435986 8d55f4 lea -0xc(%ebp),%edx 00435989 e8a2860000 call 0x43e030 <UTF8TOUTF16>

fpc_ansistr_to_ansistr converts AText from the given UTF8String to ansistring via RawbyteString. And it converts it to whatever DefaultSystemCodePage says it should. Now this is a problem because in Windows and according to the wiki this value is "The result of the GetACP OS call, which returns the Windows ANSI code page". In my case, and I guess Silvio's too, DefaultSystemCodePage=1252 not CP_UTF8, so in our case if AText is 'Greek: ???? ??? ??????' there will be problems converting that to CodePage=1252 which is solved by showing the "?" in the problematic characters

the SetMultiByteConversionCodePage(CP_UTF8) call makes DefaultSystemCodePage=CP_UTF8 which matches UTF8String and so in fpc_ansistr_to_ansistr no conversion is performed.

And so that is why SetMultiByteConversionCodePage(CP_UTF8) is needed when compiling in windows....
:)

Jesus Reyes A.

happen.

> Here it fixes the problem too. So I did a small investigation and this is

what I found:

which expects a mere and mundane ansistring (to be used later as a pchar), the assembler window shows at what point the conversion is attempted:

C:\ThePathTo\fpctrunk\packages\fcl-pdf\src\fppdf.pp:1583 str :=

UTF8ToUTF16(AText);

ansistring via RawbyteString. And it converts it to whatever DefaultSystemCodePage says it should. Now this is a problem because in Windows and according to the wiki this value is "The result of the GetACP OS call, which returns the Windows ANSI code page". In my case, and I guess Silvio's too, DefaultSystemCodePage=1252 not CP_UTF8, so in our case if AText is 'Greek: ???? ??? ??????' there will be problems converting that to CodePage=1252 which is solved by showing the "?" in the problematic characters

the SetMultiByteConversionCodePage(CP_UTF8) call makes

DefaultSystemCodePage=CP_UTF8 which matches UTF8String and so in fpc_ansistr_to_ansistr no conversion is performed.

And so that is why SetMultiByteConversionCodePage(CP_UTF8) is needed when

compiling in windows....

:)

UTF8ToUTF16 should best take a UTF8String then. It would fit the purpose of the function better anyway...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160407/8bb83b74/attachment.html>

This should not exist to begin with, I think.
The UTF8Decode function of the system unit performs the same function.

Jesus, thank you for looking into this, we'll get to work with it !

Michael.

Michael Van Canneyt wrote

This should not exist to begin with, I think. The UTF8Decode function of the system unit performs the same function.

Indeed, you are correct. I'll make the change.

Regards,
- Graeme -

Just thought I would mention, I've seen the above code listed a few times now to remove the famous "Local variable does not seem to be initialized" compiler hint.

Well, it might work on FPC 3.0 (not test), but it does nothing for FPC 2.6.4 - the hint is always there. I know it is a harmless hint, if the code was correct to start work.

Either way, I found a solution for that, which I used in fpGUI's DocView help viewer application too.

Implement the following:

procedure FillMem(Dest: pointer; Size: longint; Data: Byte );
begin
  FillChar(Dest^, Size, Data);
end;

Then write your code as follows:

  FillMem(@Buffer, SizeOf(TBuffer), 0);

Now the compiler hint is gone under both FPC 2.6.4 and FPC 3.0 - and no ugly IFDEF's required. And as a bonus the name FillMem() is more descriptive and technically accurate of what the code does than FillChar()

Regards,
- Graeme -

You will get the hint here then, no ?

Michael.

Michael Van Canneyt wrote

You will get the hint here then, no ?

No. Seems the compiler is more lenient with pointer types.

Regards,
- Graeme -

> You will get the hint here then, no ? No. Seems the compiler is more lenient with pointer types.

That's a bug then, isn't it? After all, it's about the var parameter, not what is passed into it... The variable that is passed into FillChar is not yet initialized, but it could be read inside FillChar, which is why the compiler nags.

If Move+FillChar would use out instead of var (as they should - they don't read the dest value, that's the whole point), that would be fixed once and for all. But I remember having this discussion before and apparently it was like this for some compatibility reason...

There are a few places in API imports as well where a "_Out_ FOO * pBar" was imported as "var bar: FOO" but it is actually only used for output, with the same technically useless hints. But that's getting off-topic, sorry.

Regards,
Martok

Martok wrote

If Move+FillChar would use out instead of var (as they should - they don't read the dest value, that's the whole point), that would be fixed once and for all. But I remember having this discussion before and apparently it was like this for some compatibility reason...

I was about to mention that. This was discussed before, and there was a reason (which eludes me now) why FillChar() will not be changed. And that is also when I came up with the FillMem() solution.

If FPC starts nagging about uninitialised pointer parameters (as in the usage by FillMem(...) ), then I am afraid there is *no solution* to get rid of that compiler hint for FPC 2.6.4.

I haven't looked at what FPC 3.0's Default() function does yet, but maybe FPC could implement a FillMem() exactly like FillChar() but with an out parameter instead. So then FillChar() is there for whatever backward compatibility, and FillMem() could be used going forward to get rid of that compiler hint. Then again, I probably made this suggestion in our previous discussions on this topic. ;-)

Regards,
- Graeme -

Am 07.04.2016 13:49 schrieb "Graeme Geldenhuys" <

: On 2016-04-07 12:25, Martok wrote: > If Move+FillChar would use out instead of var (as they should - they

don't read

> the dest value, that's the whole point), that would be fixed once and

for all.

> But I remember having this discussion before and apparently it was like

this for

"out" and "var" behave different in rather subtle points and thus code that would currently work with FillChar would no longer work then or have subtle errors.

If FPC starts nagging about uninitialised pointer parameters (as in the usage by FillMem(...) ), then I am afraid there is *no solution* to get rid of that compiler hint for FPC 2.6.4.

There is no warning/hint/whatever because a parameter is always assumed to be initialized (except it's am "out" one).

Default() essentially creates a variable of the given type and sets it to zero (there are optimizations for simple types).

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160407/d23a829f/attachment-0001.html>

Graeme Geldenhuys wrote

I was about to mention that. This was discussed before, and there was a reason (which eludes me now) why FillChar() will not be changed.

Because out parameters are finalised on caller side:

"
Procedure FillChar1(out x;count:SizeInt;Value:Byte);
begin
 fillchar(x,count,value);
end;
procedure tmainfo.exe(const sender: TObject);
type
 testty = record
  a: int32;
  b: string;
 end;
 ptestty = ^testty;
var
 po1: ptestty;
begin
 getmem(po1,sizeof(testty));
 pointer(po1^.b):= pointer(123467); //random value
// fillchar(po1^,sizeof(testty),0); //OK
 fillchar1(po1^,sizeof(testty),0); //crash because out paramters are
                                   //finalized on caller side
 po1^.b:= 'abc';
 finalize(po1^);
 freemem(po1);
end;
"

Martin

Ah, yes, that was the subtle point I mentioned :)

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160407/126eefe2/attachment-0001.html>