Hi,

I wrote this console utility app in 5 minutes. All it is supposed to do, is iterate over *.jpg images (65 at the moment) in a folder (no recursive searches), and generate a snipped of HTML that I need to paste into my wives website.

I first thought my utility app is in some recursive loop or something because the output didn't appear instantly - like I thought it would.

I added some debug output between the generate HTML. To my surprise the FPImage code reading a 35Kb jpg file takes +- 8 seconds and a 65KB file takes +- 30 seconds - on a Quad Core machine! WTF???

Is this normal? Is FPImage really that slow reading small roughly 650x450 pixel jpg images? Or am I doing something unbelievably stupid? :-/

The reason I'm reading in the jpg files is so I can get their Width and Height values. Avg. jpeg file sizes are between 35-45KB, and largest file size is 65KB.

Here is the relevant code snippet....

var
  imgreader: TFPReaderJPEG;
  stream: TFileStream;
  img:TFPCustomImage;

....

  writeln('processing file: ', filelist[i]);
  stream := TFileStream.Create(filelist[i], fmOpenRead);
  writeln(1);
  imgreader := TFPReaderJPEG.Create;
  writeln(2, FormatDateTime(' hh:mm:ss.zz', Now) + '  (start reading)');
  img := imgreader.ImageRead(stream, nil);
  writeln(3, FormatDateTime(' hh:mm:ss.zz', Now) + '  (done)');

Here is the program output... Note the timestamps between start reading and done reading (lines marked with '2' and '3').

-------------------[ start ]--------------------------- $ ./project1 img/babies/full
Finished search. Found 65 matches
---------------------------------------
<div id="thumbnails">
<table cellspacing="10">
<tr align="center" valign="middel"> <!-- Row starts --> <td>
processing file: img/babies/full/babies.01.jpg
1
2 16:00:30.210 (start reading)
3 16:00:38.338 (done)
<img src="img/babies/full/babies.01.jpg" alt="" id="screen1" class="imglarge" onmouseout="reducethumb(1); return false;" onclick="reducethumb(1); return false;">
<img src="img/babies/thumbs/babies.01.jpg" alt="" class="imgsmall" id="thumb1" onclick="expandthumb(1, 650, 413);">
</td>
</tr>
<tr align="center" valign="middel"> <!-- Row starts --> <td>
processing file: img/babies/full/babies.02.jpg
1
2 16:00:38.338 (start reading)
3 16:01:10.447 (done)
<img src="img/babies/full/babies.02.jpg" alt="" id="screen2" class="imglarge" onmouseout="reducethumb(2); return false;" onclick="reducethumb(2); return false;">
<img src="img/babies/thumbs/babies.02.jpg" alt="" class="imgsmall" id="thumb2" onclick="expandthumb(2, 650, 435);">
</td>
</tr>
<tr align="center" valign="middel"> <!-- Row starts --> <td>
processing file: img/babies/full/babies.03.jpg
1
2 16:01:10.447 (start reading)
3 16:01:28.859 (done)
<img src="img/babies/full/babies.03.jpg" alt="" id="screen3" class="imglarge" onmouseout="reducethumb(3); return false;" onclick="reducethumb(3); return false;">
<img src="img/babies/thumbs/babies.03.jpg" alt="" class="imgsmall" id="thumb3" onclick="expandthumb(3, 405, 650);">
</td>
</tr>
<tr align="center" valign="middel"> <!-- Row starts --> <td>
processing file: img/babies/full/babies.04.jpg
1
2 16:01:28.859 (start reading)

....snip....
-------------------[ end ]---------------------------

Try the following code:

Img:=TFPMemoryImage.Create(0,0);
Img.UsePalette:=False;
Img.loadFromFile(FileList[i]);

That should work much faster. The reason is probably that the default image created by the reader uses a palette, which is of course dog slow when reading large images.

Michael.

Michael wrote

....snip.... That should work much faster. The reason is probably that the default image

Ah, thanks Michael.... that did the trick. I knew I had to be doing something wrong.

Interesting. What is supposed to be a "large image"? I ask because the original post mentioned 35 kB and 65 kB (and indirectly also dimensions around 600x400 if I understand the program output correctly) which I certainly wouldn't consider as "large" (and the mentioned time necessary for reading it surely not appropriate regardless of using palette or not). I guess that some optimization may be reasonable because such pictures could be loaded and displayed (including conversion from true colour to 256 colours palette) faster even on a 486 machine...

Tomas

It depends on the colors used. The default palette algorithm simply allocates a slot per found color in the jpg and returns the index of the slow. For a picture, this will create a huge palette, in which the search for a color is in essence linear, so n^2 lookups.

So, dog slow.

You could obviously speed this up by creating a palette yourself, and limiting the number of colors.

Or, best for pictures is not to use a palette, which is what my solution entails.

Michael.

Interesting. What is supposed to be a "large image"? I ask because the original post mentioned 35 kB and 65 kB (and indirectly also dimensions around 600x400 if I understand the program output correctly) which I

I suppose he was referring to the "color palette", and not the physical image size. I guess that is why The Gimp limits png images to 256 colors when I saved it as Indexed (which then generates a color palette). So I guess it is a standard issue with Index/Palette images.... thus limit your colors, or don't use a palette.

I guess one change in FPImage (depending on how often this occurs obviously), could be to default to UsePalette = False instead.

Tomas Hajny hat am 30. Mai 2011 um 16:50 geschrieben:

Is there any size, where a palette image is fast?

I think this is a newbie trap and the default for UsePalette should be changed.

Mattias

OK, I see. The point is not about what I need or not (I don't need to work with pictures at the moment) but about what will happen if people start using the current code as it is provided in FCL. I'm sure there are various possibilities for some "optimization" - should it be using a different (faster) algorithm for creating the palette (e.g. different algorithms depending on whether the original picture contains a palette or not), not creating the palette by default for pictures not using it (or not doing it at all unless explicitly requested, i.e. changing the default behaviour as suggested by Matthias), limiting the maximum colour depth for pictures with palette (as mentioned by Ben), etc. In any case, I'm afraid that the current default behaviour will not work well for majority of FPC users and their use cases.

Tomas

Pardon me for jumping in, but is there an equivalent fix for slow operations in a TLazIntfImage?

It depends on the colors used. The default palette algorithm simply allocates a slot per found color in the jpg and returns the index of the slow. For a picture, this will create a huge palette, in which the search for a color is in essence linear, so n^2 lookups.

Isn't it possible to automatically disable the palette automatically for images with a bit depth of 16 bits or more? I'm not aware of any image format that uses palettes at those bit depths (and even if there are such formats, they are at least not common in general use cases).

Jonas

In our previous episode, Mattias Gaertner said:

Not just newbies :-)

http://bugs.freepascal.org/view.php?id=17621

Marco van de Voort wrote

http://bugs.freepascal.org/view.php?id=17621

Indeed, I was remembering that too =)

And also, I have a program which loads a huge image (20MB) using TPortableNetworkGraphics and it takes minutes to load the image. I have simply supposed that it was because of the file system usage, but now I am starting to think that maybe the fcl-image pallete causes the slowness.

So I have to suggest: Is there a convincing reason for having Pallete set to true by default? What do you think about the trivial solution of setting it off by default?

Felipe Monteiro de Carvalho wrote

So I have to suggest: Is there a convincing reason for having Pallete set to true by default? What do you think about the trivial solution of setting it off by default?

It seems I have opened a hornets nest. :) But from the many replies, it seems everybody expects UsePalette to be False by default (instead of what it is now True). Maybe a new default should be considered.

It is now off by default.

Committed in revision 17613.

Michael.

Great. :)

Mattias