Dear all,

I have an issue using threads on OSX.
Basically I have a thread reading a file and looking for a string match. I have no problem on win and linux but on osx my thread updates the form but my application does not respond.
I opened a topic on the forum
(https://forum.lazarus.freepascal.org/index.php/topic,38146.0.html), I would like a more satisfactory answer. I am trying here.

If possible, I would like to understand if this behaviour is related to OSX or SSD drive. If related to OSX I can simply add IFDEF DARWIN calling sleep?
If it is related to SSD how can I manage it?

Andrea Mauri

James Richters on the fpc-pascal ML commented on SSD commit problems in March. However I don't see why something like that should spill over to the UI.

I notice that multiple people have already told you not to use a tight loop, and that applies to any OS.

What exactly means tight? Is it something that I could measure in some way? The sample test I submitted to the forum is not my real code, that is a little more complex, and it does anything I do. What should I add? Since the same code works on win and linux (and as mattias wrote on the forum also on Mac). But not on my Mac (I can see the label changing but I cannot interact with my form), I supposed that it could be related to SSD drive.
I didn't find a definitive answer.
Anyway what is your suggestion? What should I add? -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170905/c64baa80/attachment.html>

It doesn't have to do with the SSD, this even happens on my old PowerBook which is much less powerful and doesn't even have an SSD (though I still plan to change that, mind you :P ).

It is however solved if you add a "Yield;" after the Synchronize call. So my suspicion is that the scheduling of Mac OS X is somehow messing things up. Either the scheduling is favoring the one thread while it shouldn't (though even reducing the priority doesn't help) or the UI framework of Mac OS X (and whatever else is involved in the main thread) is so heavy weight that the timeslice of the mainthread is used up more often than not thus leading to the read thread being scheduled much more often.
By using Yield (or also Sleep(), but Yield is without any pause) the thread is explicitly giving up its timeslice and that seems to be enough here...

Sidenote: you might want to change your code a bit so that it looks like this:

=== code begin ===

TMyThread = class
private
  fStatusMessage: String;
  procedure DoStatusEvent; // this will be passed to DoSynchronize
  procedure StatusEvent(const aStatus: String); // this will set
fStatusMessage and call Synchronize
  ...
end;
procedure TMyThread.StatusEvent(const aStatus: String);
begin
  fStatusMessage := aStatus;
  Synchronize(@DoStatusEvent);
  //Yield; // <- you can add this here
end;

=== code end ===

This is a bit cleaner as it keeps changes to the status handling in one location. :)

Regards,
Sven

Il 05/09/2017 22:51, Sven Barth via Lazarus ha scritto:

I changed the code following your hints and I added Yield. This is not enough, it seems that the application is a little bit more responding (I can click on Tedit and after seconds the text is highlighted) but not enough to be usable. Actually for me it works only if I add at least Sleep(1).
It is a pity that there is not a unique solution.

Can you try ThreadSwitch after Sleep ?

while (not Terminated) do
    begin
      if eof(f) then
        reset(f);
      ReadLn(f, newStatus);
      if NewStatus  fStatusText then
      begin
        fStatusText := newStatus;
        //Queue(@ShowStatus);
        Synchronize(@Showstatus);
        Sleep(5);
        ThreadSwitch; /// <<<<
      end
      else  // also you need to cover what happens in ANY OTHER outcome of any test, e.g.
        ThreadSwitch;
      // because if NewStatus = fStatusText, you will IMMEDIATELY jump to the first instruction of the thread
      // so adding the ThreadSwitch at the very end of the loop, you give the rest of the application (the main thread) a chance to
      // actually do anything
    end;

Thread.Execute will execute ALL THE TIME if you don't yield/threadswitch in some code path, and WILL saturate your CPU.

-l,

In addition to why this isn't a problem on Windows or Linux: they have supposedly better schedulers...

but if not all codepaths on a thread are covered to give some time to main thread, try to imagine what happens:

- could the un-yielded code path lead to over-zealous enqueuing of the ShowStatus routine?
- if yes, what would happen in such case?

2017-09-06 16:26 GMT+02:00 el es via Lazarus
:

Actually using sleep(1) I can interact with my application. ThreadSwitch does not introduce any change. If I use sleep(1) it works, if I don't use it but I use Yield or ThreadSwitch I cannot interact with my application. -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170906/4cc81d31/attachment.html>

Am 06.09.2017 16:48 schrieb "el es via Lazarus" <

of any test, e.g.

> ThreadSwitch; > // because if NewStatus = fStatusText, you will IMMEDIATELY jump

to the first instruction of the thread

> // so adding the ThreadSwitch at the very end of the loop, you

give the rest of the application (the main thread) a chance to

> // actually do anything > end; > > Thread.Execute will execute ALL THE TIME if you don't

yield/threadswitch in some code path, and WILL saturate your CPU.

supposedly better schedulers...

but if not all codepaths on a thread are covered to give some time to

main thread, try to imagine what happens:

- could the un-yielded code path lead to over-zealous enqueuing of the

ShowStatus routine?

- if yes, what would happen in such case?

Both Synchronize and Sleep already causes a thread switch (the former cause it will wait for the main thread to finish processing). Also I already suggested TThread.Yield which is the same as ThreadSwitch. And as he wrote the Sleep(1) solves the problem for him. It's a pity though that this seems to be necessary on Mac OS X :/

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170906/e68a2c72/attachment.html>

It is neccessary in Windows for me too (but i have the opposite problem : the thread-processing can be a lot shorter - hardware communications - than the time slice of the main thread, giving several Queue() or Synchronize() calls as a result;)
Because of that, I have to use a tthreadlist as a buffer between the thread and the main thread.

The time where I was hunting an elusive problem that did not happen under debugger, is never lost ;)

-l.

Note that his example works on my Mac with or without sleep. Maybe if Andrea posts his full example we may find out more.

Mattias

Unfortunately I cannot post the real case, it is included in a bigger unit involving other stuffs.
Anyway the example I posted is quite the same.

In order to better clarify my issue:
- I have to manage text files including some logical blocks (items), these files can be huge (gigabytes) and they can store millions of items (some file types define an item using a single line). My application deals with these files enabling the user to visualize the items in a grid and rapidly move along the file.
I encounter it in two different threads:
1. a thread that I launch when the user load one of these files. This thread simply reads the file, counts the items and stores the position of the items (streambuffer positions).
2. a thread reading a file and looking for a substring.

1. source

procedure TCountItemsThread.Execute;
var
   streamreader: TStreamReader;
   streampos, streamsize: Int64;
   bufferpos, nitems: Integer;
begin
   _itemfile:= TItemsFile.Create(_filename, _fileformat);
   try
     streamreader:= _itemfile._streamreader;
     streamreader.Reset;
     streamsize:= streamreader.BaseStream.Size;
     _itempos.Clear;
     _itembufpos.Clear;
     nitems:= 0;
     while not Terminated and not streamreader.Eof do
     begin
       // stores items source position in file as Position and
BufferPosition
       streampos:= streamreader.Position;
       if streamreader.BufferRead < (BUFFER_SIZE - 1) then
         streampos:= streampos - (streamreader.BufferRead)
       else
         streampos:= streampos - (BUFFER_SIZE - 1);
       bufferpos:= streamreader.BufferPosition;
       if (bufferpos = 0) and (streampos > 0 ) then
       begin
         streampos:= streampos - (BUFFER_SIZE - 1);
         bufferpos:= BUFFER_SIZE - 1;
       end;
       if _itemsfile.nextItem then begin
         _itempos.Add(streampos);
         _itembufpos.Add(bufferpos);
         _progress:= round((streampos / streamsize) * 100);
         Inc(nitems);
         if (_itempos.Count mod 1000) = 0 then begin
           Synchronize(@CountingItems);
         end;
       end;
     end;
     Synchronize(@EndCountItems);
   finally
     _itemfile.Free;
   end;
end;

nextItem simply parse the file and move the stream to the next item position. If the file has single line items I cannot interact with my app even on win.

2. source

procedure TFilterNameThread.Execute;
var
   substr, str1: string;
   i, step, occurrences: Integer;
   clonedfile: TItemFile;
begin
   // if substr is empty do not perform search
   FFindItems.Size:= FItemFile.CountItems;
   if Trim(FindString) = '' then begin
     for i:= 0 to FItemFile.CountItems - 1 do begin
       FFindItems[i]:= True;
     end;
     _itemidx:= FItemFile.CountItems;
     Synchronize(@ShowProgress);
     Exit;
   end;
   // store original FItemFile.Selected[] and sets FItemFile.Selected[]
to False
   _itemidx:= 0;
   FFindItems.Clearall;
   Synchronize(@ShowProgress);
   clonedfile:= FItemFile.CloneFile;
   try
     step:= clonedfile.CountItems div 100;
     if step = 0 then
       step:= 1;
     substr:= FindString;
     if not CaseSensitive then
       substr:= upcase(substr);
     occurrences:= 0;
     while (not Terminated) and (_itemidx < clonedfile.CountItems) do begin
       if CheckAll or Selected[_itemidx] then
       begin
         str1:= clonedfile.ItemName[_itemidx];
         if not CaseSensitive then
           str1:= upcase(str1);
         if Pos(substr, str1) > 0 then begin
           FFindItems[_itemidx]:= True;
           Synchronize(@ShowProgress);
           Inc(occurrences);
         end;
       end;
       if (_itemidx mod step) = (step - 1) then begin
         Sleep(5);
         Synchronize(@ShowProgress);
       end;
       Inc(_itemidx);
     end;
     Synchronize(@ShowProgress);
   finally
     clonedfile.Free;
   end;
end;

Am 06.09.2017 17:36 schrieb "Mattias Gaertner via Lazarus" <

cause

Maybe your file is not suitable? I used a /var/log/messages.log file from one of my Linux computers and with that I can reproduce it with the given code.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170906/f45dc907/attachment.html>

2017-09-06 19:34 GMT+02:00 Sven Barth via Lazarus <

reported on the forum with this one.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170906/585adaa4/attachment-0001.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: nci_download_red.smi
Type: application/smil
Size: 1172 bytes
Desc: not available
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170906/585adaa4/attachment-0001.bin>

Il 06/09/2017 19:45, Andrea Mauri ha scritto:

As far as I understood. This behaviour is quite unpredictable, the thread seems to work properly or not with different solutions. Sometimes it works on osx sometimes not.
I have to admit that I would prefer that it was a bug or an error in my code.
Do you have any suggestion to improve my code (even with ifdef) in order to be able to deploy an application that works on lin/win/osx?