In my data acquisition program I also need to timestamp the retrieved data from the instrument. I want this to be as accurate as possible and also portable Windows->Linux->Embedded Linux

In Delphi/FPC there are two ways to get a time value that I know of:

GetTickCount gives a value that is in millisecond resolution, but I don't know how accurate it actually is on the various platforms.

Now() retrieves a TDateTime value, but again I don't know how accurate it is. Resolution-wise since it is a double it should suffice since it is based on the year 1900 and we have about 40500 days since then and thus about 3.500.000.000 seconds. So the double resolution should get a bit into the decimals of a second. But how many accurate decimals can be expected? Can I reach millisecond accuracy?

If so, which would be the better way?

There is a known problem of windows vs linux with timestamps. Windows for some reason is less accurate and change the clock every usage of Windows, while in Linux it's considered more accurate. I'm saying it from a bad experience so far.

Secondly. you always have to know where the clock is set for. that is timezone etc... If it's UTC or timezone changes the way you should use the timestamp then a local one, unless you are interested in the local time only.

Ido
LINESIP - Opening the source for communication
http://www.linesip.com
http://www.linesip.co.il
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110220/ff049146/attachment-0001.html>

In fact I am interested mainly in the time *difference* between successive measured data points. So if I could get a fix at the first measure to a complete time stamp, then all following need only a fairly accurate *relative* time from the first.

I wonder if GetTickCount will get me a better value to milliseconds than Now()?

I have heard of a "multimedia" time on Windows, which is more accurate, but I am not sure how to use it and if it exists on Linux...

Al 20/02/11 12:42, En/na Bo Berglund ha escrit:

I wonder if GetTickCount will get me a better value to milliseconds than Now()?

Not under linux:

function GetTickCount: DWord;
begin
  Result := DWord(Trunc(Now * 24 * 60 * 60 * 1000));
end;

Bye

Bo Berglund wrote

In fact I am interested mainly in the time *difference* between successive measured data points.

You can try EpicTimer too. It's only a single unit, easy to use, supports multiple timers, stopwatch and various display modes. It's not very portable [read not fully implemented] for exotic platforms, but works perfectly for my needs under Linux and Windows.

Now I have tested both the GetTickCount and Now() methods. I have synched my tests against a GPS receiver to get an accurate reference.

Turns out that GetTickCount has a sawtooth-like overlaid timing noise signal which is about 15 ms.

But using Now() I don't see this.

In both cases I could detect a 12 ppm drift in the clock of mt PC.

i gotta ask... is that even accurate? what are those "ticks" that are being counted? by the above, they surely do not appear to be the machine's actual clock timer ticks... can those even be accessed in today's world or do we really need to revert some things like this back to the DOS world???

of course, by now, you've seen the post that shows that GetTickCount uses now() as the base with which to multiply by :? ;)

This is a Delphi wrapped API call in Windows that returns the number of milliseconds since Windows started. I had the impression that they were at least locked to the PC clock somehow and actually gave a millisecond resolution, but now I believe that the "counter" is in fact working in a different way than one would believe by the name.

When I run a test against a fairly accurate GPS time source I see that there is an uncertainty of about 16 ms in the time given such that if GetTickCount is executed at exactly 1 second intervals the last two digits of the value is not the same as it should be. Instead it slowly (over a period of 16 seconds) reduces by 16 ms and then suddenly restores to a max value.
It is like a superimposed sawtooth with an amplitude of 16 ms.

The posted variation is apparently a Linux platform compatible version that returns a millisecond value the same way as the Windows version does, since there is no Windows GetTickCount API call in Linux.

I used this also on Windows in my tests and then the value does not exhibit the sawtooth anymore. So I assume that Now() gets its time from another source than the timer counter driving GetTickCount and this is more accurate.

Hello Lazarus-List,

Monday, February 21, 2011, 7:23:54 AM, you wrote:

BB> When I run a test against a fairly accurate GPS time source I see that BB> there is an uncertainty of about 16 ms in the time given such that if BB> GetTickCount is executed at exactly 1 second intervals the last two BB> digits of the value is not the same as it should be. BB> Instead it slowly (over a period of 16 seconds) reduces by 16 ms and BB> then suddenly restores to a max value.
BB> It is like a superimposed sawtooth with an amplitude of 16 ms.

Quote from MSDN:

"The resolution of the GetTickCount function is limited to the resolution of the system timer, which is typically in the range of 10 milliseconds to 16 milliseconds. The resolution of the GetTickCount function is not affected by adjustments made by the GetSystemTimeAdjustment function."

http://wiki.lazarus.freepascal.org/EpikTimer

Felipe,

Please stop recommending this component.
EpikTimer is not a well-written component.

It's precision is not better than Now() unless you're on i386, Windows. For all other platforms it's useless and introduces uncalled-for dependencies.

Michael.

Al 21/02/11 07:23, En/na Bo Berglund ha escrit:

The posted variation is apparently a Linux platform compatible version that returns a millisecond value the same way as the Windows version does, since there is no Windows GetTickCount API call in Linux.

The problem with the current implementation of GetTickCount under Linux is that's it's not monotonic: I use it to measure elapsed time (usually short spans, in the range from half a second to a few seconds), and, while under Windows it's guaranteed to work (I don't mind the wrap-around after 24 days, since the difference between two consecutive GetTickCount still gives a positive difference representing the elapsed time), under Linux if somebody adjusts the clock my timekeeping will fail (I have some checks in place to fail gracefully, but there are still some corner cases).

Maybe a better way to implement a compatibility function would be to use clock_get_time with CLOCK_MONOTONIC, though I don't know if it's accessible from freepascal.

http://linux.die.net/man/3/clock_gettime

Bye

That was the non-Windows implementation.

Regards,
Sven

I should have added that instead of a sawtooth like noise component using Frac(Now()) * 24*3600 to get the second since midnight gives random noise in the seconds values of about 3-4 ms. Another observation is that there seems to be no data smaller than ms. If I format this value as a float with more decimals than 3 I always only see zeroes.

That is logical, since TDateTime only is accurate to millisecond precision.

Michael.

Bo Berglund wrote

I should have added that instead of a sawtooth like noise component using Frac(Now()) * 24*3600 to get the second since midnight gives random noise in the seconds values of about 3-4 ms.

No wonder. In a non-realtime OS like Windows of Linux, you can't create sound using a timer in a user land program.

-Michael

2011/2/22 Michael Schnell :

In this case I wouldn't call "random noise" sound, since you cannot hear it. :-)

Vincent

Granted that, but right now I am mostly interested in finding a way to time an external event source such that I can measure the time between events.

After googling a bit I have come up with the following, which is a variation of a delay function that I created years ago in Delphi to enable me to output pulses that are timed to milliseconds. The delay is now converted to a time retrieval function:

function TForm1.GetMillisecondTime: Int64;
var
  Freq,
  Ts: Int64;
begin
  QueryPerformanceCounter(Ts);
  QueryPerformanceFrequency(Freq);
  if (Ts = 0) or (Freq = 0) then exit;
  Freq := Freq div 1000;
  Result := TS div Freq;
end;

When I try this in Lazarus I get an "Identifier not found" error (not surprising).
What should I do in order to get this to compile?
Any suitable unit to add to the uses clause?

If you are on windows: add the windows unit.

If you are on another platform: these functions simply do not exist on other platforms.

Michael.

Bo Berglund wrote

> That is logical, since TDateTime only is accurate to millisecond precision.

While TDateTime might provide mSec resolution, the functions that generate such a value are _much_ less exact in a user land program. Even is such a function would work perfectly, the OS might delay the program for hundreds of mSecs until the value is used.

-Michael

Op 2011-02-22 15:08, Michael Schnell het geskryf:

is such a function would work perfectly, the OS might delay the program for hundreds of mSecs until the value is used.

Unless [under Linux], you run that application as root, and give it real-time (or equivalent) priority. That should at least help a bit, giving greater priority over other "normal" processes.

Regards,
- Graeme -

On top of this you can add some "realtime Patches" to Linux to reduce the latency (while degrading the overall performance).

-Michael

Thanks,
I have now tested it (on Windows) and found that although the QueryPerformanceCounter gives more resolution it is also drifting compared to real time (77 ms in 30 s) making it rather unusable... It would be good for a highres timer but not for keeping track of time over longer periods.

I took steps to make the calculations less vulnerable to resolution issues by first snatching a counter value and then for all subsequent calculations I subtracted this from what I read before switching to double precision arithmetic.

Have to look elsewhere.

Bo Berglund wrote

Have to look elsewhere.

The only really usable solution might be to attach an external time source to your PC and use that time signal.

I don't know of a suitable hardware though (I haven't yet done anything related to such accurate timings).

Regards,
Sven

Op 2011-02-23 10:50, Sven Barth het geskryf:

I don't know of a suitable hardware though (I haven't yet done anything related to such accurate timings).

I have heard of users using something as simple as GPS devices. What accuracy that gives, I am not sure - but I do know each GPS syncs with 9?12 atomic clocks [depending or reception].

Regards,
- Graeme -

Well what do you expect from _non-military_ grade hardware? No quartz of whatever form is ever perfect in every temperature.

Try finding the master quartz on your main board and try using the 'freeze spray' on it or just touch it with your finger,
or (dangerous) solder a small trimmer capacitor parallel to one of the decoupling caps there, and you'll discover what happens... the oscillators in the chipsets probably have some temperature compensation, but not one is ever perfect, _and_ it is also affected by the chipset (and so the logic gates driving the quarts) heating up...

Internal clock base is the only reference for _whatever_ you run in software. People have discovered remote daytime clock update over internet for this; But to rdtsc() and compare that to external source ... hmm.

If in Linux, there is a feature I read about on LKML, that lets you connect external stable clock signal to the serial port and drive the OS wall clock that way; (IIRC and IIUC applies...)
I think it was called PPS or PPM. It was merged 2-3 stable releases ago I think.

L.

Right, that would be an idea (and Bo is using that already to check the accuracy of the system time ;) )

Regards,
Sven

My Linux version:

function GetTickCount: Cardinal;
var
  ts: TTimeSpec;
  i: Int64;
begin
  if clock_gettime(CLOCK_MONOTONIC, @ts)  0 then begin //kernels 2.4.* does not support
    Result := 0;
    Exit;
  end;
  i := ts.tv_sec;
  i := i*1000 + ts.tv_nsec div 1000000;
  Result := i and $ffffffff; //cut to unsig 32bit
end;

Petr

Al 23/02/11 13:17, En/na Petr Kristan ha escrit:

That's what I suggested a few messages ago (though I read somewhere that CLOCK_MONOTONIC can drift, and there's a new, really monotonic, CLOCK_MONOTONIC_RAW), but that's not the version in the lcl.
BTW: in which unit are clock_gettime and related constants defined?

Bye

I found one interesting example here (now we are getting OT...): http://www.lammertbies.nl/comm/info/GPS-time.html

It uses a Garmin GPS 18x LVC sensor with bare wire attachments. One wire contains the PPS (pulse per second) output, which generates one pulse of 100 ms every second and starts within a microsecond of UTC second transition!

The page also contains instructions on how to make a Linux time server based on this device.

Yeah, but my efforts are not successful in using a USB connected GPS sensor and trying to get accuracy from the telegrams sent by it. I think much of the time wobble I see is caused by USB latency in the track from the GPS sensor to the USB hub to the next USB hub and then into the laptop.
Add to this the way the device driover (Prolific USB2Serial) might buffer up the data before it reaches the Windows serial system and there are lots of places time can be lost.

As I posted just a minute ago I found the perfect solution in a Garmin device with microsecond accurate pulse output.

Have just ordered one for testing.

Bo Berglund wrote

On Wed, 23 Feb 2011 10:20:19 +0100, Sven Barth wrote:

<snip>

In any case, most GPS engines have a certain latency in even sending out the serial time signals, and, as I understand it, don't consider the time as important as the position in the internal program, in terms of what they output. The internal time, is of course, critical. So if you are interested in anything like millisecond timing, the 1 PPS signal (as in other post) is definitely the way to go. This is used regularly by amateur (and quite possibly professional) astronomers for timing stuff, eg to get exact times of a video stream on a frame by frame basis. If you can't find enough information from your existing sources, try a search on "occultation timing".

cheers,
John Sunderland

Op 2011-02-23 18:56, Luca Olivetti het geskryf:

BTW: in which unit are clock_gettime and related constants defined?

libc (FPC wrapper to the libc library)

...more specifically...

packages/libc/src/timeh.inc

I wonder if we can't use Petr's GetTickCount() implementation as default for FPC+Linux/BSD? Anybody raised this question to the FPC developers?

Regards,
- Graeme -

I really don't understand why you invest so much effort in the fruitless try to deal with mSec accuracy in a Linux user land program.

-Michael

Michael Schnell wrote

I really don't understand why you invest so much effort in the fruitless try to deal with mSec accuracy in a Linux user land program.

Maybe, but you don't know what I am after, right?

I need to synchronize two separated units such that they operate in the same time even though they cannot be connected to each other. The only thing I can think of is to use time sync and this in turn depends on the ability to retrieve a sensibly good time reference.

It seems like I am out of luck doing this with readily available hardware such as PC:s, notebooks or embedded linux cards running software (in this case an FPC application) so I have to bite the bullet and design a printed circuit board with the necessary microprocessor, port and GPS parts and do it in hardware + software very close to the hardware instead...
But then I can get microsecond synchronism or close to it.

Bo Berglund wrote

It seems like I am out of luck doing this with readily available hardware such as PC:s, notebooks or embedded linux cards running software

IMHO, the _Hardware_ is not the problem at all, but the OS is. You need to use a decent realtime OS (or none) to get msec resolution. There are several projects to make Linux more realtime aware. Those that are usable for _real_ realtime stuff (i.e. the realtime-processes need to meed predefined latency specifications without any exception) use a realtime OS below Linux and have Linux run as the lowest priority process of same.

-Michael

agreed 100%

i still have some (old) code around here, somewhere, that sets the tics in the timer chip and then also installs a timer interrupt routine to ensure that the clock still ticks over at the proper rate... when i was first messing around with this, my system clock covered several days in minutes :lol:

but yes, this can't be done like it was with today's OS' breaking the boundary between the hardware and the software... granted, it does make coder's jobs somewhat easier (because they don't have to deal with the hardware) but not always like in this case where much finer timing is necessary...

i'm probably going to use the wrong terms here but would it not be possible for the OP (bo?) to do a "ring 0" "driver" that hooks the timer and possibly adjusts it to count faster like i mention above? that driver could then feed that data to his package, couldn't it?

No, no, no and again no.

Don't suggest anyone to use the unit libc. It's only there for compatibilty for Kylix and works only on i386-linux. If someone wants to have "clock_gettime" then a "fpclock_gettime" should be added to e.g. "baseunix" or "linux" which uses the "syscall_nr_clock_gettime" syscall.

I wonder if we can't use Petr's GetTickCount() implementation as default for FPC+Linux/BSD? Anybody raised this question to the FPC developers?

But as a fallback for 2.4 kernels the old Now variant could be used.

Regards,
Sven

Op 2011-02-25 13:45, Sven Barth het geskryf:

Yes, yes, yes and again yes.... I know that. :) That is why I explicitly said "FPC wrapper to the libc library" .... I did NOT say the 'libc unit'.

But as a fallback for 2.4 kernels the old Now variant could be used.

Correct. Petr's implementation doesn't have that bit, but it's one simple line to add.

Regards,
- Graeme -

You were a bit vague in that regard... so I wanted to be sure :P

Regards,
Sven

That is why I said that I would design a PCB with a MCU on board and write a program to fix the synching problem that way. A PIC processor with an UART and a few I/O pins to read the PPS pulse and NMEA data from the GPS 18x LVC is what is needed.
No operating system at all in this board....

I need close to millisecond timing.

Al 24/02/11 08:22, En/na Graeme Geldenhuys ha escrit:

Two things:

1) in fpc 2.4.2 CLOCK_MONOTONIC isn't defined (only CLOCK_REALIME)

and

2) I couldn't find clock_gettime before because I relied on lazarus code completion. It turns out that lazarus doesn't uses the correct source for libc but the one for netware (I think), i.e. it uses

/usr/share/fpcsrc/fpc-2.4.2/rtl/netwlibc/libc.pp

instead of

/usr/share/fpcsrc/fpc-2.4.2/packages/libc/src/libc.pp

(I have the fpc source directory defined as /usr/share/fpcsrc/fpc-2.4.2)

Bye

Luca Olivetti wrote

>> BTW: in which unit are clock_gettime and related constants defined?

The answer should be : in no portable unit.

Unit libc doesn't count since it is only for certain legacy reasons, and Lazarus should NOT use it.

Al 26/02/11 17:58, En/na Marco van de Voort ha escrit:

I know that, I know the reasons, and I agree.
But those are posix functions, so they could be defined somewhere else than libc.

Bye

We don't specify posix compliance for a reason. That is because we want to keep baseunix to be somewhat portable, and implementable on POSIX systems that are not the newest Linux or FreeBSD.

These functions have been optional for a long time as the "real time extensions", and while they are now standarized, e.g. the MONOTONIC functionality is still optional (in the 2004 edition)

Besides these careful approach issues, there are also practical issues. Before a call is added to the supported set, there must be a good reason (1), and it must first be somewhat proven to be implementable on all supported *nixes, versions and flavours(2). Some calls are hard to implement without linking to libc, and baseunix must remain implementable without libc.

A workaround for such cases would be to keep it out of the RTL, and stuff it in a 3rd party package. Or in case of a great need add it to packages/.

(1) this discussion is about a gettickcount implementation, but I'm not entirely sure why this is needed so badly.
(2) embedded (like uclibc) issues shouldn't be underestimated.

Al 26/02/11 19:43, En/na Marco van de Voort ha escrit:

(1) this discussion is about a gettickcount implementation, but I'm not entirely sure why this is needed so badly.

I use it extensively to keep track of time and
I can currently live with the workaround DWord(Trunc(Now * 24 * 60 * 60 * 1000)) but I have to put some safeguards since that is not monotonic. Under windows I can be sure that the difference between GetTickCount and PreviousGetTickCount will be positive (at least if I don't need to time events more than 24 days apart), under Linux I have no such guarantee. So if, e.g. I have

function lapsed(origin:dword):dword;
begin
  result:=GetTickCount-origin;
end;

under windows I can do, eg

if lapsed(starttime)>5000 then
...
with no worries, while under linux I need
(to avoid waiting forever in case someone touches the system clock)

if (lapsed(starttime)>5000) or (lapsed(starttime)<0) then
...

which, apart from being quite inelegant, could fire well before 5 seconds have passed.

Bye

Marco van de Voort wrote

(1) this discussion is about a gettickcount implementation, but I'm not entirely sure why this is needed so badly.

When you are doing something soft real time it is a must. Things like:
- Is my image processing algorithms under the 40ms allowed time frame. (If you need feedback at 25Hz to control your robot :)
- Is the rigid body simulation step taking too much time, or is the drawing stuff. What is the cost of adding more bodies to the simulation? (I'm clicking "add" and I'm seeing how much the time increases)
- ...

Essentially, when you have to know how much time you are paying for a specific task in the middle of many others.

(2) embedded (like uclibc) issues shouldn't be underestimated.

Some time you need more muscle than what is typically available on those systems.

Paulo Costa

Marco van de Voort wrote

(1) this discussion is about a gettickcount implementation, but I'm not entirely sure why this is needed so badly.

I am by no means a libc expert, though I have tried myself to find a more fine-grained timing call under Linux. Windows (and I believe the GetTickCount implementation for Windows) uses the higher precision timing calls from the multimedia API. I can't confirmed this because I'm not at a computer with FPC source code - but from memory I think that is how it was done. The current linux version of GetTickCount is very rudimentary. As a few Google searches will show, many other developers are saying that using clock_gettime() under linux gives you the equivalent timing as the multimedia API under Windows.

So couldn't the GetTickCount for linux use the improved function calls? I'm not talking about all other unix functions, start with maybe only Linux and work from there.

Graeme Geldenhuys wrote

So couldn't the GetTickCount for linux use the improved function

I forgot to add...This will at least gives us improved timing under Linux, and still allow use to use the relatively cross-platform GetTickCount() call between the most popular platforms (Linux and Windows) that FPC supports.

calls? I'm not talking about all other unix functions, start with

sorry for the typo, that should read "...all other unix platforms..."

I don't know whether you stuff the GetTickCount API call under "Multimedia API", but the GetTickCount implementation for Windows uses just that.

See this snippet from lcl/lclintf.pas:

=== source begin ===
{$IFDEF WINDOWS}
{$IFDEF MSWindows}
function GetTickCount:DWORD; stdcall; external 'kernel32.dll' name
'GetTickCount';
{$ELSE}
function GetTickCount:DWORD; stdcall; external KernelDLL name
'GetTickCount';
{$ENDIF}
{$ELSE}
function GetTickCount: DWord;
{$ENDIF}
=== source end ===

And GetTickCount uses the data provided by the Kernel for that (I don't know yet how the kernel calculates that data though...)

Regards,
Sven

Sven Barth wrote

I don't know whether you stuff the GetTickCount API call under "Multimedia API", but the GetTickCount implementation for Windows uses just that.

My apologies then. As I mentioned I don't have a computer with FPC source code in front of me. I must have thought of another "wrapper" timing function then, which under linux used the "fake" GetTickCount(), and under Windows the multimedia timing API.

Thanks for correcting me though.

You are welcome :)

Regards,
Sven

Graeme Geldenhuys wrote

The current linux version of GetTickCount is very rudimentary.

Possibly. But IMHO it is a windows knock-off function, and is defined as it is defined to keep code more or less running. If it drags in heaps of requirements, its portability status will have to be reviewed IMHO

As a few Google searches will show, many other developers are saying that using clock_gettime() under linux gives you the equivalent timing as the multimedia API under Windows.

Then make a nice linux specific unit, and post it on some website.

So couldn't the GetTickCount for linux use the improved function calls?

I don't see the use of that. It will only complicate documentation to change behaviour across platforms even more.

I'm not talking about all other unix functions, start with maybe only Linux and work from there.

No. That is not the way we do the Unix rtl. If we had done that, and caved at each challenge, we would be back at the unit libc chaos by now.

unit libc by now

Ah. it is in lazarus. (I had in mind it was in sysutils). Nevermind what I said then.

This is exactly, what I would do, too. :-)

-Michael

Bo Berglund wrote

I would design a PCB with a MCU on board...

Yep. But this PCB would need to do* all *the realtime stuff, store the results in a kind of FiFo and communicate them towards the PC via a serial interface or USB (doing USB is easy with an appropriate PIC24). The PC software itself still is not
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110228/66a84eea/attachment.html>