Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2014-08

best? safest? fastest?

5 messages · 2 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

waldo kitty

i suspect this is going to be like the long-standing joke of

cheap, fast, stable: choose two

over the years, i've seen two schools of code for dealing with dates... years, specifically... one school is string based and the other is math based... both have their faults and pluses...

eg: string based fault : prepend '19' to single digit year value math based fault : 2003 - 1900 = 103 (3 is intended result)

so what i have is satellite tracking two line element files... these files contain an epoch for the numbers contained in the TLE... the epoch is laid out as

YYdoy.frac-doy == 14211.03356282

where

YY : 1 or 2 digit year without century ('62' or ' 3' or '05') doy : day of year, 1, 2 or 3 digits, jan 1 is day 0 (' 0' or ' 15' or '020') frac-doy : 8 digit fractional day of the year

what i'm needing to do is to choose a route to make the year portion four digits so that historical TLEs can be used... the program makes a comparison of the epoch to determine which of two is newer... the newer one is placed into the in-memory database for later usage... working with historical TLEs from 1957 to 1999 mandates four digit years be used or historical TLEs will override TLEs from 2000 to 2056 since 00 thru 56 are less than 57 (1957 first launch)...

previous to any processing, the epoch taken from the TLE in the file is processed so that leading blanks in the epoch are replaced with zeros... today's epochs don't really need this as creation and processing methods have evloved over the years... this is done as a precaution for those times when processing historical TLEs which have not been converted to having leading zeros instead of blanks (aka spaces)...

we've launched over 40000 objects and there are possibly hundreds or thousands of TLEs for each one... at least one site is known to have over 9 million TLEs in their database... these routines are executed for each and every TLE loaded as we fill our in-memory database...

finally (yeah i know, but i thought the history and methodology was important) my questions, as the subject alludes to, are which of the following methods would you choose? why? would you choose another method? why?

all input is greatly welcomed! as always, thanks for your time and attention! :)

{******************************************************************************} {******************************************************************************}

// make sure we have the full four digit year for the epoch!
// eg: 14211.03356282 becomes 2014211.03356282 which is 2014-07-30 00:48:19.827

// method 1 (string manipulation)
if (Integer_Value(MyEpochStr,1,2) < 57) then // first launch in 1957

   MyEpochStr := FloatToStrF('20'+MyEpochStr,ffFixed,16,8)            // value <

57 so year is 21st century (2000 thru 2056)
else

   MyEpochStr := FloatToStrF('19'+MyEpochStr,ffFixed,16,8);           // value

is >= 57 so year is 20th century (1957 thru 1999)
MyEpoch := Real_Value(MyEpochStr,1,Length(MyEpochStr)); // convert to real for mathmatical comparison

{******************************************************************************}

// method 2 (combination string and math??)
if (Integer_Value(MyEpochStr,1,2) < 57) then                         // first
launch in 1957
   MyEpochStr := FloatToStrF(Real_Value(MyEpochStr,1,Length(MyEpochStr)) +
2000000,ffFixed,16,8)
else
   MyEpochStr := FloatToStrF(Real_Value(MyEpochStr,1,Length(MyEpochStr)) +
1900000,ffFixed,16,8);
MyEpoch        := Real_Value(MyEpochStr,1,Length(MyEpochStr));       // convert
to real for mathmatical comparison

{******************************************************************************}

// method 3 (math manipulation)
MyEpoch := Real_Value(MyEpochStr,1,Length(MyEpochStr)); // convert to real for mathmatical comparison
if (int(MyEpoch / 1000) < 57) then // first launch in 1957

   MyEpoch := (MyEpoch / 1000 + 2000) * 1000                          // value <

57 so year is 21st century (2000 thru 2056)
else

   MyEpoch := (MyEpoch / 1000 + 1900) * 1000;                         // value

is >= 57 so year is 20th century (1957 thru 1999)
MyEpochStr := FloatToStrF(MyEpoch,ffFixed,16,8); // store to MyEpochStr for later use (16 characters!)

{******************************************************************************} {******************************************************************************}

support routines for the above:

   Function Integer_Value(buffer : string; start, length : integer) : integer;
   var
     MyResult : integer = 0;
     answer   : integer - 0;
   begin
     buffer := Copy(buffer,start,length);
     Convert_Blanks(buffer);
     if buffer = '' then
       buffer := '0';
     Val(buffer,answer,MyResult);
     if MyResult = 0 then
       Integer_Value := answer
     else
       Integer_Value := 0;
   end; // Function Integer_Value
   Function Real_Value(buffer : string; start, length : integer) : double;
   var
     MyResult : integer = 0;
     answer   : double = 0.0;
   begin
     buffer := Copy(buffer,start,length);
     Convert_Blanks(buffer);
     if buffer = '' then
       buffer := '0';
     Val(buffer,answer,MyResult);
     if MyResult = 0 then
       Real_Value := answer
     else
       Real_Value := 0.0;
   end; // Function Real_Value

for those interested in it, here's the fractional-doy breakout routine (in displayEpochString)...

   Function getJulianDay_Year(year : integer) : double;
   var
     dYear   : double = 0.0;
     a       : double = 0.0;
     b       : double = 0.0;
     dResult : double = 0.0;
   begin
     dYear := year - 1;
     A := Math.Floor(dYear / 100);
     B := 2 - A + Math.Floor(A / 4);
     dResult := Math.Floor(365.25 * dYear) + 1721422.9 + B;
     getJulianDay_Year := dResult;
   end;
   Function getJulianDay_SatEpoch(year : integer; dSatEpochDoY : integer) : double;
   var
     dResult : double = 0.0;
   begin
// this tidy section not needed any more with four digit years in epoch
//    //Tidy up the year and put it into the correct century
//    year := year mod 100;
//    if (year < 57) then                               // first launch in 1957
//      year := year + 2000
//    else
//      year := year + 1900;
     dResult := getJulianDay_Year(year);               // julian day of first of
year
     dResult := dResult + dSatEpochDoY;                // add in jDoY
     getJulianDay_SatEpoch := dResult;
   end;
   Function displayEpochString(SatEpoch : double) : string;
   var
     test           : boolean = False;
     DT             : TDateTime;
     Y              : word = 0;
     Mo             : word = 0;
     D              : word = 0;
     H              : word = 0;
     Mi             : word = 0;
     S              : word = 0;
     MS             : word = 0;
     tstr           : string = '';
     MyEpochStr     : string = '';
     MyJEpoch       : double = 0.0;
     dWork          : double = 0.0;
     dWork1         : double = 0.0;
     MyEpochDayFrac : double = 0.0;
     MyEpochDoY     : integer = 0;
     MyEpochYear    : integer = 0;
   begin
     tstr := FloatToStr(SatEpoch);                     // whole epoch number
     if pos('.',tstr) < 1 then
       tstr := tstr + '.0';
     MyEpochStr  := AddChar('0',Copy(tstr,1,pos('.',tstr)-1),5);
     MyEpochYear := Integer_Value(MyEpochStr,1,4);     // epoch year -
characters 1-4 because we adjusted to 4 digit year
     MyEpochDoY  := Integer_Value(MyEpochStr,5,3);     // epoch DoY - characters
5-8 because we adjusted to 4 digit year
     MyEpochDayFrac := Frac(Real_Value(tstr,pos('.',tstr),9)); // epoch
fractional day starts with decimal point
     MyJEpoch := getJulianDay_SatEpoch(MyEpochYear, MyEpochDoY); // JD for year
and DoY only. time portion calc'd below...
     test := TryJulianDateToDateTime(MyJEpoch,DT);     // convert julian epoch
to TDateTime record
     if test then
       begin
         DecodeDateTime(DT,Y,Mo,D,H,Mi,S,MS);          // pull TDateTime record
apart
         dWork := MyEpochDayFrac * 24;                 // calc the hours from
the fractional day of the epoch
         H := trunc(dWork);                            // store whole hours
         dWork1 := Frac(dWork);                        // get frac hours
         dWork := dWork1 * 60;                         // calc the minutes from
the fractional hours
         Mi := trunc(dWork);                           // store whole minutes
         dWork1 := Frac(dWork);                        // get frac minutes
         dWork := dWork1 * 60;                         // calc the seconds from
the fractional minutes
         S := trunc(dWork);                            // store whole seconds
         dWork1 := Frac(dWork);                        // get frac seconds
         dWork := dWork1 * 1000;                       // calc the milliseconds
from the fractional seconds
         MS := trunc(dWork);                           // store whole
milliseconds - trunc() b/c we don't care about fractional part or rounding at
this level
         DT := EncodeDateTime(Y,Mo,D,H,Mi,S,MS);       // convert it all back to
TDateTime record
         displayEpochString:=FormatDateTime('YYYY-MM-DD hh:nn:ss.zzz',DT); //
send back a nicely formatted DateTime string
       end
     else
       displayEpochString:='XXXX-XX-XX XX:XX:XX.XXX';      // send back all X's
because Julian to DateTime failed
   end;
NOTE: No off-list assistance is given without prior approval. Please *keep mailing list traffic on the list* unless private contact is specifically requested and granted.
Mark Morgan Lloyd
to waldo kitty
waldo kitty wrote 11 lines
i suspect this is going to be like the long-standing joke of cheap, fast, stable: choose two over the years, i've seen two schools of code for dealing with dates... years, specifically... one school is string based and the other is math based... both have their faults and pluses... eg: string based fault : prepend '19' to single digit year value math based fault : 2003 - 1900 = 103 (3 is intended result)

I'd be inclined to start off using your method 1, i.e. text manipulation until the format is consistent.

Flatten the original record and save it in a database, create a new flat text record starting with a database reference and then comprising slightly-massaged text. Run consistency checks on the new records until everything looks good (e.g. no non-numeric garbage resulting from a failed EBCDIC or GOST conversion), and only then parse it into numeric fields.

Out of curiosity, I presume that times are GMT/UTC but does a day start at midnight or noon (civil and astronomers' convention respectively)? Do the coordinates have to take into account changed ephemeris changes as the equinox precesses?

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
waldo kitty
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote 15 lines
waldo kitty wrote: > i suspect this is going to be like the long-standing joke of > > cheap, fast, stable: choose two > > > over the years, i've seen two schools of code for dealing with dates... years, > specifically... one school is string based and the other is math based... both > have their faults and pluses... > > eg: string based fault : prepend '19' to single digit year value > math based fault : 2003 - 1900 = 103 (3 is intended result) I'd be inclined to start off using your method 1, i.e. text manipulation until the format is consistent.

i don't understand "until the format is consistent"... the format has been in use since the 60s at least (AFAIK) ;)

FWIW: i have taken some time and reworked things to be math based while still taking the required text format into account... i've seen a very nice increase in processing speed and now need to just make sure that i don't run into any of the basic and well known flaws that math processing of date strings seem to have ;)

Flatten the original record and save it in a database, create a new flat text

this appears that you are speaking of a sql database or similar? that may be a later feature but for now, everything is/has to be done with the raw TLE files...

i'm not sure what you mean by "flatten", either... currently i break down the TLEs into their major records for storage in the in-memory ""database""... the processing i posted is done before that storage takes place...

the goal of the program is to build the in-memory database from all specified TLE files and then to write out new TLE files which may be filtered on a selection property so that only certain matching TLE records are saved...

eg: take 100 TLE files and build one new TLE file containing only geosynch objects

record starting with a database reference and then comprising slightly-massaged text. Run consistency checks on the new records until everything looks good (e.g. no non-numeric garbage resulting from a failed EBCDIC or GOST conversion), and only then parse it into numeric fields.

yeah, we don't have to worry about EBCDIC or similar because all that's done when those tools generate the TLE files they output... FORTRAN was used originally but these days, there's numerous languages used... the format is plain old 7bit ASCII characters of 0-9... if there's a 0th line with the object's ""name"" in it, it is only (IME) A-Z0-9... for my purposes, lowercase letters are no problem, though ;)

Out of curiosity, I presume that times are GMT/UTC but does a day start at midnight or noon (civil and astronomers' convention respectively)?

i'm not sure on this either... i just know that my calculations return the same results as all the other programs and web sites i've validated against... i'm pretty sure that UTC is the base and i assume that the day starts at midnight... again, this because my calculations return the same results and projections as other programs and web sites i've validated against...

more information on the TLE format is available in numerous places on the web... one well known one is

http://celestrak.com/columns/v04n03/

Do the coordinates have to take into account changed ephemeris changes as the equinox precesses?

i presume so but am not sure... each TLE record stands on its own and can be used to create tracking forward and backward in time for X period of time... as time progresses from the epoch, the certainty of the calculations falls off for numerous reasons... solar flux, perturbations, atmospheric drag, boosting the orbit, etc...

NOTE: No off-list assistance is given without prior approval. Please *keep mailing list traffic on the list* unless private contact is specifically requested and granted.
Mark Morgan Lloyd
to waldo kitty
waldo kitty wrote 22 lines
On 8/6/2014 4:08 AM, Mark Morgan Lloyd wrote: > waldo kitty wrote: >> i suspect this is going to be like the long-standing joke of >> >> cheap, fast, stable: choose two >> >> >> over the years, i've seen two schools of code for dealing with >> dates... years, >> specifically... one school is string based and the other is math >> based... both >> have their faults and pluses... >> >> eg: string based fault : prepend '19' to single digit year value >> math based fault : 2003 - 1900 = 103 (3 is intended result) > > I'd be inclined to start off using your method 1, i.e. text > manipulation until > the format is consistent. i don't understand "until the format is consistent"... the format has been in use since the 60s at least (AFAIK) ;)

What I mean is, while you're doing the initial processing to e.g. add century digits to the date and possibly to check number of decimal places etc.

Show quoted text 12 lines
FWIW: i have taken some time and reworked things to be math based while still taking the required text format into account... i've seen a very nice increase in processing speed and now need to just make sure that i don't run into any of the basic and well known flaws that math processing of date strings seem to have ;) > Flatten the original record and save it in a database, create a new > flat text this appears that you are speaking of a sql database or similar? that may be a later feature but for now, everything is/has to be done with the raw TLE files...

Databases, even for plain-text records, can be incredibly useful.

i'm not sure what you mean by "flatten", either... currently i break down the TLEs into their major records for storage in the in-memory ""database""... the processing i posted is done before that storage takes place...

I was thinking that the first thing you could do was convert the two lines into a single one for processing, but on reflection it would be better to save the original with as little modification as possible- possibly with any accession info you had (i.e. what body had provided that particular TLE).

the goal of the program is to build the in-memory database from all specified TLE files and then to write out new TLE files which may be filtered on a selection property so that only certain matching TLE records are saved...

The problem there being that once the program stops you've then got to rebuild the next time. What I normally do when handling large bodies of tabular info is to either have a series of database tables or a series of text files, where ideally the text files are absolutely predictable (all fields a known length and appropriately padded).

What I'm normally looking for is rate-of-change over multiple records with irregular timestamps, which is an awkward job however it's done.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
waldo kitty
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote 10 lines
waldo kitty wrote: > On 8/6/2014 4:08 AM, Mark Morgan Lloyd wrote: >> I'd be inclined to start off using your method 1, i.e. text manipulation until >> the format is consistent. > > i don't understand "until the format is consistent"... the format has been in > use since the 60s at least (AFAIK) ;) What I mean is, while you're doing the initial processing to e.g. add century digits to the date and possibly to check number of decimal places etc.

oh, ok... yeah, that's all done before we reach this stage... as i posted previously, the format is already laid out so the string spacing is already a known factor... it is only a couple of places where one might run into historical TLE records that are space filled instead of zero filled in the mathematical number areas... the NORAD, COSPAR and Epoch areas are the only ones that one might find

Show quoted text 12 lines
> FWIW: i have taken some time and reworked things to be math based while still > taking the required text format into account... i've seen a very nice increase > in processing speed and now need to just make sure that i don't run into any > of the basic and well known flaws that math processing of date strings seem to > have ;) > >> Flatten the original record and save it in a database, create a new flat text > > this appears that you are speaking of a sql database or similar? that may be a > later feature but for now, everything is/has to be done with the raw TLE files... Databases, even for plain-text records, can be incredibly useful.

agreed very much... however, in this case, with the old convoluted string stuffings that were being done, processing 80000+ records takes only a minute or two whereas the previous tool was limited by the old 64k memory limit and processing the same files took over 45 minutes and resulted in only 10% of the total output of the new tool i've written... it is very late and my math may be off but output file sizes of 274K compared to 2.5Meg is 10%, right?

but as previously noted, a real database is a possibility for the future... currently, my personal processing uses a text file containing all the latest TLEs that i have been able to accumulate... granted, the historical ones are not available in this format but then again, i don't have room for 9million records like other systems ;)

Show quoted text 8 lines
> i'm not sure what you mean by "flatten", either... currently i break down the > TLEs into their major records for storage in the in-memory ""database""... the > processing i posted is done before that storage takes place... I was thinking that the first thing you could do was convert the two lines into a single one for processing, but on reflection it would be better to save the original with as little modification as possible- possibly with any accession info you had (i.e. what body had provided that particular TLE).

when i move to incorporate ""real"" database capabilities, something like this will be being done... that's an understood :) the reason i hesitate to do this now is due to the processing speed which has been achieved compared to the old tool written by someone else... adding database support will really slow all the processing down compared to what has been achieved at this point... granted, flat file text processing is ""slow"" and ""ugly"" but in this case, it is a major GoodThing<tm> :)

Show quoted text 6 lines
> the goal of the program is to build the in-memory database from all specified > TLE files and then to write out new TLE files which may be filtered on a > selection property so that only certain matching TLE records are saved... The problem there being that once the program stops you've then got to rebuild the next time.

that's not really a problem in this case... when i allow others to use the tool (which is still considered by myself as alpha but may be beta or gamma level by others), they may choose to use it the same way that i do... or not... as noted above, i maintain a master ""database"" file which is loaded first and then the latest live data and finally all the "just in" update files... from these are built the desired output files... every once in a while i output an updated ""master database""... out of over 40000 possible records, i think i'm missing only a thousand or so and those are unlikely to ever be filled due to their military aspect and the lack of publicly available data on their orbital activities... so back to your statement above, i guess i'm already maintaining a fairly up-to-date ""master database" which is quickly loaded and then updated by the other files being processed...

What I normally do when handling large bodies of tabular info is to either have a series of database tables or a series of text files, where ideally the text files are absolutely predictable (all fields a known length and appropriately padded).

i guess that's what i'm doing as described above ;)

What I'm normally looking for is rate-of-change over multiple records with irregular timestamps, which is an awkward job however it's done.

i do have extensive logs which i regularly review... one night's processing generates some 45Meg or so for roughly 60 output files ;)

NOTE: No off-list assistance is given without prior approval. Please *keep mailing list traffic on the list* unless private contact is specifically requested and granted.