Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2012-01

sorting and merging array of records

34 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

waldo kitty

i have a li'l project i've been wanting to do for a really long time... as i'm trying to do more and more with FPC, i've experimented a bit and found that i should be able to handle this project much easier without the old DOS limitations on memory usage and without having to use special stuff like extended and expanded memory...

so, the basics are that i'm also an amateur satellite observer... i keep up with the satellites via what's known as TLE (two line element) files... the ones i use are actually three lines where the first line is the satellite's common english name and the other two lines are the actual TLE element lines... there are certain enforced limits on these lines... as such, i have defined a record

type
   sat_name   = string[25];
   line_data  = string[69];
   two_line   = array [1..2] of line_data;
   three_line_data = record
                       satname : sat_name;
                       satdata : two_line;
                     end;
const
   max_sats = 65536;
var
   sat_data             : array [1..max_sats] of three_line_data;
   abuf                 : two_line;
   new_data             : three_line_data;

right now i'm loading the array via a simple readln into the array elements...

Procedure Input_Satellite(index : word);
   begin
   if not EOF(fsat) then
     begin
     Readln(fsat,sat_data[index].satname);
     Readln(fsat,sat_data[index].satdata[1]);
     Readln(fsat,sat_data[index].satdata[2]);
     end; {if}
   end; {Procedure Input_Satellite}

this is fine for the initial loading but i'm needing to read additional TLE files and insert those records into the above array... my problem is that i need the array sorted on a sectional value from sat_data[index].satdata[1]

   sat_catnr    := Copy(abuf[1],3,5);
and
   new_catnr    := Copy(abuf[1],3,5);

which are loaded and compared between sat_data[index].sat_data and new_data.sat_data

in the case of collisions (same catnr), i need to choose between the greater of a pair of values from the stored record (sat_data) and the new record (new_data) read from a file...

Function Real_Value(buffer : string;
               start,length : integer) : double;
   var
     result : integer;
     answer : double;
   begin
   buffer := Copy(buffer,start,length);
   Convert_Blanks(buffer);
   if buffer = '' then
     buffer := '0';
   Val(buffer,answer,result);
   if result = 0 then
     Real_Value := answer
   else
     Real_Value := 0.0;
   end; {Function Real_Value}

via...

   sat_epoch    := Real_Value(abuf[1],19,14);
and
   new_epoch    := Real_Value(abuf[1],19,14);

which are loaded and compared between sat_data[index].satdata and new_data.satdata...

my problem is that i cannot find any similar examples where an array of records is built, sorted and duplicates are eliminated based on specific factors of the two records being compared...

uncle has been failing me for the last several hours and i'm loosing sight of what i'm trying to get done... i've seen references to using a tstringlinst as well as something apparently non-existent called a tarray... in all of the stuff i've seen on tstringlinst, there's nothing that i find that is close to what i'm (thinking i'm) looking for... and here's where i keep getting diverted away from the final goal due to trying to follow everything else :/

Tomas Hajny
to waldo kitty

On Tue, January 10, 2012 05:14, waldo kitty wrote:

Hi,

.
.

Show quoted text 97 lines
are certain enforced limits on these lines... as such, i have defined a record type sat_name = string[25]; line_data = string[69]; two_line = array [1..2] of line_data; three_line_data = record satname : sat_name; satdata : two_line; end; const max_sats = 65536; var sat_data : array [1..max_sats] of three_line_data; abuf : two_line; new_data : three_line_data; right now i'm loading the array via a simple readln into the array elements... Procedure Input_Satellite(index : word); begin if not EOF(fsat) then begin Readln(fsat,sat_data[index].satname); Readln(fsat,sat_data[index].satdata[1]); Readln(fsat,sat_data[index].satdata[2]); end; {if} end; {Procedure Input_Satellite} this is fine for the initial loading but i'm needing to read additional TLE files and insert those records into the above array... my problem is that i need the array sorted on a sectional value from sat_data[index].satdata[1] sat_catnr := Copy(abuf[1],3,5); and new_catnr := Copy(abuf[1],3,5); which are loaded and compared between sat_data[index].sat_data and new_data.sat_data in the case of collisions (same catnr), i need to choose between the greater of a pair of values from the stored record (sat_data) and the new record (new_data) read from a file... Function Real_Value(buffer : string; start,length : integer) : double; var result : integer; answer : double; begin buffer := Copy(buffer,start,length); Convert_Blanks(buffer); if buffer = '' then buffer := '0'; Val(buffer,answer,result); if result = 0 then Real_Value := answer else Real_Value := 0.0; end; {Function Real_Value} via... sat_epoch := Real_Value(abuf[1],19,14); and new_epoch := Real_Value(abuf[1],19,14); which are loaded and compared between sat_data[index].satdata and new_data.satdata... my problem is that i cannot find any similar examples where an array of records is built, sorted and duplicates are eliminated based on specific factors of the two records being compared... uncle has been failing me for the last several hours and i'm loosing sight of what i'm trying to get done... i've seen references to using a tstringlinst as well as something apparently non-existent called a tarray... in all of the stuff i've seen on tstringlinst, there's nothing that i find that is close to what i'm (thinking i'm) looking for... and here's where i keep getting diverted away from the final goal due to trying to follow everything else :/

I've tried something similar recently. After trying to figure out how to use the classes included with FPC and build something on top of them, I resorted to using the old TP/BP compatible object model with TSortedCollection (as provided in unit objects).

Tomas

Unknown author
to Tomas Hajny
Tomas Hajny wrote 109 lines
On Tue, January 10, 2012 05:14, waldo kitty wrote: Hi, . . > are certain enforced limits on these lines... as such, i have defined a > record > > type > sat_name = string[25]; > line_data = string[69]; > two_line = array [1..2] of line_data; > three_line_data = record > satname : sat_name; > satdata : two_line; > end; > > const > max_sats = 65536; > > var > sat_data : array [1..max_sats] of three_line_data; > abuf : two_line; > new_data : three_line_data; > > right now i'm loading the array via a simple readln into the array > elements... > > Procedure Input_Satellite(index : word); > begin > if not EOF(fsat) then > begin > Readln(fsat,sat_data[index].satname); > Readln(fsat,sat_data[index].satdata[1]); > Readln(fsat,sat_data[index].satdata[2]); > end; {if} > end; {Procedure Input_Satellite} > > this is fine for the initial loading but i'm needing to read additional > TLE > files and insert those records into the above array... my problem is that > i need > the array sorted on a sectional value from sat_data[index].satdata[1] > > sat_catnr := Copy(abuf[1],3,5); > and > new_catnr := Copy(abuf[1],3,5); > > which are loaded and compared between sat_data[index].sat_data and > new_data.sat_data > > in the case of collisions (same catnr), i need to choose between the > greater of > a pair of values from the stored record (sat_data) and the new record > (new_data) > read from a file... > > Function Real_Value(buffer : string; > start,length : integer) : double; > var > result : integer; > answer : double; > begin > buffer := Copy(buffer,start,length); > Convert_Blanks(buffer); > if buffer = '' then > buffer := '0'; > Val(buffer,answer,result); > if result = 0 then > Real_Value := answer > else > Real_Value := 0.0; > end; {Function Real_Value} > > > via... > > sat_epoch := Real_Value(abuf[1],19,14); > and > new_epoch := Real_Value(abuf[1],19,14); > > > which are loaded and compared between sat_data[index].satdata and > new_data.satdata... > > my problem is that i cannot find any similar examples where an array of > records > is built, sorted and duplicates are eliminated based on specific factors > of the > two records being compared... > > uncle has been failing me for the last several hours and i'm loosing sight > of > what i'm trying to get done... i've seen references to using a > tstringlinst as > well as something apparently non-existent called a tarray... in all of the > stuff > i've seen on tstringlinst, there's nothing that i find that is close to > what i'm > (thinking i'm) looking for... and here's where i keep getting diverted > away from > the final goal due to trying to follow everything else :/ I've tried something similar recently. After trying to figure out how to use the classes included with FPC and build something on top of them, I resorted to using the old TP/BP compatible object model with TSortedCollection (as provided in unit objects).

The classes unit also has a TCollection, which can be sorted on any field of the collection items ?

Michael.

Tomas Hajny
to Unknown author
Show quoted text 118 lines
On Tue, 10 Jan 2012, Tomas Hajny wrote: > On Tue, January 10, 2012 05:14, waldo kitty wrote: > > > Hi, > > . > . >> are certain enforced limits on these lines... as such, i have defined a >> record >> >> type >> sat_name = string[25]; >> line_data = string[69]; >> two_line = array [1..2] of line_data; >> three_line_data = record >> satname : sat_name; >> satdata : two_line; >> end; >> >> const >> max_sats = 65536; >> >> var >> sat_data : array [1..max_sats] of three_line_data; >> abuf : two_line; >> new_data : three_line_data; >> >> right now i'm loading the array via a simple readln into the array >> elements... >> >> Procedure Input_Satellite(index : word); >> begin >> if not EOF(fsat) then >> begin >> Readln(fsat,sat_data[index].satname); >> Readln(fsat,sat_data[index].satdata[1]); >> Readln(fsat,sat_data[index].satdata[2]); >> end; {if} >> end; {Procedure Input_Satellite} >> >> this is fine for the initial loading but i'm needing to read additional >> TLE >> files and insert those records into the above array... my problem is >> that >> i need >> the array sorted on a sectional value from sat_data[index].satdata[1] >> >> sat_catnr := Copy(abuf[1],3,5); >> and >> new_catnr := Copy(abuf[1],3,5); >> >> which are loaded and compared between sat_data[index].sat_data and >> new_data.sat_data >> >> in the case of collisions (same catnr), i need to choose between the >> greater of >> a pair of values from the stored record (sat_data) and the new record >> (new_data) >> read from a file... >> >> Function Real_Value(buffer : string; >> start,length : integer) : double; >> var >> result : integer; >> answer : double; >> begin >> buffer := Copy(buffer,start,length); >> Convert_Blanks(buffer); >> if buffer = '' then >> buffer := '0'; >> Val(buffer,answer,result); >> if result = 0 then >> Real_Value := answer >> else >> Real_Value := 0.0; >> end; {Function Real_Value} >> >> >> via... >> >> sat_epoch := Real_Value(abuf[1],19,14); >> and >> new_epoch := Real_Value(abuf[1],19,14); >> >> >> which are loaded and compared between sat_data[index].satdata and >> new_data.satdata... >> >> my problem is that i cannot find any similar examples where an array of >> records >> is built, sorted and duplicates are eliminated based on specific >> factors >> of the >> two records being compared... >> >> uncle has been failing me for the last several hours and i'm loosing >> sight >> of >> what i'm trying to get done... i've seen references to using a >> tstringlinst as >> well as something apparently non-existent called a tarray... in all of >> the >> stuff >> i've seen on tstringlinst, there's nothing that i find that is close to >> what i'm >> (thinking i'm) looking for... and here's where i keep getting diverted >> away from >> the final goal due to trying to follow everything else :/ > > I've tried something similar recently. After trying to figure out how to > use the classes included with FPC and build something on top of them, I > resorted to using the old TP/BP compatible object model with > TSortedCollection (as provided in unit objects). The classes unit also has a TCollection, which can be sorted on any field of the collection items ?

When looking at it, using or descending Classes.TCollection for simple data records seemed to be considerably more complex than with Objects.TSortedCollection (I studied our documentation and also searched on WWW for general Delphi examples). I even found an implementation of the TP/BP TSortedCollection for the Delphi model on the Web (based on Delphi classes) which kind of suggested that the original TP/BP objects could provide some additional value (by saving some effort necessary otherwise). Admittedly, it might easily be due to lack of my knowledge, but these were my results.

Tomas

Sven Barth
to waldo kitty: i have a li'l project i've been wanting to do for a really long time... as i'm trying to…
waldo kitty wrote 11 lines
my problem is that i cannot find any similar examples where an array of records is built, sorted and duplicates are eliminated based on specific factors of the two records being compared... uncle has been failing me for the last several hours and i'm loosing sight of what i'm trying to get done... i've seen references to using a tstringlinst as well as something apparently non-existent called a tarray... in all of the stuff i've seen on tstringlinst, there's nothing that i find that is close to what i'm (thinking i'm) looking for... and here's where i keep getting diverted away from the final goal due to trying to follow everything else :/

My suggestion is to use a generic list from unit fgl.

Following example:

type
   TSatDataList = specialize TFPGList<three_line_data>;

In the following I assume that you know how to use classes, if not feel free to ask back ;)

When now reading the file you do the following (based on your code):

Procedure Input_Satellite(aInputFile: TextFile; aList: TSatDataList);
   var
     data: three_line_data;
   begin
   if not EOF(aInputFile) then
     begin
     Readln(aInputFile,data.satname);
     Readln(aInputFile,data.satdata[1]);
     Readln(aInputFile,data.satdata[2]);
     aList.Add(data);
     end; {if}
   end; {Procedure Input_Satellite}

For checking whether a element in the list is the same as another one you can do the following (assuming the new item is called "data"):

duplicate := False;
for i := 0 to aList.Count - 1 do
   if IsSame(aList[i], data) then begin // I don't know how exactly your
check looks like, so I just put a placeholder here :)
     // correct the old data instead of inserting
     aList[i].{...}
     duplicate := True;
     Break; // if you're sure that there is no more dupliacate in the
list you can break here
   end;
if not dupliate then
   aList.Add(data); // no duplicate found so add the new element

For sorting the list you need to define a compare function:

function SatDataCompare(aLeft, aRight: three_line_data): Integer;
// aLeft and aRight are of the same type with which you specialized the
TFPGList
// returns
// -1 if aLeft < aRight
// 0 if aLeft = aRight
// 1 if aLeft > aRight
begin
   // compare the two items and return the correct value
end;
// somewhere else:
aList.Sort(@SatDataCompare);

What I wrote here might not be perfect, but it should give you a start.

Regards,
Sven

waldo kitty
to Sven Barth
Sven Barth wrote 5 lines
// somewhere else: aList.Sort(@SatDataCompare); What I wrote here might not be perfect, but it should give you a start.

thank you very much! it has definitely given me some ideas... one thing that concerns me, though, is that i'm using FPC 2.4.5 console mode... this fgl unit is available in this FPC for this line?

type TSatDataList = specialize TFPGList<three_line_data>;

the other thing i'm trying to figure out is if i can easily and speedily pull portions of the satdata fields for the sort and duplicate functions... i'm thinking that it would be faster to expand the record into each of its actual fields and fill them during import... i think i'd rather copy(x,y,z) to individual record field during import rather than pulling the needed fields during the sorting and merging phases...

and one more trip through the raft of data can adjust the social name for the satellite :)

i'm trying to figure out how to also work out a method of deleting TLEs of birds who are down... another list to process and walk through :P

thanks again for the info!

waldo kitty
to Tomas Hajny: . . I've tried something similar recently. After trying to figure out how to use the…
Tomas Hajny wrote 17 lines
On Tue, January 10, 2012 05:14, waldo kitty wrote: > uncle has been failing me for the last several hours and i'm loosing sight > of > what i'm trying to get done... i've seen references to using a > tstringlinst as > well as something apparently non-existent called a tarray... in all of the > stuff > i've seen on tstringlinst, there's nothing that i find that is close to > what i'm > (thinking i'm) looking for... and here's where i keep getting diverted > away from > the final goal due to trying to follow everything else :/ I've tried something similar recently. After trying to figure out how to use the classes included with FPC and build something on top of them, I resorted to using the old TP/BP compatible object model with TSortedCollection (as provided in unit objects).

yes, i believe i have some hard (l)earned TP code with that... i just hunted it up and it appears that i built my own objects with TSortedCollection... i've got to try to go back over it and see if i can figure it out... it does a lot of digging message bases to assemble linkage paths for the messages' distribution...

i'm also going to take a look at what sven barth has offered... i might be able to knock this out in another few hours IF i can use the specialize thing he pointed me to...

thanks to both you and he for your responses ;)

Sven Barth
to waldo kitty: thank you very much! it has definitely given me some ideas... one thing that concerns me,…
waldo kitty wrote 13 lines
On 1/10/2012 05:39, Sven Barth wrote: > > // somewhere else: > aList.Sort(@SatDataCompare); > > What I wrote here might not be perfect, but it should give you a start. thank you very much! it has definitely given me some ideas... one thing that concerns me, though, is that i'm using FPC 2.4.5 console mode... this fgl unit is available in this FPC for this line? > type > TSatDataList = specialize TFPGList<three_line_data>;

Yes, this feature is available in 2.4.5 (though I would nevertheless suggest to you to update to the newly released 2.6.0 :) )

Show quoted text 8 lines
the other thing i'm trying to figure out is if i can easily and speedily pull portions of the satdata fields for the sort and duplicate functions... i'm thinking that it would be faster to expand the record into each of its actual fields and fill them during import... i think i'd rather copy(x,y,z) to individual record field during import rather than pulling the needed fields during the sorting and merging phases...

Yes, that might indeed be faster or better.

You could also try to experiment with TFPGMap if your satellite data has a unique key. You specialize it like the following:

type
   TYourMap = specialize TFPGMap<YourKeyType, YourKeyData>;

Where YourKeyType is the type of your key (e.g. String, Integer, etc) and YourKeyData is your data record. For more info I'd advice you to take a look at the class itself and its parent class TFPSMap, because I haven't used that class yet.

and one more trip through the raft of data can adjust the social name for the satellite :)

Can't you adjust that when reading the data as well?

i'm trying to figure out how to also work out a method of deleting TLEs of birds who are down... another list to process and walk through :P

Are they listed in another file?
If so then I suggest the following: read the file and for each record read you search in the list for that satellite and do a "YourList.Delete(Index)" where "Index" is your for-loop's counter variable (if a satellite is contained more than once in your list I'd suggest you to use a downwards counting loop from "YourList.Count - 1 downto 0" otherwise you'll get "List index out of bounds"-exceptions if you access the last elements (depending on how many you deleted). Otherwise you can cancel the loop using "Break" again.

thanks again for the info!

You're welcome. That's what we're here for ;)

Regards,
Sven

waldo kitty
to Sven Barth: My suggestion is to use a generic list from unit fgl. Following example: type…
Sven Barth wrote 11 lines
// -1 if aLeft < aRight // 0 if aLeft = aRight // 1 if aLeft > aRight begin // compare the two items and return the correct value end; // somewhere else: aList.Sort(@SatDataCompare); What I wrote here might not be perfect, but it should give you a start.

i've been working with what you sent and have run into a problem i don't understand or how to find and fix...

Error: Operator is not overloaded
You’re trying to use an overloaded operator when it is not overloaded for this type.

i don't know what this is pointing to because it is evidently not in my sources but in a unit i'm loading... the actual compiler message is

(750,50) Error: Operator is not overloaded
satsort.pas(434) Fatal: There were 1 errors compiling module, stopping satsort.pas(0) Fatal: Compilation aborted

i don't know if this is because i wrote an IsSameStr function because i could not find an IsSame one... i actually have two... one checks if two string vars have the same string and the other checks if two double vars have the same double... IsSameStr is boolean for true or false but IsSameReal returns an integer of -1 (var1 < var2), 0 (var1 = var2) or 1 (var1 > var2)...

am i in the right target area?

i should mention my fpc version is

╔═[■]════════════════════ About ═════════════════════════╗ ║ ║ ║ FreePascal IDE for Win32 for i386 ║ ║ Target CPU: i386 ║ ║ Version 1.0.12 2011/06/01 ║ ║ (Compiler Version 2.4.5) ║ ║ ║ ║ Copyright (C) 1998-2009 by ║ ║ ║ ║ Bérczi Gábor ║ ║ Pierre Muller ║ ║ and ║ ║ Peter Vreman ║ ║ OK ▄ ║ ║ ▀▀▀▀▀▀▀▀▀ ║ ╚════════════════════════════════════════════════════════╝

Sven Barth
to waldo kitty
waldo kitty wrote 28 lines
On 1/10/2012 05:39, Sven Barth wrote: > // -1 if aLeft < aRight > // 0 if aLeft = aRight > // 1 if aLeft > aRight > begin > // compare the two items and return the correct value > end; > > // somewhere else: > aList.Sort(@SatDataCompare); > > What I wrote here might not be perfect, but it should give you a start. i've been working with what you sent and have run into a problem i don't understand or how to find and fix... Error: Operator is not overloaded You’re trying to use an overloaded operator when it is not overloaded for this type. i don't know what this is pointing to because it is evidently not in my sources but in a unit i'm loading... the actual compiler message is (750,50) Error: Operator is not overloaded satsort.pas(434) Fatal: There were 1 errors compiling module, stopping satsort.pas(0) Fatal: Compilation aborted

Oh damn it, I forgot about that.

Is it possible for you to update to version 2.6? The problem is the following: the list I mentioned to you uses a equality operator ("=") of two values of the type you specialize with. Thus you need an overloaded operator in case of records. But you can't use a global overloaded operator (which you could do in FPC 2.4.5), as the specialization will only see those types, functions and operators that were visible when the generic (the TFPGList) was declared. So you'll need to use advanced records for that (which are only available from 2.6.0 on).

Here is what you would do if you'd use 2.6.0:

// at the top of your unit/program:
{$mode objfpc} // or whatever mode you use
{$modeswitch advancedrecords} // you don't need this in mode Delphi only
// the declaration of your record (for sake of an example I'll reuse
your three_line_data record)
type
   three_line_data = record
                       satname : sat_name;
                       satdata : two_line;
                       // now comes the important part
                       class operator = (aLeft, aRight: three_line_data)
Result: Boolean; // You need the "Result" only if you're not using mode
Delphi or ObjFPC
                     end;
// later in your program file or in the implementation section of your unit:
class operator three_line_data.= (aLeft, aRight: three_line_data)
Result: Boolean;
begin
   Result := aLeft.satname = aRight.satname; // I don't know whether
this is your real comparison for equality, but it's just an example
end;

Now you should be able to successfully specialize the TFPGList.

If you can't update to 2.6.0:
1. I'd like to know why :)
2. You can then use the TList which is based on pointers (and requires a bit manual housekeeping, but I can help you here as well if you want)

Show quoted text 6 lines
i don't know if this is because i wrote an IsSameStr function because i could not find an IsSame one... i actually have two... one checks if two string vars have the same string and the other checks if two double vars have the same double... IsSameStr is boolean for true or false but IsSameReal returns an integer of -1 (var1 < var2), 0 (var1 = var2) or 1 (var1 > var2)...

The IsSame was merely a placeholder for a function that compares your two satellites records (and that you already seem to have created ;) )

Regards,
Sven

waldo kitty
to Sven Barth: Yes, this feature is available in 2.4.5 (though I would nevertheless suggest to you to…
Sven Barth wrote 17 lines
On 10.01.2012 18:15, waldo kitty wrote: > On 1/10/2012 05:39, Sven Barth wrote: >> >> // somewhere else: >> aList.Sort(@SatDataCompare); >> >> What I wrote here might not be perfect, but it should give you a start. > > thank you very much! it has definitely given me some ideas... one thing > that concerns me, though, is that i'm using FPC 2.4.5 console mode... > this fgl unit is available in this FPC for this line? > > > type > > TSatDataList = specialize TFPGList<three_line_data>; Yes, this feature is available in 2.4.5 (though I would nevertheless suggest to you to update to the newly released 2.6.0 :) )

yes, i've got to figure out how to do that... my windows' installations are pulled from SVN following the procedure on one of the wiki pages but i have a .bat file that is scripted for this process... the only problem is that i generally have to start clean if i want to update fpc because something goes wrong during the make but i don't recall what... i do something almost exactly the same with lazarus (from the same page) but i manually compile it from within lazarus instead of trying from the command line... i'm rather scared to mess up what is currently a working installation :/

Show quoted text 8 lines
> the other thing i'm trying to figure out is if i can easily and speedily > pull portions of the satdata fields for the sort and duplicate > functions... i'm thinking that it would be faster to expand the record > into each of its actual fields and fill them during import... i think > i'd rather copy(x,y,z) to individual record field during import rather > than pulling the needed fields during the sorting and merging phases... Yes, that might indeed be faster or better.

i implemented it by adding two additional fields to the record and filling that data as above... then i use those two fields in the dupe finding and comparison flow... i don't know if it works due to an error i just wrote about in another message in this thread... i'm hoping you or someone else has an idea where to look as i'm completely lost :(

Show quoted text 10 lines
You could also try to experiment with TFPGMap if your satellite data has a unique key. You specialize it like the following: type TYourMap = specialize TFPGMap<YourKeyType, YourKeyData>; Where YourKeyType is the type of your key (e.g. String, Integer, etc) and YourKeyData is your data record. For more info I'd advice you to take a look at the class itself and its parent class TFPSMap, because I haven't used that class yet.

i think i'll leave that alone for now ;)

Show quoted text 5 lines
> and one more trip through the raft of data can adjust the social name > for the satellite :) > Can't you adjust that when reading the data as well?

yes but it is still a loop of several hundred or thousand comparisons for each item in the list... i've not updated the XRF (crossreference) files in almost a year and there have been a lot of new birds launched... the XRF files currently number 37 files (of up to 1000 entries each) containing 15194 entries in total... not all satellites are represented because their common names are fine (by my standards) or they are not flying any more... the only thing this list is really for is to fix up the common names to what i want to use... for example satellite 04321 has two common names... AO-05 and Australis-OSCAR-5... amateur radio folk likely prefer AO-05 whereas myself and others prefer the OSCAR notation... it makes it easier for grouping tasks since not all OSCAR units are the same for the first characters... AO-06 is AMSAT-OSCAR-6 so you can see that the 'A' means different things... there's all kinds of common names, too... debris for certain ones (hundreds of entries for the chinese satellite that was destroyed) with a more informative name, rocket body parts, despin weights, shields, numerous covers, etc... [/rambling talk]

> i'm trying to figure out how to also work out a method of deleting TLEs > of birds who are down... another list to process and walk through :P Are they listed in another file?

they probably will be... the XRF files have two leading spaces (i didn't design the format) and i could probably hijack the first space to put an 'X' if the satellite is no longer flying... then i could use you suggestion below to eliminate those entries from the list...

Show quoted text 7 lines
If so then I suggest the following: read the file and for each record read you search in the list for that satellite and do a "YourList.Delete(Index)" where "Index" is your for-loop's counter variable (if a satellite is contained more than once in your list I'd suggest you to use a downwards counting loop from "YourList.Count - 1 downto 0" otherwise you'll get "List index out of bounds"-exceptions if you access the last elements (depending on how many you deleted). Otherwise you can cancel the loop using "Break" again.

i'll try to remember this and will likely do this during the common name adjustment phase... another field in the record for flying or not and then skip those not flying during the output file writing ;)

Marco van de Voort
to Tomas Hajny: When looking at it, using or descending Classes.TCollection for simple data records…

In our previous episode, Tomas Hajny said:

Show quoted text 15 lines
>> TSortedCollection (as provided in unit objects). > > The classes unit also has a TCollection, which can be sorted on any field > of > the collection items ? When looking at it, using or descending Classes.TCollection for simple data records seemed to be considerably more complex than with Objects.TSortedCollection (I studied our documentation and also searched on WWW for general Delphi examples). I even found an implementation of the TP/BP TSortedCollection for the Delphi model on the Web (based on Delphi classes) which kind of suggested that the original TP/BP objects could provide some additional value (by saving some effort necessary otherwise). Admittedly, it might easily be due to lack of my knowledge, but these were my results.

IIRC the problem with TP collections was that the forall operator used hackery to call a nested procedure. With proper ISO procedure variable support in 2.6.0 that could be done good :-)

waldo kitty
to Sven Barth: Oh damn it, I forgot about that. Is it possible for you to update to version 2.6? The…
Sven Barth wrote 19 lines
On 10.01.2012 20:12, waldo kitty wrote: > i don't know what this is pointing to because it is evidently not in my > sources but in a unit i'm loading... the actual compiler message is > > (750,50) Error: Operator is not overloaded > satsort.pas(434) Fatal: There were 1 errors compiling module, stopping > satsort.pas(0) Fatal: Compilation aborted > Oh damn it, I forgot about that. Is it possible for you to update to version 2.6? The problem is the following: the list I mentioned to you uses a equality operator ("=") of two values of the type you specialize with. Thus you need an overloaded operator in case of records. But you can't use a global overloaded operator (which you could do in FPC 2.4.5), as the specialization will only see those types, functions and operators that were visible when the generic (the TFPGList) was declared. So you'll need to use advanced records for that (which are only available from 2.6.0 on).

hummm... i may not need this specialization, then... i'm not comparing the entire record but portions of the fields in the records... i've made a few changes that should make this easier for me to do... let me show an example of an input record...

ISS (Zarya)
1 25544U 98067A 11355.91136867 .00021672 00000-0 28116-3 0 6701 2 25544 51.6441 296.4760 0024445 190.7247 276.2995 15.58432251750217

this breaks out like this...

line columns example description
0 1-24 ISS (Zarya) common name

1 1 1 element line number 1 3-7 25545 catalog number
1 8 U elset classification 1 10-17 98067A international designator 1 19-32 11355.91136867 epoch (UTC)
1 34-43 .00021672 1st deriv of mean motion 1 45-52 00000-0 2nd deriv of mean motion 1 54-61 28116-3 B* drag term
1 63 0 element set type
1 65-68 670 element number
1 69 1 checksum

2 1 2 element line number 2 3-7 25545 catalog number
2 9-16 51.6441 orbit inclination
2 18-25 296.4760 rt ascension acend node 2 27-33 0024445 eccentricity
2 35-42 190.7247 arg of perigee (degrees) 2 44-51 276.2995 mean anomaly (degrees) 2 53-63 15.58432251 mean motion (revs/day) 2 64-68 75021 rev number at epoch 2 69 3 checksum

so i now have the following to define the record... since i only need the catalog number to check for duplicates and the epoch to determine which one to keep, i've added those two fields to the record and pull them during the reading of the data (see Input_Satellite_List procedure below)...

type
   cat_nbr    = string[5];
   sat_name   = string[25];
   line_data  = string[69];
   two_line   = array [1..2] of line_data;
   three_line_data = record
                       satname : sat_name;
                       satdata : two_line;
                       catnbr  : cat_nbr;
                       epoch   : double;
                     end;
var
   aList      : TSatDataList;

[...]

Function IsSameReal(num1,num2 : double) : integer;
begin
   if (num1 < num2) then IsSameReal := -1
   else
     if (num1 = num2) then IsSameReal := 0
     else
       if (num1 > num2) then IsSameReal := 1;
end;
Function IsSameStr(str1,str2 : cat_nbr) : boolean;
begin
   IsSameStr := str1 = str2;
end;
Procedure Input_Satellite_List(var aInputFile: TextFile);
var
   data : three_line_data;
   dupe : boolean;
   i,x  : integer;
begin
   if not EOF(aInputFile) then
     begin
       Readln(aInputFile,data.satname);
       Readln(aInputFile,data.satdata[1]);
       Readln(aInputFile,data.satdata[2]);
       data.catnbr := Copy(data.satdata[1],3,5);
       data.epoch  := Real_Value(data.satdata[1],19,14);
       dupe := False;
       for i := 0 to aList.Count -1 do
         if IsSameStr(aList[i].catnbr,data.catnbr) then  // make the dupe check
           begin                        // if catnbrs are the same
             dupe := True;              // first mark dupe true
             x := IsSameReal(aList[i].epoch,data.epoch); // compare epochs
             case x of
              -1: begin                 // aList < data, replace with data
                    aList[i] := data;
                  end;
              0 : begin                 // aList = data, ignore data
                  end;
              1 : begin                 // aList > data, ignore data
                  end;
             end; {case}
           end; {if IsSameStr}
       if not dupe then
         aList.Add(data);               // not a dupe so add it
     end; {if not EOF}
end; {Procedure Input Satellite List}

with this not working like this i just need to shove them into a list or collection or something to hold them for more processing and then writing the list out to a plain 7bit ascii text file...

i'm not sure where the equality operator you speak of above comes from or why i need it in play :/

something else i need to make sure of is that this will compile on my OS/2 FPC 2.4.2 since that is the environment it will be under when it goes production... unless, of course, there's a shiney new 0s2260full.zip file for me to play with ;)

this tool i'm working on is to replace an old DOS one with those old memory limits... that old tool takes 20+ minutes to process all the TLEs and then can only put 2797 of them in the output file when there are actually several hundred more that should be in there, too...

should i maybe continue to use an array like i was at first? is an array on the stack or heap? i tried doing something before with setlength to enlarge the array when a new record needed to be added but the sizes weren't right for some reason and it just kept ganking on me so i am coming at this from some other direction ;) is there something better, easier, faster to use?

[TRIM]

Show quoted text 5 lines
Now you should be able to successfully specialize the TFPGList. If you can't update to 2.6.0: 1. I'd like to know why :)

as i wrote in another message, my windows' boxen update from SVN and every time i've tried to update FPC and make whatever, it breaks and i'm scared to break what i currently have working... i always have to blow the directory away, then svn it all back down, and then do the make thing... the ugly part is that i also have lazarus built with and using this svn'd version of FPC so i definitely do not want to be breaking that, either...

a second reason is i don't do the svn thing on the OS/2 box and so have to rely on an install package for that OS... the last time i did this, i unzipped os2242full.zip to /fpc and went from there... i recall having to manually configure a lot of stuff, though... include paths and the like... i still don't have the help working properly in any environment which greatly hampers things :/

2. You can then use the TList which is based on pointers (and requires a bit manual housekeeping, but I can help you here as well if you want)

TList? hummm... better? smaller? faster? easier to use? small memory footprint? i'm game to try things as long as it works in the end ;)

Show quoted text 9 lines
> i don't know if this is because i wrote an IsSameStr function because i > could not find an IsSame one... i actually have two... one checks if two > string vars have the same string and the other checks if two double vars > have the same double... IsSameStr is boolean for true or false but > IsSameReal returns an integer of -1 (var1 < var2), 0 (var1 = var2) or 1 > (var1 > var2)... The IsSame was merely a placeholder for a function that compares your two satellites records (and that you already seem to have created ;) )

that's what i thought :) i'm sure you saw them above?

Sven Barth
to waldo kitty
waldo kitty wrote 31 lines
On 1/10/2012 14:41, Sven Barth wrote: > On 10.01.2012 20:12, waldo kitty wrote: >> i don't know what this is pointing to because it is evidently not in my >> sources but in a unit i'm loading... the actual compiler message is >> >> (750,50) Error: Operator is not overloaded >> satsort.pas(434) Fatal: There were 1 errors compiling module, stopping >> satsort.pas(0) Fatal: Compilation aborted >> > > Oh damn it, I forgot about that. > > Is it possible for you to update to version 2.6? The problem is the > following: > the list I mentioned to you uses a equality operator ("=") of two > values of the > type you specialize with. Thus you need an overloaded operator in case of > records. But you can't use a global overloaded operator (which you > could do in > FPC 2.4.5), as the specialization will only see those types, functions > and > operators that were visible when the generic (the TFPGList) was > declared. So > you'll need to use advanced records for that (which are only available > from > 2.6.0 on). hummm... i may not need this specialization, then... i'm not comparing the entire record but portions of the fields in the records... i've made a few changes that should make this easier for me to do... let me show an example of an input record...

It's not about YOU comparing the entire record. The TFPGList uses that internally for it's own bookkeeping.

The specialization is mostly for type safety and ease of use. As I show you below a simple pointer based TList works as well (if you follow some advices). In the end it should be easier to handle than an array ;)

Show quoted text 40 lines
ISS (Zarya) 1 25544U 98067A 11355.91136867 .00021672 00000-0 28116-3 0 6701 2 25544 51.6441 296.4760 0024445 190.7247 276.2995 15.58432251750217 this breaks out like this... line columns example description 0 1-24 ISS (Zarya) common name 1 1 1 element line number 1 3-7 25545 catalog number 1 8 U elset classification 1 10-17 98067A international designator 1 19-32 11355.91136867 epoch (UTC) 1 34-43 .00021672 1st deriv of mean motion 1 45-52 00000-0 2nd deriv of mean motion 1 54-61 28116-3 B* drag term 1 63 0 element set type 1 65-68 670 element number 1 69 1 checksum 2 1 2 element line number 2 3-7 25545 catalog number 2 9-16 51.6441 orbit inclination 2 18-25 296.4760 rt ascension acend node 2 27-33 0024445 eccentricity 2 35-42 190.7247 arg of perigee (degrees) 2 44-51 276.2995 mean anomaly (degrees) 2 53-63 15.58432251 mean motion (revs/day) 2 64-68 75021 rev number at epoch 2 69 3 checksum so i now have the following to define the record... since i only need the catalog number to check for duplicates and the epoch to determine which one to keep, i've added those two fields to the record and pull them during the reading of the data (see Input_Satellite_List procedure below)...

Wouldn't it be easier for you to parse the complete two lines into a full blown record (just asking). Then you don't need to parse them later on and can just do a "YourRecord.MeanAnomaly" (for example). Using a TStringList with the correct parameters can also remove the burden from you to seperate the values by hand.

Show quoted text 10 lines
Function IsSameReal(num1,num2 : double) : integer; begin if (num1 < num2) then IsSameReal := -1 else if (num1 = num2) then IsSameReal := 0 else if (num1 > num2) then IsSameReal := 1; end;

You can shorten this by using the "CompareValue" function from unit "Math":

function IsSameReal(num1,num2: Double): Integer;
begin
   IsSameReal := CompareValue(num1, num2);
end;

(though I would name that function CompareReal instead of IsSameReal ;) )

i'm not sure where the equality operator you speak of above comes from or why i need it in play :/

It's inside the implementation of TFPGList. I don't remember exactly where, but it's used in there.

Show quoted text 5 lines
something else i need to make sure of is that this will compile on my OS/2 FPC 2.4.2 since that is the environment it will be under when it goes production... unless, of course, there's a shiney new 0s2260full.zip file for me to play with ;)

[...]

Show quoted text 15 lines
as i wrote in another message, my windows' boxen update from SVN and every time i've tried to update FPC and make whatever, it breaks and i'm scared to break what i currently have working... i always have to blow the directory away, then svn it all back down, and then do the make thing... the ugly part is that i also have lazarus built with and using this svn'd version of FPC so i definitely do not want to be breaking that, either... a second reason is i don't do the svn thing on the OS/2 box and so have to rely on an install package for that OS... the last time i did this, i unzipped os2242full.zip to /fpc and went from there... i recall having to manually configure a lot of stuff, though... include paths and the like... i still don't have the help working properly in any environment which greatly hampers things :/

You'll need to ask Tomas Hajny then as he is the one who is most familiar with OS/2. But there seems to indeed be a os2260full.zip at SourceForge :) (So it might be worthwhile to investigate in an update, not only because of generics ;) )

Show quoted text 6 lines
> 2. You can then use the TList which is based on pointers (and requires > a bit > manual housekeeping, but I can help you here as well if you want) TList? hummm... better? smaller? faster? easier to use? small memory footprint? i'm game to try things as long as it works in the end ;)

TList is basically an array wrapped in a class (though I would suggest TFPList as that is faster than TList (TList is based on TFPList)). Both can be found in unit Classes.

The following you need to know: T(FP)List works on pointers, thus you need to
1. declare a pointer type to your record type (e.g. Pthree_line_data = ^three_line_data)
2. allocate a Pthree_line_data variable if you want to add a new entry to the list

E.g.
Procedure Input_Satellite_List(var aInputFile: TextFile);
var
   data : Pthree_line_data;
   ...
begin
   if not EOF(aInputFile) then
   begin
     New(data); // <= allocate the data
     ...
     data^.catnbr := ... // you must dereference the pointer using "^"
to access the fields
     ...
     aList.Add(data);
   end;
end;

3. if you remove an entry from the list you need to dispose the entry:

E.g.

procedure SomeProcWhichMightRemoveAEntry(aList: TList { or TFPList });
begin
   ...
   Dispose(Pthree_line_data(aList[IndexToDelete])); // the cast is
necessary for "Dispose" to work correctly
   ...
end;

4. If you work with a list entry you need to cast it to your pointer type (either using a cast like above in the Dispose or by declaring a local variable which will hold a casted variant)

E.g.

procedure SomeProcWhichWorksWithEntry(aList: TList { or TFPList });
var
   data: Pthree_line_data;
begin
   ...
   // variant 1
   Pthree_line_data(aList[IndexToWorkWith])^.catnbr := ...
   // or
   if Pthree_line_data(aList[IndexToWorkWith])^.catnbr = ... then
   ...
   // variant 2
   data := Pthree_line_data(aList[IndexToWorkWith]);
   data^.catnbr := ...
   // or
   if data^.catnbr = ... then
   ...
end;

Variant 2 might be preferable if you often access the element after each other.

5. At the end of the list's lifetime you need to Dispose all items (or before you clear the list)

E.g.

for i := 0 to aList.Count - 1 do
   Dispose(Pthree_line_data(aList[i]));
aList.Clear; // or aList.Free depending on the context

I hope this helps you along, but feel free to ask more if you need help. :)

Regards,
Sven

Tomas Hajny
to waldo kitty: hummm... i may not need this specialization, then... i'm not comparing the entire record…

On 10 Jan 12, at 16:02, waldo kitty wrote:
.
.

something else i need to make sure of is that this will compile on my OS/2 FPC 2.4.2 since that is the environment it will be under when it goes production... unless, of course, there's a shiney new 0s2260full.zip file for me to play with ;)

...which there is, of course... ;-) (equally to os2244full.zip which you apparently skipped for some reason).

Show quoted text 5 lines
a second reason is i don't do the svn thing on the OS/2 box and so have to rely on an install package for that OS... the last time i did this, i unzipped os2242full.zip to /fpc and went from there... i recall having to manually configure a lot of stuff, though... include paths and the like... i still don't have the help working properly in any environment which greatly hampers things :/

You should consider using the provided installer instead of just manually unpacking the file. Not that it updates the paths for you (but that shouldn't be necessary if you let the installer to update your previous installation in the same directory) but it configures the IDE to use the supplied docs at least (note that the creation of the help index from the supplied HTML docs takes quite some time though).

BTW, the supplied HTML docs cannot be used if installing onto FAT with OS/2 due to the 8.3 restrictions (not that I'd expect you to use FAT with your OS/2 unless having very good reasons, just for completeness sake ;-) ).

Tomas

Marco van de Voort
to Tomas Hajny

In our previous episode, Tomas Hajny said:

BTW, the supplied HTML docs cannot be used if installing onto FAT with OS/2 due to the 8.3 restrictions (not that I'd expect you to use FAT with your OS/2 unless having very good reasons, just for completeness sake ;-) ).

We really should switch the IDE to CHM. Basically installation is copying to a directory and putting a line in the fp.ini

Tomas Hajny
to Marco van de Voort
Marco van de Voort wrote 8 lines
In our previous episode, Tomas Hajny said: > BTW, the supplied HTML docs cannot be used if installing onto FAT > with OS/2 due to the 8.3 restrictions (not that I'd expect you to use > FAT with your OS/2 unless having very good reasons, just for > completeness sake ;-) ). We really should switch the IDE to CHM. Basically installation is copying to a directory and putting a line in the fp.ini

Well, I just played with using CHMs with FP IDE under OS/2 and realized that I get a nice immediate SIGSEGV when using the context help with the CHM help files, whereas everything works fine with both the standard HTML docs and also the INF files created by Graeme (rtl.inf + fcl.inf). :/ It could well be that that this is due to some bug in the OS/2 RTL triggered by different RTL functions used in the CHM support units (i.e. I don't want to blame the CHM support implementation nor its integration to the FP IDE necessarily), but that's what I get right now...

Tomas

waldo kitty
to Sven Barth: It's not about YOU comparing the entire record. The TFPGList uses that internally for…
Sven Barth wrote 8 lines
On 10.01.2012 22:02, waldo kitty wrote: > hummm... i may not need this specialization, then... i'm not comparing > the entire record but portions of the fields in the records... i've made > a few changes that should make this easier for me to do... let me show > an example of an input record... It's not about YOU comparing the entire record. The TFPGList uses that internally for it's own bookkeeping.

ahhh!

The specialization is mostly for type safety and ease of use. As I show you below a simple pointer based TList works as well (if you follow some advices). In the end it should be easier to handle than an array ;)

i never did like pointers... hard to wrap my head around what was what and how to access certain things... but then again, i'm self-taught ;)

Show quoted text 42 lines
> ISS (Zarya) > 1 25544U 98067A 11355.91136867 .00021672 00000-0 28116-3 0 6701 > 2 25544 51.6441 296.4760 0024445 190.7247 276.2995 15.58432251750217 > > > this breaks out like this... > > line columns example description > 0 1-24 ISS (Zarya) common name > > 1 1 1 element line number > 1 3-7 25545 catalog number > 1 8 U elset classification > 1 10-17 98067A international designator > 1 19-32 11355.91136867 epoch (UTC) > 1 34-43 .00021672 1st deriv of mean motion > 1 45-52 00000-0 2nd deriv of mean motion > 1 54-61 28116-3 B* drag term > 1 63 0 element set type > 1 65-68 670 element number > 1 69 1 checksum > > 2 1 2 element line number > 2 3-7 25545 catalog number > 2 9-16 51.6441 orbit inclination > 2 18-25 296.4760 rt ascension acend node > 2 27-33 0024445 eccentricity > 2 35-42 190.7247 arg of perigee (degrees) > 2 44-51 276.2995 mean anomaly (degrees) > 2 53-63 15.58432251 mean motion (revs/day) > 2 64-68 75021 rev number at epoch > 2 69 3 checksum > > > so i now have the following to define the record... since i only need > the catalog number to check for duplicates and the epoch to determine > which one to keep, i've added those two fields to the record and pull > them during the reading of the data (see Input_Satellite_List procedure > below)... Wouldn't it be easier for you to parse the complete two lines into a full blown record (just asking).

in the future, i will want to do this... however, this tool is a first step into mangling^H^H^H^H^H^Hssaging TLE data... the only purpose for this tool is to merge TLE files together into one master file with /ALL/ available satellite TLEs listed... the current old DOS program simply cannot do this due to memory constraints... as such, i only need know the catalog number and the epoch of the two... once the final sorted merged list is completed, it is all going to be written back to a plain 7bit ascii text file like it all originated from... this text file is then used for my other satellite and astronomy related applications ;)

Then you don't need to parse them later on and can just do a "YourRecord.MeanAnomaly" (for example). Using a TStringList with the correct parameters can also remove the burden from you to seperate the values by hand.

at this time, this only needs be done with these two (sub)fields in the TLE entries... i'm only interested in these two but am storing everything to make it easier to later run thru the list or collection or array and write the three original lines to a master TLE text file...

Show quoted text 16 lines
> Function IsSameReal(num1,num2 : double) : integer; > > begin > if (num1 < num2) then IsSameReal := -1 > else > if (num1 = num2) then IsSameReal := 0 > else > if (num1 > num2) then IsSameReal := 1; > end; You can shorten this by using the "CompareValue" function from unit "Math": function IsSameReal(num1,num2: Double): Integer; begin IsSameReal := CompareValue(num1, num2); end;

interesting! in fact, i don't really even need "IsSameReal" as i can simply use CompareValue directly with these two numbers?

(though I would name that function CompareReal instead of IsSameReal ;) )

hahaha! true :)

Show quoted text 25 lines
> something else i need to make sure of is that this will compile on my > OS/2 FPC 2.4.2 since that is the environment it will be under when it > goes production... unless, of course, there's a shiney new > 0s2260full.zip file for me to play with ;) > [...] > > as i wrote in another message, my windows' boxen update from SVN and > every time i've tried to update FPC and make whatever, it breaks and i'm > scared to break what i currently have working... i always have to blow > the directory away, then svn it all back down, and then do the make > thing... the ugly part is that i also have lazarus built with and using > this svn'd version of FPC so i definitely do not want to be breaking > that, either... > > a second reason is i don't do the svn thing on the OS/2 box and so have > to rely on an install package for that OS... the last time i did this, i > unzipped os2242full.zip to /fpc and went from there... i recall having > to manually configure a lot of stuff, though... include paths and the > like... i still don't have the help working properly in any environment > which greatly hampers things :/ You'll need to ask Tomas Hajny then as he is the one who is most familiar with OS/2. But there seems to indeed be a os2260full.zip at SourceForge :) (So it might be worthwhile to investigate in an update, not only because of generics ;) )

it is something i'll have to play with... my windows' boxen should be all set for me to pick and choose which FPC version i use... i've not (yet) done similar for lazarus but that's for another place ;)

Show quoted text 10 lines
>> 2. You can then use the TList which is based on pointers (and requires >> a bit >> manual housekeeping, but I can help you here as well if you want) > > TList? hummm... better? smaller? faster? easier to use? small memory > footprint? i'm game to try things as long as it works in the end ;) TList is basically an array wrapped in a class (though I would suggest TFPList as that is faster than TList (TList is based on TFPList)). Both can be found in unit Classes.

ok... this is another one of the several possible solutions i've been looking at... uncle google has been showing me a lot but i don't seem to find much that has a decent example that's close enough to my needs to whack it into submission...

The following you need to know: T(FP)List works on pointers, thus you need to 1. declare a pointer type to your record type (e.g. Pthree_line_data = ^three_line_data) 2. allocate a Pthree_line_data variable if you want to add a new entry to the list

this looks vaguely similar to what i did in TP/BP with a TSortedCollection which built a sorted list of paths that messages took during their transit from the originating machine to my machine... those paths were all numeric and there was only one path per "record" so it was much easier to work with than what i'm flailing about with now...

[TRIM]

Show quoted text 10 lines
5. At the end of the list's lifetime you need to Dispose all items (or before you clear the list) E.g. for i := 0 to aList.Count - 1 do Dispose(Pthree_line_data(aList[i])); aList.Clear; // or aList.Free depending on the context I hope this helps you along, but feel free to ask more if you need help. :)

this is looking kinda familiar, really... the last time i did this, ie: the message path program mentioned above, i went on "vacation" to the mountains and spent the whole 7 days holed up with my pascal books, a computer, and a lot of sample data to work with... i did get the task accomplished but never really got to enjoy my "vacation in the mountains"... at least not more than simply getting away where there were no distractions...

"i'll be bach" :P

waldo kitty
to Sven Barth: It's not about YOU comparing the entire record. The TFPGList uses that internally for…
Sven Barth wrote 12 lines
On 10.01.2012 22:02, waldo kitty wrote: > TList? hummm... better? smaller? faster? easier to use? small memory > footprint? i'm game to try things as long as it works in the end ;) TList is basically an array wrapped in a class (though I would suggest TFPList as that is faster than TList (TList is based on TFPList)). Both can be found in unit Classes. The following you need to know: T(FP)List works on pointers, thus you need to 1. declare a pointer type to your record type (e.g. Pthree_line_data = ^three_line_data) 2. allocate a Pthree_line_data variable if you want to add a new entry to the list

i tried this but could only get so far and then not further so i backed up and punted the ball... now i'm trying this with a sortedcollection and while i can apparently insert items, i haven't figured out how to access them and print their contents... unless their contents are garbage like the following prints out...

program satsort;
uses {$IFDEF DEBUG}heaptrc,{$ENDIF}objects,math,crt,dos;
type
   vector     = array [1..4] of double;
   cat_nbr    = string[5];
   sat_name   = string[25];
   line_data  = string[69];
   two_line   = array [1..2] of line_data;
   PTLERec    = ^TTLERec;
   TTLERec    = record
                  satname : sat_name;
                  satdata : two_line;
                  catnbr  : cat_nbr;
                  epoch   : double;
                end;
   TTLERecObj = object(TObject)
                  TLEData : PTLERec;
                  constructor Init;
                  destructor Done; virtual;
                end;
   constructor TTLERecObj.Init;
   begin
     new(TLEData);
   end;
   destructor TTLERecObj.Done;
   begin
     dispose(TLEData);
   end;
type
   PTLECollection = ^TTLECollection;
   TTLECollection = object(TSortedCollection)
     TLERec : TTLERecObj;
     function Compare(Key1, Key2 : Pointer): longint; virtual;
   end;
function TTLECollection.Compare(key1, key2: pointer): longint;
begin
   if PTLERec(key1)^.catnbr = PTLERec(key2)^.catnbr then
     compare := 0
   else if PTLERec(key1)^.catnbr < PTLERec(key2)^.catnbr then
     compare := -1
   else
     compare := 1;
end;
const
   data_type : byte = 3;
//  max_sats = {$IFDEF FPC}65536{$ELSE}250{$ENDIF};
var
   data_drive,data_dir,
   work_drive,work_dir  : string;
   UTC_offset           : double;
   DST                  : boolean;
   fsat,fobs            : text;
   obs_name             : string[25];
   piss                 : vector;
   gigo                 : char;
   my_sats              : word;
   aTLERec              : PTLERec;
   aTLECollection       : PTLECollection;

[...]

Function Input_Satellite_List(var aInputFile: TextFile) : word;
var
   data    : PTLERec;
   sat_cnt : word;
   i,x     : longint;
begin
   sat_cnt := 0;
   while not EOF(aInputFile) do
     begin
       new(data);
       Readln(aInputFile,data^.satname);
       Readln(aInputFile,data^.satdata[1]);
       Readln(aInputFile,data^.satdata[2]);
       data^.catnbr := Copy(data^.satdata[1],3,5);
       data^.epoch  := Real_Value(data^.satdata[1],19,14);
       inc(sat_cnt);
       aTLECollection^.insert(data);
       dispose(data);
       {$IFDEF DEBUG}
       writeln(PTLERec(aTLECollection)^.satname);
       writeln(PTLERec(aTLECollection)^.satdata[1]);
       writeln(PTLERec(aTLECollection)^.satdata[2]);
       writeln(PTLERec(aTLECollection)^.catnbr);
       writeln(PTLERec(aTLECollection)^.epoch);
       {$ENDIF}
     end; {while not EOF}
   Input_Satellite_List := sat_cnt;
end; {Procedure Input Satellite List}
Sven Barth
to waldo kitty
waldo kitty wrote 23 lines
On 1/10/2012 16:41, Sven Barth wrote: > On 10.01.2012 22:02, waldo kitty wrote: >> TList? hummm... better? smaller? faster? easier to use? small memory >> footprint? i'm game to try things as long as it works in the end ;) > > TList is basically an array wrapped in a class (though I would suggest > TFPList > as that is faster than TList (TList is based on TFPList)). Both can be > found in > unit Classes. > > The following you need to know: T(FP)List works on pointers, thus you > need to > 1. declare a pointer type to your record type (e.g. Pthree_line_data = > ^three_line_data) > 2. allocate a Pthree_line_data variable if you want to add a new entry > to the list i tried this but could only get so far and then not further so i backed up and punted the ball... now i'm trying this with a sortedcollection and while i can apparently insert items, i haven't figured out how to access them and print their contents... unless their contents are garbage like the following prints out...

You know that you're handling with pointer here as well? ;)

I never used TP's T(Sorted)Collection, so the following are mostly guesses, but I'll try.

Here is how you should (in my opinion) access an item at index Idx in the list:

var
   rec: PTLERec;
begin
   ...
   rec := PTLERec(aTLECollection^.At(Idx));
   // You can now access the record's contents by dereferencing:
   if rec^.catnbr > ... then
   ...
end;
Show quoted text 9 lines
function TTLECollection.Compare(key1, key2: pointer): longint; begin if PTLERec(key1)^.catnbr = PTLERec(key2)^.catnbr then compare := 0 else if PTLERec(key1)^.catnbr < PTLERec(key2)^.catnbr then compare := -1 else compare := 1; end;

You declared TTLERec.catnbr as a string, so you can't use "<" or ">" for comparison. You'd need to convert it to a ordinal type first.

Show quoted text 20 lines
Function Input_Satellite_List(var aInputFile: TextFile) : word; var data : PTLERec; sat_cnt : word; i,x : longint; begin sat_cnt := 0; while not EOF(aInputFile) do begin new(data); Readln(aInputFile,data^.satname); Readln(aInputFile,data^.satdata[1]); Readln(aInputFile,data^.satdata[2]); data^.catnbr := Copy(data^.satdata[1],3,5); data^.epoch := Real_Value(data^.satdata[1],19,14); inc(sat_cnt); aTLECollection^.insert(data); dispose(data);

Don't do this! You'll free the memory you allocated for your record. The collection will only contain a pointer to this data! (Many of the rules I mentioned for T(FP)List apply here as well)

Show quoted text 7 lines
{$IFDEF DEBUG} writeln(PTLERec(aTLECollection)^.satname); writeln(PTLERec(aTLECollection)^.satdata[1]); writeln(PTLERec(aTLECollection)^.satdata[2]); writeln(PTLERec(aTLECollection)^.catnbr); writeln(PTLERec(aTLECollection)^.epoch); {$ENDIF}

Ehm... you know that it can't work if you cast a TTLECollection pointer to a TTLERec pointer? These are two different structures. No wonder why you only see garbage as you most likely see the contents of the Collection (which consists mostly of pointers). See at the top of the mail how you'd access the contents correctly. Normally I'd suggest you to simply guery the "Count - 1"th item, but in this exact case you'd need to search the item again as you're using a sorting list and the last inserted item might not be at the last position of the list (as Insert maintains the order of the list).

Regards,
Sven

waldo kitty
to Sven Barth
Sven Barth wrote 8 lines
Am 11.01.2012 09:27, schrieb waldo kitty: > i tried this but could only get so far and then not further so i backed > up and punted the ball... now i'm trying this with a sortedcollection > and while i can apparently insert items, i haven't figured out how to > access them and print their contents... unless their contents are > garbage like the following prints out... You know that you're handling with pointer here as well? ;)

yes...

I never used TP's T(Sorted)Collection, so the following are mostly guesses, but I'll try.

thanks... i finally got something working before reading your message... i ended up digging out my old Tom Swan "Mastering Turbo Pascal 6" book... there was a clear example that i was able to follow with a few tweaks for my data structure... it was easier to flip the pages back and forth while looking than to keep trying to jump between different windows on the screen... gimme a nice paperback book like this one any day! ;)

in any case, i'm needing a touch more assistance if you or anyone else are willing to give it... code at the end... questions coming up now...

1. right now the compare is working on the catalog number (TTLERec.catnbr) and with duplicates:=FALSE there are no duplicates... however, i need to be able to choose which record to keep when there is a duplicate catnbr but the epoch (TTLERec.epoch) is different... right now it is throwing out all but the first... how can i tell it how to decide which one to throw away? i saw earlier that one of the parent objects has neat functions like AtPut which would easily allow me to overwrite an existing record that's too old... i just don't know if i can use that in the middle of an insert or a search or just exactly where i would even put code to do this...

2. something else i'm running into is with duplicates:=FALSE, there's a whole bucket load of records that are not disposed of when i wipe out the collection... heaptrc hollers right nasty to me about'em on exit... i can only assume that these are the duplicates but i don't understand why they are still hanging around if insert or add threw them away already :/

[TRIM]

Show quoted text 9 lines
> data^.catnbr := Copy(data^.satdata[1],3,5); > data^.epoch := Real_Value(data^.satdata[1],19,14); > inc(sat_cnt); > aTLECollection^.insert(data); > dispose(data); Don't do this! You'll free the memory you allocated for your record. The collection will only contain a pointer to this data! (Many of the rules I mentioned for T(FP)List apply here as well)

uh? for some reason i thought that the insert was copying the data to a place and then setting the pointer to there for the collection... i tried with and without the dispose(data) but it still looked the same... of course i was tired and might not have been looking at the right writeln debug output...

Show quoted text 8 lines
> writeln(PTLERec(aTLECollection)^.catnbr); > writeln(PTLERec(aTLECollection)^.epoch); > {$ENDIF} Ehm... you know that it can't work if you cast a TTLECollection pointer to a TTLERec pointer? These are two different structures. No wonder why you only see garbage as you most likely see the contents of the Collection (which consists mostly of pointers).

i told ya i get confused with pointers at times :lol: i know it is a pointer but what's under it and how to get to it? :laugh: and with no easily found clear examples like i was able to find in my book, well... i should mention that it was also like very early in the morning and i hadn't slept yet ;)

like i wrote above, i have it working now... i just need to get the duplicates elimination to keep the right one... this thing is fast, too! one goal accomplished :P

Show quoted text 5 lines
See at the top of the mail how you'd access the contents correctly. Normally I'd suggest you to simply guery the "Count - 1"th item, but in this exact case you'd need to search the item again as you're using a sorting list and the last inserted item might not be at the last position of the list (as Insert maintains the order of the list).

yeah, that was a debug attempt to read what was just put in... i resorted to filling the collection and then doing a foreach on them in a later step... now i have some output files that i can compare and see what's happening...

i've saved all the posts in this thread for reference... i know there's some good tidbits in them... once i get more used to my code, i might actually see more of what you were showing me :)

so... here's my current batch of code... it ain't pretty, its my brother ;)

program satsort;
uses {$IFDEF DEBUG}heaptrc,{$ENDIF}strings,objects,math,crt,dos;
type
   Tvector    = array [1..4] of double;
   Tcat_nbr   = pstring;
   Tsat_name  = pstring;
   Tline_data = pstring;
   PTLERec    = ^TTLERec;
   TTLERec    = object(TObject)
                  satname  : Tsat_name;
                  satdata1 : Tline_data;
                  satdata2 : Tline_data;
                  catnbr   : Tcat_nbr;
                  epoch    : double;
                  constructor Init(sname,sdata1,sdata2,cnbr:string; edate:double);
                  destructor Done; virtual;
                end;
   PTLEColl   = ^TTLEColl;
   TTLEColl   = object(TSortedCollection)
                  function Compare(Key1,Key2:Pointer):sw_integer; virtual;
                  function KeyOf(Item:Pointer):Pointer; virtual;
                end;
// TTLERec
constructor TTLERec.Init(sname,sdata1,sdata2,cnbr:string; edate:double);
begin
   satname  := NewStr(sname);
   satdata1 := NewStr(sdata1);
   satdata2 := NewStr(sdata2);
   catnbr   := NewStr(cnbr);
   epoch    := edate;
end;
destructor TTLERec.Done;
begin
   dispose(satname);
   dispose(satdata1);
   dispose(satdata2);
   dispose(catnbr);
end;
// TTLEColl
function TTLEColl.Compare(key1,key2:pointer):sw_integer;
begin
   compare := strcomp(key1,key2);
end;
function TTLEColl.KeyOf(Item:Pointer):Pointer;
begin
   KeyOf := PTLERec(Item)^.catnbr;
end;
const
   data_type : byte = 3;
var
   TLENewName,
   data_drive,data_dir,
   work_drive,work_dir  : string;
   UTC_offset           : double;
   DST,allowdupes       : boolean;
   fsat,fobs            : text;
   obs_name             : string[25];
   piss                 : Tvector;
   gigo                 : char;
   my_sats              : longint;
   aTLERec              : PTLERec;
   aTLEColl             : PTLEColl;

[...]

Procedure ShowOneRec(p: PTLERec; var oFile: text);
begin
   writeln(oFile,p^.satname^);
   writeln(oFile,p^.satdata1^);
   writeln(oFile,p^.satdata2^);
//  writeln(oFile,p^.catnbr^);
//  writeln(oFile,p^.epoch);
end;
Procedure PrintRecords(db: PCollection; oFN : string);
var
   outFile : text;
   Procedure PrintOneRec(p: PTLERec); far;
   begin
     ShowOneRec(p, outFile);
   end;
begin
   assign(outFile,oFN);
   rewrite(outFile);
   db^.ForEach(@PrintOneRec);
   close(outFile)
end;
Function Input_Satellite_List(FN: string) : longint;
var
   MySatName  : string;
   MySatData1 : string;
   MySatData2 : string;
   MyCatNbr   : string;
   MyEpoch    : double;
   sat_cnt    : longint;
begin
   sat_cnt := 0;
   Assign(fsat,fn);
   Reset(fsat);
   writeln('file '+fn+' opened...');
   while not EOF(fsat) do
     begin
       Readln(fsat,MySatName);
       Readln(fsat,MySatData1);
       Readln(fsat,MySatData2);
       MyCatNbr := Copy(MySatData1,3,5);
       MyEpoch  := Real_Value(MySatData1,19,14);
       inc(sat_cnt);
       aTLEColl^.insert(New(PTLERec, Init(MySatName, MySatData1, MySatData2,
MyCatNbr, MyEpoch)));
     end; {while not EOF}
   Close(fsat);
   Input_Satellite_List := sat_cnt;
end; {Procedure Input Satellite List}

[...]

BEGIN
   my_sats := 0;
   allowdupes := FALSE;
   writeln('initializing...');
   program_initialize('satsort');
   // initialize the list collection
   aTLEColl := New(PTLEColl, Init(2048,16));
   if aTLEColl = nil then
     begin
       writeln('*** ERROR: Can''t create TCollection');
       exit;
     end;
   aTLEColl^.duplicates := allowdupes;
   // load tles from all the available tle txt files
   load_files(data_drive+data_dir,'*.txt');
   // save current records in collection
   writeln;
   writeln('saving TLE collection to '+TLENewName);
   PrintRecords(aTLEColl,TLENewName);
   writeln('hit enter to continue...');
   repeat until keypressed;
   gigo := readkey;
   // time to go
   writeln;
   writeln('cleaning up and heading out...');
   writeln;
   writeln('aTLEColl^.FreeAll');
   aTLEColl^.FreeAll;
   writeln;
   writeln('dispose(aTLEColl,done)');
   dispose(aTLEColl,done);
   writeln('hit enter to terminate...');
   repeat until keypressed;
   gigo := readkey;
   program_end;
END.
Tomas Hajny
to waldo kitty

On 11 Jan 12, at 17:46, waldo kitty wrote:
.
.

Show quoted text 9 lines
1. right now the compare is working on the catalog number (TTLERec.catnbr) and with duplicates:=FALSE there are no duplicates... however, i need to be able to choose which record to keep when there is a duplicate catnbr but the epoch (TTLERec.epoch) is different... right now it is throwing out all but the first... how can i tell it how to decide which one to throw away? i saw earlier that one of the parent objects has neat functions like AtPut which would easily allow me to overwrite an existing record that's too old... i just don't know if i can use that in the middle of an insert or a search or just exactly where i would even put code to do this...

The best solution is probably overriding the Insert method to fit your slightly modified logic. Copy the default implementation of TSortedCollection.Insert as found in objects.pp in FPC source tree and modify it according your needs (if Search returns true, perform the additional check for the epoch and depending on the result either use AtInsert (equally to the default implementation) or Dispose the previously stored record at position I ("I" returned by Search, "At (I)" used to access the previously stored record) and then use AtPut.

Show quoted text 5 lines
2. something else i'm running into is with duplicates:=FALSE, there's a whole bucket load of records that are not disposed of when i wipe out the collection... heaptrc hollers right nasty to me about'em on exit... i can only assume that these are the duplicates but i don't understand why they are still hanging around if insert or add threw them away already :/

If you already had them in the collection, they're not added again. You only dispose records added to the collection at the end, but these are lost this way. You can also sort this out in the overridden Insert method (if you don't want to use the newly created record, dispose it).

Show quoted text 15 lines
[TRIM] >> data^.catnbr := Copy(data^.satdata[1],3,5); >> data^.epoch := Real_Value(data^.satdata[1],19,14); >> inc(sat_cnt); >> aTLECollection^.insert(data); >> dispose(data); > > Don't do this! You'll free the memory you allocated for your record. The > collection will only contain a pointer to this data! (Many of the rules I > mentioned for T(FP)List apply here as well) uh? for some reason i thought that the insert was copying the data to a place and then setting the pointer to there for the collection... i tried with and without the dispose(data) but it still looked the same... of course i was tired and might not have been looking at the right writeln debug output...

.
.

No, Insert doesn't do any copying by default. In your current code, the copying is performed by calling the PTLERec constructor (in "New(PTLERec, Init(...") but if the pointer isn't inserted, it is thrown away currently. If you use the overridden Insert method as suggested above, you can Dispose it from within the Insert call if you don't need to insert it.

Tomas

waldo kitty
to Tomas Hajny
Tomas Hajny wrote 21 lines
On 11 Jan 12, at 17:46, waldo kitty wrote: . . > 1. right now the compare is working on the catalog number (TTLERec.catnbr) and > with duplicates:=FALSE there are no duplicates... however, i need to be able to > choose which record to keep when there is a duplicate catnbr but the epoch > (TTLERec.epoch) is different... right now it is throwing out all but the > first... how can i tell it how to decide which one to throw away? i saw earlier > that one of the parent objects has neat functions like AtPut which would easily > allow me to overwrite an existing record that's too old... i just don't know if > i can use that in the middle of an insert or a search or just exactly where i > would even put code to do this... The best solution is probably overriding the Insert method to fit your slightly modified logic. Copy the default implementation of TSortedCollection.Insert as found in objects.pp in FPC source tree and modify it according your needs (if Search returns true, perform the additional check for the epoch and depending on the result either use AtInsert (equally to the default implementation) or Dispose the previously stored record at position I ("I" returned by Search, "At (I)" used to access the previously stored record) and then use AtPut.

i've been mulling this over for a few hours and i'm wondering if overriding the insert method or simply doing the lofic in my importation routine would be best... right now my code blindly inserts the record but i could do a search for a matching "key" (catnbr by keyof ??) and then if i find a match, look at the epoch of the two records, one in vars and the other in the collection... if the one in the vars is newer, then do an AtPut to overwrite the one currently stored... otherwise, just go on to read the next file record into the vars and check it from there...

without looking at the "code to copy" if i want to override the insert method, it almost seems that there's a bug if it just throws away the record we're trying to insert... it would seem that if the code locates a "duplicate" record, it would properly dispose of unwanted data... unless i perform the previously mentioned logic checks manually to catch this, i don't see this happening... the number of records left in memory at the end of the program's execution is the same number as those not added because they are "dupes"...

Show quoted text 11 lines
> 2. something else i'm running into is with duplicates:=FALSE, there's a whole > bucket load of records that are not disposed of when i wipe out the > collection... heaptrc hollers right nasty to me about'em on exit... i can only > assume that these are the duplicates but i don't understand why they are still > hanging around if insert or add threw them away already :/ If you already had them in the collection, they're not added again. You only dispose records added to the collection at the end, but these are lost this way. You can also sort this out in the overridden Insert method (if you don't want to use the newly created record, dispose it).

as above, i don't know that they are /in/ the collection... i'm (currently) simply calling the insert method and leaving the work up to it... if it should be handling this gracefully, it isn't... at least not in a way that heaptrc likes ;)

Show quoted text 22 lines
> [TRIM] >>> data^.catnbr := Copy(data^.satdata[1],3,5); >>> data^.epoch := Real_Value(data^.satdata[1],19,14); >>> inc(sat_cnt); >>> aTLECollection^.insert(data); >>> dispose(data); >> >> Don't do this! You'll free the memory you allocated for your record. The >> collection will only contain a pointer to this data! (Many of the rules I >> mentioned for T(FP)List apply here as well) > > uh? for some reason i thought that the insert was copying the data to a place > and then setting the pointer to there for the collection... i tried with and > without the dispose(data) but it still looked the same... of course i was tired > and might not have been looking at the right writeln debug output... . . No, Insert doesn't do any copying by default. In your current code, the copying is performed by calling the PTLERec constructor (in "New(PTLERec, Init(...") but if the pointer isn't inserted, it is thrown away currently.

thrown away by the default object insert code? why doesn't it properly wipe it out since it is tossing the pointer into the bitbucket?

If you use the overridden Insert method as suggested above, you can Dispose it from within the Insert call if you don't need to insert it.

understood... and with an eye on the above discussion about doing the logic myself using the available routines, i could also take care of it there by deciding to AtPut, or, insert or dispose instead of blindly performing an insert and hoping for the best??

i'll probably have broken my code by the time you read this... but i'll very likely be attempting to implement the logic in my Input_Satellite_List routine ;) OB-)

i just gotta figure out the way that the search and such work as in if they are returning pointers to the record or the record data itself... pointers are still very alien to me :? :/

Sven Barth
to waldo kitty
waldo kitty wrote 8 lines
On 1/11/2012 20:35, Tomas Hajny wrote: > No, Insert doesn't do any copying by default. In your current code, > the copying is performed by calling the PTLERec constructor (in > "New(PTLERec, Init(...") but if the pointer isn't inserted, it is > thrown away currently. thrown away by the default object insert code? why doesn't it properly wipe it out since it is tossing the pointer into the bitbucket?

It can not make any assumption about your pointer. In theory your collection could contain Integers that were simply casted to Pointers. In that case a "disposing of the pointer's data" would be fatal!

i just gotta figure out the way that the search and such work as in if they are returning pointers to the record or the record data itself... pointers are still very alien to me :? :/

The methods return Pointers which you can cast to PTLERec. This you can then access like a usual object (e.g. dereferencing, etc)

Regards,
Sven

Tomas Hajny
to waldo kitty: i've been mulling this over for a few hours and i'm wondering if overriding the insert…
waldo kitty wrote 44 lines
On 1/11/2012 20:35, Tomas Hajny wrote: > On 11 Jan 12, at 17:46, waldo kitty wrote: > . > . >> 1. right now the compare is working on the catalog number >> (TTLERec.catnbr) and >> with duplicates:=FALSE there are no duplicates... however, i need to be >> able to >> choose which record to keep when there is a duplicate catnbr but the >> epoch >> (TTLERec.epoch) is different... right now it is throwing out all but >> the >> first... how can i tell it how to decide which one to throw away? i saw >> earlier >> that one of the parent objects has neat functions like AtPut which >> would easily >> allow me to overwrite an existing record that's too old... i just don't >> know if >> i can use that in the middle of an insert or a search or just exactly >> where i >> would even put code to do this... > > The best solution is probably overriding the Insert method to fit > your slightly modified logic. Copy the default implementation of > TSortedCollection.Insert as found in objects.pp in FPC source tree > and modify it according your needs (if Search returns true, perform > the additional check for the epoch and depending on the result either > use AtInsert (equally to the default implementation) or Dispose the > previously stored record at position I ("I" returned by Search, "At > (I)" used to access the previously stored record) and then use AtPut. i've been mulling this over for a few hours and i'm wondering if overriding the insert method or simply doing the lofic in my importation routine would be best... right now my code blindly inserts the record but i could do a search for a matching "key" (catnbr by keyof ??) and then if i find a match, look at the epoch of the two records, one in vars and the other in the collection... if the one in the vars is newer, then do an AtPut to overwrite the one currently stored... otherwise, just go on to read the next file record into the vars and check it from there...

Yes, you can certainly do that. The Insert method does both the Search call and checking the Duplicates field already. If you do it on your side (rather than overriding the Insert method), the same calls will be performed twice in some cases (in particular if the inserting is really necessary). However, both approaches are certainly possible.

Show quoted text 6 lines
without looking at the "code to copy" if i want to override the insert method, it almost seems that there's a bug if it just throws away the record we're trying to insert... it would seem that if the code locates a "duplicate" record, it would properly dispose of unwanted data... unless i perform the

No. There are two tasks. One is dynamic allocation of the object, the other is insertion. Although you perform both tasks on one line in your program, these are two distinct tasks. The insert code cannot know whether you still may need the (previously allocated) object or not in the very general case. The insert code doesn't throw anything away - you do it by not storing the result of the "New (PTLERec, Init (..." to some variable and only send it as a parameter to a method which may or may not store it somewhere else.

Show quoted text 6 lines
previously mentioned logic checks manually to catch this, i don't see this happening... the number of records left in memory at the end of the program's execution is the same number as those not added because they are "dupes"...

There are certainly different approaches possible. Yes, you could e.g. have a static object (allocated on stack or as global data), populate it with the values read from the file first, search the collection using this static object and only allocate a new one dynamically if you need to insert it.

Show quoted text 21 lines
>> 2. something else i'm running into is with duplicates:=FALSE, there's a >> whole >> bucket load of records that are not disposed of when i wipe out the >> collection... heaptrc hollers right nasty to me about'em on exit... i >> can only >> assume that these are the duplicates but i don't understand why they >> are still >> hanging around if insert or add threw them away already :/ > > If you already had them in the collection, they're not added again. > You only dispose records added to the collection at the end, but > these are lost this way. You can also sort this out in the overridden > Insert method (if you don't want to use the newly created record, > dispose it). as above, i don't know that they are /in/ the collection... i'm (currently) simply calling the insert method and leaving the work up to it... if it should be handling this gracefully, it isn't... at least not in a way that heaptrc likes ;)

It behaves as specified; the responsibility for disposing the duplicates is on your side (since you allocate them in your code also).

Show quoted text 31 lines
>> [TRIM] >>>> data^.catnbr := Copy(data^.satdata[1],3,5); >>>> data^.epoch := Real_Value(data^.satdata[1],19,14); >>>> inc(sat_cnt); >>>> aTLECollection^.insert(data); >>>> dispose(data); >>> >>> Don't do this! You'll free the memory you allocated for your record. >>> The >>> collection will only contain a pointer to this data! (Many of the >>> rules I >>> mentioned for T(FP)List apply here as well) >> >> uh? for some reason i thought that the insert was copying the data to a >> place >> and then setting the pointer to there for the collection... i tried >> with and >> without the dispose(data) but it still looked the same... of course i >> was tired >> and might not have been looking at the right writeln debug output... > . > . > > No, Insert doesn't do any copying by default. In your current code, > the copying is performed by calling the PTLERec constructor (in > "New(PTLERec, Init(...") but if the pointer isn't inserted, it is > thrown away currently. thrown away by the default object insert code? why doesn't it properly wipe it out since it is tossing the pointer into the bitbucket?

Hopefully clarified above (you throw it away by not storing the pointer at the allocation time).

Show quoted text 10 lines
> If you use the overridden Insert method as > suggested above, you can Dispose it from within the Insert call if > you don't need to insert it. understood... and with an eye on the above discussion about doing the logic myself using the available routines, i could also take care of it there by deciding to AtPut, or, insert or dispose instead of blindly performing an insert and hoping for the best??

Yes - it's up to you to decide how to structure the task.

Show quoted text 5 lines
i'll probably have broken my code by the time you read this... but i'll very likely be attempting to implement the logic in my Input_Satellite_List routine ;) OB-)

Possible, but likely resulting in some useless overhead (computing performance-wise) if you still intend to use the method Insert in that case.

Show quoted text 5 lines
i just gotta figure out the way that the search and such work as in if they are returning pointers to the record or the record data itself... pointers are still very alien to me :? :/

Understood - powerful, but the power brings some complexity indeed.

Tomas

waldo kitty
to Tomas Hajny
Tomas Hajny wrote
On Thu, January 12, 2012 03:34, waldo kitty wrote:

[TRIM]

Show quoted text 14 lines
> without looking at the "code to copy" if i want to override the insert > method, it almost seems that there's a bug if it just throws away the > record we're trying to insert... it would seem that if the code locates > a "duplicate" record, it would properly dispose of unwanted data... > unless i perform the No. There are two tasks. One is dynamic allocation of the object, the other is insertion. Although you perform both tasks on one line in your program, these are two distinct tasks. The insert code cannot know whether you still may need the (previously allocated) object or not in the very general case. The insert code doesn't throw anything away - you do it by not storing the result of the "New (PTLERec, Init (..." to some variable and only send it as a parameter to a method which may or may not store it somewhere else.

AHH! i think i understand this now... you are saying that instead of doing this...

aTLEColl^.insert(New(PTLERec, Init(MySatName, MySatData1, MySatData2, MyCatNbr,
MyEpoch)));

i should do it in two steps like this...

aTLERec := New(PTLERec, Init(MySatName, MySatData1, MySatData2, MyCatNbr, MyEpoch));
aTLEColl^.insert(aTLERec);

but the the problem still comes of how do i know if the record was inserted into the list or not? insert doesn't seem to return a true or false on success... i guess this is where it is required(?) to override the insert method? does everyone have to override the insert method if they are in duplicates := FALSE mode? how do they handle the throwing away of the unneeded data??

[TRIM]

Show quoted text 12 lines
>>> 2. something else i'm running into is with duplicates:=FALSE, there's >>> a whole bucket load of records that are not disposed of when i wipe >>> out the collection... heaptrc hollers right nasty to me about'em on >>> exit... i can only assume that these are the duplicates but i don't >>> understand why they are still hanging around if insert or add threw >>> them away already :/ >> >> If you already had them in the collection, they're not added again. >> You only dispose records added to the collection at the end, but >> these are lost this way. You can also sort this out in the overridden >> Insert method (if you don't want to use the newly created record, >> dispose it).

i'm looking at this override of the insert method...

Show quoted text 8 lines
> > as above, i don't know that they are /in/ the collection... i'm > (currently) simply calling the insert method and leaving the work up > to it... if it should be handling this gracefully, it isn't... at least > not in a way that heaptrc likes ;) It behaves as specified; the responsibility for disposing the duplicates is on your side (since you allocate them in your code also).

i think i see now...

[TRIM]

Show quoted text 7 lines
> i'll probably have broken my code by the time you read this... but i'll > very likely be attempting to implement the logic in my Input_Satellite_List > routine ;) OB-) Possible, but likely resulting in some useless overhead (computing performance-wise) if you still intend to use the method Insert in that case.

yeah, that went over like a lead balloon... i'm digging into the insert method override, instead... just gotta figure out how to access the epoch in the current record and the one in the passed record for the decision making comparison...

waldo kitty
to waldo kitty
waldo kitty wrote
On 1/12/2012 07:20, Tomas Hajny wrote: > On Thu, January 12, 2012 03:34, waldo kitty wrote:

[TRIM]

Show quoted text 12 lines
>> i'll probably have broken my code by the time you read this... but i'll >> very likely be attempting to implement the logic in my Input_Satellite_List >> routine ;) OB-) > > Possible, but likely resulting in some useless overhead (computing > performance-wise) if you still intend to use the method Insert in that > case. yeah, that went over like a lead balloon... i'm digging into the insert method override, instead... just gotta figure out how to access the epoch in the current record and the one in the passed record for the decision making comparison...

yeah, i know... bad form to reply to my own message but...

I GOT IT!! WHOOHOO! finally...

procedure TTLEColl.Insert (Item: Pointer);
var
   I: Sw_Integer;
   old: Pointer;
begin
    If NOT Search(KeyOf(Item), I) OR Duplicates Then   { Item valid }
      AtInsert(I, Item)                                { Insert the item }
    else                                               // otherwise
      // compare the epochs & replace existing record if < item^.epoch
      if PTLERec(At(i))^.epoch < PTLERec(Item)^.epoch then
        begin
          old := At(i);                  // save existing pointer first!
          AtPut(i, Item);                // now put in the new record
          PTLERec(old)^.Done;            // time to remove the old data
          dispose(old);                  // and finally dispose it
        end
      else
        begin                            // we're tossing this one out
          PTLERec(Item)^.Done;           // so remove the data first
          dispose(Item);                 // and then dispose the item
        end;
end;

now the list is sorted, newer entries replace older ones and there's no more whining from heaptrc about left over fluff on the heap... yeah!

and i said it before but dang this thing's fast... compared to the old tool that i've been using for eons... i mean look at it like this... we're comparing a turtle with a lightening bolt... somehow "fast" just doesn't seem to be strong enough to describe the difference in speed :P

Tomas Hajny
to waldo kitty: [TRIM] AHH! i think i understand this now... you are saying that instead of doing this...…
waldo kitty wrote 41 lines
On 1/12/2012 07:20, Tomas Hajny wrote: > On Thu, January 12, 2012 03:34, waldo kitty wrote: [TRIM] >> without looking at the "code to copy" if i want to override the insert >> method, it almost seems that there's a bug if it just throws away the >> record we're trying to insert... it would seem that if the code locates >> a "duplicate" record, it would properly dispose of unwanted data... >> unless i perform the > > No. There are two tasks. One is dynamic allocation of the object, the > other is insertion. Although you perform both tasks on one line in your > program, these are two distinct tasks. The insert code cannot know > whether > you still may need the (previously allocated) object or not in the very > general case. The insert code doesn't throw anything away - you do it by > not storing the result of the "New (PTLERec, Init (..." to some variable > and only send it as a parameter to a method which may or may not store > it > somewhere else. AHH! i think i understand this now... you are saying that instead of doing this... aTLEColl^.insert(New(PTLERec, Init(MySatName, MySatData1, MySatData2, MyCatNbr, MyEpoch))); i should do it in two steps like this... aTLERec := New(PTLERec, Init(MySatName, MySatData1, MySatData2, MyCatNbr, MyEpoch)); aTLEColl^.insert(aTLERec); but the the problem still comes of how do i know if the record was inserted into the list or not? insert doesn't seem to return a true or false on success... i guess this is where it is required(?) to override the insert method? does everyone have to override the insert method if they are in duplicates := FALSE mode? how do they handle the throwing away of the unneeded data??

There are multiple options. You can search before (as suggested by you) and only call Insert if not finding the result, you can override the Insert call as suggested by me, you can add another insertion method with a different signature (e.g. a function instead of a procedure providing information whether the record was inserted or not as its result), you could also search for the record after calling Insert and compare the pointer returned by At for the found index returned by Search to your original pointer (i.e. checking that both pointers refer to the same address, not just that they contain the same data), you could possibly also ignore the lost memory altogether in certain scenarios (assuming that the memory is deallocated anyway when the program is finished with its work, the amount of data is limited to sensible values compared to overall amount of memory in your machine and the program does just some one-time processing and thus does not live for very long), etc.

.
.

Show quoted text 5 lines
yeah, that went over like a lead balloon... i'm digging into the insert method override, instead... just gotta figure out how to access the epoch in the current record and the one in the passed record for the decision making comparison...
PROCEDURE TTLECollection.Insert (Item: Pointer);
VAR I: Sw_Integer;
BEGIN
   If NOT Search(KeyOf(Item), I) OR Duplicates Then   { Item valid }
     AtInsert(I, Item)                                { Insert the item }
   else
     if PTLERec (At (I))^.Epoch > PTLERec (Item)^.Epoch then
       begin
         FreeItem (At (I));
         AtPut (I, Item);
       end
     else
      FreeItem (Item); (* or "Dispose (PTLERec (Item), Done)" instead *)
END;

Tomas

waldo kitty
to Tomas Hajny
Tomas Hajny wrote
On Thu, January 12, 2012 17:23, waldo kitty wrote:

[TRIM]

Show quoted text 10 lines
> but the the problem still comes of how do i know if the record was > inserted into the list or not? insert doesn't seem to return a true > or false on success... i guess this is where it is required(?) to > override the insert method? does everyone have to override the > insert method if they are in duplicates := FALSE mode? how do they > handle the throwing away of the unneeded data?? There are multiple options. You can search before (as suggested by you) and only call Insert if not finding the result, you can override the Insert call as suggested by me,

this is what i've done and it works as per my previous message in this thread... however, i have found another problem... the program mostly works.. i mean that if a dupe catnbr with an epoch >= to the existing record is found it is thrown out but when another dupe catnbr is found it, too, is also inserted... this seems to only be happening with entries that have more than one duplicate... it doesn't make sense... i'm suspecting the compare routine to be the problem... epochs are read from the file and converted to a double by this original routine from TSKelso's SGP4-26A source code archive...

Function Real_Value(buffer : string; start,length : integer) : double; Takes the segment of {buffer} beginning at {start} and having length {length} and converts it to a double precision real number.

Function Real_Value(buffer : string; start,length : integer) : double;
var
   MyResult : integer;
   answer   : double;
begin
   buffer := Copy(buffer,start,length);
   Convert_Blanks(buffer);
   if buffer = '' then
     buffer := '0';
   Val(buffer,answer,MyResult);
   if MyResult = 0 then
     Real_Value := answer
   else
     Real_Value := 0.0;
end; {Function Real_Value}

but yet i find this in my debug output...

03597 5.04184917406000E+003 ..................... ** inserting record [...]
03597 1.20109577665600E+004 ..................... ** inserting record

here's my compare and insert routines... i'm suspecting my use of CompareValue is in error :/

Function CompareReal(num1,num2 : double) : integer;
begin
   CompareReal := CompareValue(num1, num2);
end;
procedure TTLEColl.Insert (Item: Pointer);
var
   I: Sw_Integer;
   old: Pointer;
   foo: integer;
begin
    If NOT Search(KeyOf(Item), I) OR Duplicates Then   { Item valid }
      begin
        {$IFDEF DEBUG}
        write(PTLERec(Item)^.catnbr^);
        write('  ');
        write(PTLERec(Item)^.epoch);
        write('   ');
        write('.....................');
        writeln('   ** inserting record');
        {$ENDIF}
        AtInsert(I, Item);                             { Insert the item }
      end
    else                                               // otherwise
      begin
        {$IFDEF DEBUG}
        write(PTLERec(At(i))^.catnbr^);
        write('  ');
        write(PTLERec(At(i))^.epoch);
        write('  ');
        write(PTLERec(Item)^.epoch);
        {$ENDIF}
        // compare the epochs
        foo := CompareReal(PTLERec(At(i))^.epoch,PTLERec(Item)^.epoch);
        case foo of
          -1 : begin                       // replace existing record
                 {$IFDEF DEBUG}
                 writeln('   ** replacing record');
                 {$ENDIF}
                 old := At(i);             // save existing pointer first!
                 AtPut(i, Item);           // now put in the new record
                 PTLERec(old)^.Done;       // time to remove the old data
                 dispose(old);             // and finally dispose it
               end;
           0 : begin                       // we're tossing this one out
                 {$IFDEF DEBUG}
                 writeln('   ** epochs are same');
                 {$ENDIF}
                 PTLERec(Item)^.Done;      // so remove the data first
                 dispose(Item);            // and then dispose the item
               end;
           1 : begin                       // we're tossing this one out
                 {$IFDEF DEBUG}
                 writeln('   ** existing is newer');
                 {$ENDIF}
                 PTLERec(Item)^.Done;      // so remove the data first
                 dispose(Item);            // and then dispose the item
               end;
          end; {case}
      end;
end;

[TRIM]

Show quoted text 22 lines
. . > yeah, that went over like a lead balloon... i'm digging into the insert > method > override, instead... just gotta figure out how to access the epoch in the > current record and the one in the passed record for the decision making > comparison... PROCEDURE TTLECollection.Insert (Item: Pointer); VAR I: Sw_Integer; BEGIN If NOT Search(KeyOf(Item), I) OR Duplicates Then { Item valid } AtInsert(I, Item) { Insert the item } else if PTLERec (At (I))^.Epoch> PTLERec (Item)^.Epoch then begin FreeItem (At (I)); AtPut (I, Item); end else FreeItem (Item); (* or "Dispose (PTLERec (Item), Done)" instead *) END;

that's pretty close to what i did (see above) :lol: i might be able to tighten mine up a bit instead of doing the .done and then a dispose... or does that matter?

waldo kitty
to waldo kitty
waldo kitty wrote 5 lines
but yet i find this in my debug output... 03597 5.04184917406000E+003 ..................... ** inserting record [...] 03597 1.20109577665600E+004 ..................... ** inserting record

i was looking at the wrong compare... i had thought i'd be "smart" and use strcomp but it has bit me arse! :lol:

function TTLEColl.Compare(key1, key2: pointer): sw_integer;
begin
   if PString(key1)^ = PString(key2)^ then
     compare := 0
   else if PString(key1)^ < PString(key2)^ then
     compare := -1
   else
     compare := 1;
//  compare := strcomp(PChar(key1),PChar(key2));
end;

i wonder why the difference?? reverting to the original code as above (note the commented out strcomp) and it is working... now, after scrolling thru a side by side diff) there are no dupes at all, YAAAAY!

Tomas Hajny
to waldo kitty: [TRIM] this is what i've done and it works as per my previous message in this thread...…

On 12 Jan 12, at 14:11, waldo kitty wrote:
.
.

that's pretty close to what i did (see above) :lol: i might be able to tighten mine up a bit instead of doing the .done and then a dispose... or does that matter?

I believe that there isn't a difference from a technical point of view in this case (except for the fact that decoupling the destructor from the deallocation may be a bit dangerous if you forget about it later a try to do something with the object in between).

Tomas

waldo kitty
to Tomas Hajny
Tomas Hajny wrote 10 lines
On 12 Jan 12, at 14:11, waldo kitty wrote: . . > that's pretty close to what i did (see above) :lol: i might be able to tighten > mine up a bit instead of doing the .done and then a dispose... or does that matter? I believe that there isn't a difference from a technical point of view in this case (except for the fact that decoupling the destructor from the deallocation may be a bit dangerous if you forget about it later a try to do something with the object in between).

i'm not quite sure i understand... i kinda konw what you are speaking of, though, because the HINTs and such (i don't know which since i just enabled them) tell me something about a plain "pointer being disposed makes no sense" or some such... when i put the dispose lines back, these reports were emitted by the compiler but heaptrc didn't bitch any more... removing the dispose lines to make the compiler made heaptrc whine and cry...

also, i've a problem with os2260full that i'll be changing the subject of this thread for or starting another thread... yes, i bit the bullet and wiped my old os2 2.4.0(?? or 2.4.2???) install and none of my previous projects compile... they complain about "Error: Can't call the assembler, error -1 switching to external assembling"... :/

i've manually removed all the .o and .ppu files that i may have inadvertently transferred when i scp'd everything over to my os2 box but no joy :/

waldo kitty
to Tomas Hajny: . . I believe that there isn't a difference from a technical point of view in this case…
Tomas Hajny wrote 9 lines
On 12 Jan 12, at 14:11, waldo kitty wrote: . . > that's pretty close to what i did (see above) :lol: i might be able to tighten > mine up a bit instead of doing the .done and then a dispose... or does that matter? I believe that there isn't a difference from a technical point of view in this case (except for the fact that decoupling the destructor from the deallocation

i decoupled the destructor from the deallocation? hummm... no clue... ya got a cluex4 to smack me with, please? :(

may be a bit dangerous if you forget about it later a try to do something with the object in between).

in this case, that won't happen ;)