Hi,
 
this seems to be misleading error message. TFPGObjectList works well for objects (classes). When I tried to push record to it, I got the same error message. TVec3 is not class, right?  
Vojtěch
______________________________________________________________

I’m trying to use TFPGObjectList but I get an error. Is the unit busted? If so is there another generic list class in the RTL?

 type
 TVec3List = specialize TFPGObjectList<TVec3>;
 var
 list: TVec3List;
 begin
 list := TVec3List.Create;
 list.Add(V3(1,1,1));
 list.Free;

/usr/local/lib/fpc/3.1.1/units/x86_64-darwin/rtl/fgl.ppu:fgl.pp:992: error: 14: identifier idents no member "Free" Regards,
Ryan Joseph

2018-07-01 4:01 GMT+02:00 Vojtěch Čihák :

this seems to be misleading error message. TFPGObjectList works well for objects (classes). When I tried to push record to it, I got the same error message. TVec3 is not class, right?

Indeed, it's a misleading message. The message

...identifier idents no member "Free"

doesn't talk about the line

    list.Free;

(which is fine, "list" is an instance of a class). It talks most likely about the "Free" call inside FGL implementation in

"""
procedure TFPGObjectList.Deref(Item: Pointer);
begin
  if FFreeObjects then
    T(Item^).Free;
end;
"""

If you look inside the FGL unit sources of your FPC version, you will likely find this to be at line 992 :)

Short explanation:Since your TVec3 is probably not a class, you cannot do "Free" on it.

The solution is to use TFPGList instead of TFPGObjectList. You should also define an equality operator if TVec3 is a record. See e.g. my example in https://castle-engine.io/modern_pascal_introduction.html#_operator_overloading (scroll to the example with TMyRecordList).

Or you could use Generics.Collections unit with generic TList for records. It by default compares records by comparing memory contents.

Regards,
Michalis

Thanks, that make sense now. I was really suspicious of TFPGObjectList having Object in it’s name but I didn’t know there was a TFPGList.

Or you could use Generics.Collections unit with generic TList for records. It by default compares records by comparing memory contents.

Is that part of the RTL and if so what’s the unit name? I had a hard time finding good resources on classes the RTL provides.

Regards,
Ryan Joseph

It is part of FPC 3.1.1, and the unit name is "Generics.Collections". For FPC 3.0.x you can take them from Maciej Izak's original repo https://github.com/maciej-izak/generics.collections .

Some examples of basic usage are in my
https://castle-engine.io/modern_pascal_introduction.html#generic-containers-section . Also Delphi docs are quite useful since the API is deliberately compatible:
http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.Generics.Collections .

Regards,
Michalis

Ryan Joseph wrote

Is that part of the RTL and if so what’s the unit name? I had a hard time finding good resources on classes the RTL provides.

That has been my experience as well.  Is there a definitive source of documentation for both Free Pascal and Lazarus?  Anyone? Please don't say the wiki.  And "read the source" is not useful when one doesn't even know where to look, given the lack of documentation on what's available and how it's laid out.  I have been forced to use my old Delphi reference manuals, and that is less than ideal.  At some point I'd even like to contribute to the documentation, but right now it's a Catch-22 that has me seriously considering giving up on these tools (and that from 40+ years of programming veteran!)...

-Jim

P.S. - Read this as frustration, not criticism.  I'd just like to find the hidden resource, if there is one.

On Jun 30, 2018, at 10:03 PM, Jim Lee wrote:

I know of http://docs.getlazarus.org which has some helpful stuff but it’s mainly Lazarus which I personally don’t use.

Otherwise it’s just hunting around in https://www.freepascal.org/docs-html/rtl/ for me. There’s no searching feature though. :(

FPC badly needs something like http://docs.gl (a life saving OpenGL reference).

Regards,
Ryan Joseph

Thanks Ryan, but that looks like a clone of the fpdoc-generated stuff on the wiki (with prettier colors, though).

I'll give it a couple more days for someone to chime in, and then it's back to the land of C for me.  Too bad - Pascal was always my favorite language.

-Jim

Can you explain what you think is wrong with or missing in the official documentation ? (apart from a search mechanism)

Michael.

In our previous episode, Michael Van Canneyt said:

Can you explain what you think is wrong with or missing in the official documentation ? (apart from a search mechanism)

(the CHM form of the documentation has fulltext search, indexes etc. On Windows you can just click them to open.

It also works from Lazarus, but you can also try a 3rd party CHM viewer like kchmviewer, though in my opinion the quality of those varies)

I know. we have fpIndexer to make such an index. I just need to find time to integrate building the index into the deployment process and add a link in the header.

The length of my todo list can no longer be expressed in an Int64. Maybe we need to add Int128 to the compiler...

Michael.

On Jul 1, 2018, at 6:19 AM, Marco van de Voort wrote: (the CHM form of the documentation has fulltext search, indexes etc. On Windows you can just click them to open.

The CHM reader can be searched so why doesn’t search work for the web interface? I would think there would be a standard script for displaying that format online.

Also the RTL is missing collections.generics. Is that because it’s a new addition?

Regards,
Ryan Joseph

In our previous episode, Michael Van Canneyt said:

> (the CHM form of the documentation has fulltext search, indexes etc. On > Windows you can just click them to open. I know. we have fpIndexer to make such an index.

The CHM packages has its own (afaik with phrase compression)

True. I actually have a local webserver based helpsystem that loads the CHMs on my todo. Even to just be able to easier check working on non Windows systems. gnochm and kchmviewer are more often broken than not for non trivial files.

Well, search is a big one, but there seems to be a lot of missing pieces (again, without search it's hard to tell exactly how much is missing).  There also seems to be a lot of outdated stuff, and most of what I find was auto-generated from fpdoc (when? last week or 10 years ago?  no timestamps!), with scant (or no) explanation beyond "here is a terse, un-commented example".

I haven't found any sort of map or tree that shows how the rtl and fcl are organized, or an index that shows which types/functions/classes are in which units (that's hugely important).  Basically, I expected to find the equivalent of the old Borland programmer's reference manuals.

The most common scenario for me is this: "I wonder if fpc (or Lazarus) already has <insert feature here>".  Go to the wiki and browse haphazardly, looking for something named similarly.  If that something is found, and looks promising, follow link to a half-page of "documentation" which is basically just a snippet of the interface section in the source.  Then, try to find the referenced module in my local source tree and discover that it's not quite the same, or is not what I was looking for in the first place.  Rinse and repeat.

I would love to be able to use fpc, as it's pretty much the only game in town when it comes to a cross-platform Pascal able to be used for low-level, even bare metal programming.  I also realize that documentation is often low on the list of priorities - but good documentation is vital to build a user base, and bad documentation is what drives people away.

-Jim

Jim Lee wrote

The most common scenario for me is this: "I wonder if fpc (or Lazarus) already has <insert feature here>". Go to the wiki and browse haphazardly, looking for [...]

that may be part of your problem... you're looking in a wiki instead of a more proper place... all the FPC docs i read on the web are almost exactly like the help files you can use locally with a CHM viewer or similar... what you are looking for are these...

https://www.freepascal.org/docs-html/rtl/index.html https://www.freepascal.org/docs-html/fcl/index.html

and actually, you would start here at the beginning...

https://www.freepascal.org/docs.var

it is rather interesting that the above docs.var doesn't appear to be mentioned in the wiki's page for Lazarus_Documentation... all the links in the #Free_Pascal_Compiler_Documentation section of that page point to lazarus-ccr instead of the above docs.var location...

i don't know which is right but i do know you're not looking in the right place or your google-fu is lacking a bit... i started this reply with a search for "freepascal runtime library" and the second hit was the docs-html one above... ;)

That is exactly the documentation I'm talking about.  I've seen all of them.  I got to them via the wiki, so I suppose I should have said "The wiki, and documentation linked to from there".

-Jim

Jim Lee wrote

That is exactly the documentation I'm talking about.  I've seen all of them.  I got to them via the wiki, so I suppose I should have said "The wiki, and documentation linked to from there".

oh!! ok, well... i think the docs.var one i pointed out above is the better one... i don't know why but i do... it is where i always go... it also has dates at the very bottom of when it was generated... the rtl one i linked to was generated in 2017...

bookmark that docs.var link ;)

The documentation is brought up to date at every release: all new identifiers are documented and the documentation regenerated. If you look in the bugtracker, you will see that I regularly fix documentation issues.

If you think it is too terse in places, please report them in the bugtracker. I will always fix all such reports.

It is all there, user's guide, language reference, programmer's guide, Reference to *all* rtl units. What do you think is missing ?

See https://www.freepascal.org/docs.var

index:
RTL: https://www.freepascal.org/docs-html/current/rtl/index-8.html FCL: https://www.freepascal.org/docs-html/current/fcl/index-8.html

I don't know what documentation you are referring to, but as said, all basic RTL units, and selected FCL units are documented: https://www.freepascal.org/docs.var

Packages are another story. I checked the packages. There are roughly 16.000 classes. I didn't count methods/functions, but they surely are a multiple of that. There is 1 documenter: me.

I have toyed with the idea to run fpDoc on all packages, and put the result online, so people can at least have a look in the index to see what's available.,

Feel free to chip in.

Michael.

Thanks.  The user's guide, programmer's guide, and language reference manual all say they are for version 3.0.2.  Perhaps the current version didn't require any documentation changes?

If you think it is too terse in places, please report them in the bugtracker. I will always fix all such reports.

That really wasn't my point.  From the point of view of someone just coming to fpc,  the documentation is the "face" of the product.  If that face is ugly or hard to navigate, they are likely to just walk away.   Getting involved in the bugtracker is for people already invested in fpc, and familiar enough with it to not *need* the documentation.

Perhaps the rtl is more complete than the rest - I'll take your word for it.  I know it's a tremendous amount of work!

The class library chart on that page is nice, but it's 11 years old (dated 2007).

index: RTL: https://www.freepascal.org/docs-html/current/rtl/index-8.html FCL: https://www.freepascal.org/docs-html/current/fcl/index-8.html

An auto-generated index of all identifiers used across all modules is useful, if you know what you're looking for.  Say I was looking for something like a Timer class - where do I start?  Would it be in the rtl, the fcl, or somewhere else?  Would it be named Timer, TTimer, or something else?  I can't find an index that has an entry for "timer" and points to all the timer-related things.

Pardon my ignorance, but I take it that "packages" is more like an un-curated repository of contributions rather than an official part of the fpc distribution?  If so, I wasn't aware of that.  It would explain a big part of my frustration.

-Jim

On Jul 1, 2018, at 1:28 PM, Michael Van Canneyt wrote: RTL: https://www.freepascal.org/docs-html/current/rtl/index-8.html

Without a good search feature these are really hard to use. Searching should be the first priority. Can’t we just make a little script to put on the server which searches the original indexed format which the HTML is derived from? It would be so easy to make something that’s cheap but at least works.

Regards,
Ryan Joseph

And to what page would this script then point when you find something ? If it was easy, it would have been done already.

But, here's your shot at contributing :)

Michael.

Seems I forgot to update the symlink when we released. I fixed that. Thanks for pointing it out.

I consider this an error in your reasoning.
The bugtracker is very lowlevel, and should be usable and familiar for all. Many of the documentation bugreports are from one-time users, people I never have heard from.

This is known. I better remove it, it does more damage than good now.

In the absence of a search mechanism:

You would get a long way if go to the FCL page, press ctrl-f and type timer in the search box of your browser..
The 4th hit on the page gives you fptimer, the unit that contains a timer.

But most of the time when I search for something, I just enter google, type "free pascal timer". The third hit is the same as the one I mention above. You can't beat google for ease of use and accuracy.

FPC's "packages" are not uncurated. They are mostly import units, allowing you to work with all kinds of open source libraries. On top of these e.g. the data acces units are built.

But I know that search is an issue, hence my question "what is the problem with the documentation _besides_ search".

Contrary to what it may seem it is not so easy to make a search engine for the documentation. I've had several shots at it, but have not yet come up with a good mechanism. The last mechanism I tried ended up indexing more than 24 hours before I killed it.

Contributions or ideas welcome.

Michael.

Without a good search feature these are really hard to use. Searching should be the first priority. Can’t we just make a little script to put on the server which searches the original indexed format which the HTML is derived from? It would be so easy to make something that’s cheap but at least works.

Un4seen have a pretty good documentation browser:
http://www.un4seen.com/doc/

From what it looks like, this is built on the CHM format files? If one could

figure out what they use, maybe that would be a step in the right direction?

Michael Van Canneyt wrote

Can you explain what you think is wrong with or missing in the official documentation ? (apart from a search mechanism)

Some sort of interlinking between the manuals would be nice (see how it works for the different BASS libraries, for example). I have still not figured out where exactly the dividing line between the Reference and Programmer's Guide is, and there are many places where they provide information on related topics (formal and implementation aspects, for example) and there is no way to know. I suspect it would help with some confusion around implementation-defined features, if one could simply see "here's a language construct, check out this section to see how FPC handles it". Emba's docwiki is fairly good in that regard.

The CHM manual generation is as problematic as the indexing itself. And the CHM search index is generated by Windows on first use, as far as I know.

You must make a distinction between the unit reference and the language reference/programmer's guide.

The unit references are fully interlinked. The programmer's guide and language reference are 2 separate documents which are indeed not interlinked. They never were designed to be interlinked: The programmer's guide contains advanced topics, which the beginning programmer does not .

Michael Van Canneyt wrote

The CHM manual generation is as problematic as the indexing itself. And the CHM search index is generated by Windows on first use, as far as I know.

Hm, okay. Not really helping then, sorry.

They never were designed to be interlinked: The programmer's guide contains advanced topics, which the beginning programmer does not are a few exceptions (i.e. "prog:8.2.1 Integer types" links to the ref),

but yes, I'm aware this was intentional.
Just wanted to suggest that sometimes it would be nice for newcomers to be reminded that these advanced sections exist in the first place. Especially when things are different from TP and Delphi (or even the C equivalents of an expression), one might not find this information at first, even if it's there: take Willibald's recent question about managed Result vars, the recurring question of what precludes inlining, or my confusion about the subtly different range type system. The information is there, but next to impossible to find unless one already knows where it is.

In our previous episode, Michael Van Canneyt said:

No. Both a index as a fulltextsearch is generated during CHM compilation, either by fpdoc or by fpcdocs/compilelatexchm.pp (for ref,prog,user)

You can view the index in textual (XML) form using

chmls extractindex <chm file>.

compilelatexchm doesn't generate a full text search (afaik for fpdoc it was a 3rd party patch to enable it (Vincent?)), mostly because I don't much about it. TBD.

Yes. I realize this.

I'm reviving my 'search docs' engine attempt. The idea is to create a single index for all doc pages, which would be updated when the documentation is regenerated.

It takes currently about 1000 secs to process the FCL docs (~ 5000 files). That seems acceptable to me.

I'll try to put a rudimentary search page online this week.

Michael.

In our previous episode, Michael Van Canneyt said:

It takes currently about 1000 secs to process the FCL docs (~ 5000 files). That seems acceptable to me.

Test with the LCL docs. Generating a rtl or fcl chm is measured in tens of seconds, LCL a full 8 minutes.

No doubt...

If I take the mem/file backend for fpindex, it takes something like this as well. But since it needs to go up on idefix, a DB backend is preferable, and then it takes longer.

Michael.

I found this project 2 years ago but I've never tried to use:

https://github.com/AcoonDe/OpenAcoon

Maybe it could give you some ideas.

Regards,
Marcos Douglas

The easiest thing would be to read the .chm file and search that like a .chm reader but the file is 3.4M so every request to search would require opening and reading the file (is that a deal breaker for a web server?). Another option is reading the .chm file once and build a SQL database from all the class/method names.

I’m not really sure the best way to do this over the web but I can’t think of at least the 2 most obvious ways.

Regards,
Ryan Joseph

why can’t the .chm file be read on the server? There’s already scripts to do this like https://github.com/mlocati/chm-lib. It’s fast to search those things once they’re in memory.

Reading the 3.4M file is brutal though and I don’t know if web servers can share memory between requests unless we had custom software. MySQL solves this but we need to convert the .chm index into SQL then.

Regards,
Ryan Joseph

Michael Van Canneyt wrote

On Sun, 1 Jul 2018, Jim Lee wrote: > On 07/01/18 12:28, Michael Van Canneyt wrote:

.
.

Three more comments:

1) As you all probably know, you can explicitly ask Google to restrict the search to a particular site - e.g. 'timer site:www.freepascal.org', or 'timer site:freepascal.org' (both without quotes; the latter will find the references to discussion on the forum as well, whereas the first will show the docs in the first place).

2) The text mode IDE (FP) has built-in indexing mechanism. If you install the HTML docs into it (Help / Files...), type 'timer' in a source file and press Ctrl-F1, it will find a reference to TFPTimerDriver.Timer with references to various related information.

3) Since the docs sources are readable in any text editor, using something like 'grep' (or some more sophisticated tool doing the same) may help to find all references in the docs sources if you have a checkout of the docs SVN repository (and you can obviously search across the unit sources similarly as well).

Tomas

In our previous episode, Ryan Joseph said:

You can read the core CHM structures (which are maybe a few MBs), and extract files only when needed, and then cache them, depending on RAM settings. CHM was made for this to avoid decompressing everything to show them.

The CHMs together are 24MB, so +10 MB for compressed core structures + 100-200MB to dynamically cache frequently asked pages (like tocs and sysutils), 256MB to the server app would go a long way.

But while the index (which is more or less a textsearch on titles of topics) is easy to search, I'm not sure if we have read support for the fulltextsearch. So possibly we only generate and write the tables, and not read them and do actual search. (but maybe this is in lhelp)

But the lazarus and textmode IDE "F1" key is based on the indexes, not the fulltext search. The indexes are about 15-20MB as XML, but a third of that as binary BTree. (that form deduplicates strings)

Another option is reading the .chm file once and build a SQL database from all the class/method names.

I don't think this really is a case for sql. A fastcgi server that loads the files on startup, and checks if they are updates once per 24hrs, would be enough. fastcgi, since it would be stateful.

This is core structures of the LCL.chm (14MB compressed html, 20000 html files)

compressed offset uncompr size name // meaning 1 193170819 548926 /#STRINGS // strings. Indexes, binary toc strings are deduplicated
stored here
0 112 149 /#SYSTEM // settings 1 174179574 118460 /#TOCIDX // binary form of toc, also available (but larger) as
xml.
1 190311922 1001152 /#TOPICS // <topic title, urltbl entry> map 1 191313074 1106149 /#URLSTR // urls with urltbl reference 1 192419223 751596 /#URLTBL // urltbl , topics lookup table for bidirectional lookup
1 193719745 3407168 /$FIftiMain // fulltextsearch data 1 174176823 2751 /$OBJINST
1 175356422 3549260 /$WWKeywordLinks/BTree // index in binary format

TOC is the giant page with a treeview with all topics. (more than one even, by unit and alphabetically, the treeview on the first tab if you open the chm in windows)

So it is about 12MB data uncompressed indexes and metadata for everything. The uncompressed indexes and metadata are meant to be searched directly.

The LCL.chm is about 60% of the size of the current chms, so that would extrapolate to all indexes being about 10/6*12=20MB. Probably less, because the LCL has little content and an enormous amount of topics

Michael Van Canneyt wrote

Seems I forgot to update the symlink when we released. I fixed that. Thanks for pointing it out.

No problem.

Maybe so.  I consider it rude to submit an issue to a bugtracker as a fly-by.  The submitter should be willing to research and work with the developer(s) to reproduce/resolve the issue.  That's not really feasible unless one is an active user.

This is known. I better remove it, it does more damage than good now.

I wouldn't remove it altogether, but maybe move it off the page with the main documentation links and clearly label it as outdated.  That may give someone an incentive to update it while not confusing new users.

With this, I differ.  Google != accuracy!  Google is a distraction, a time sink, and a rabbit trail inducer.  The last thing I want to do while in the middle of coding is to get online and search for something.  I'm usually "in the zone" and want to have the needed documentation in front of me on a separate screen.  Offline. OFFICIAL documentation, not some anonymous blogger's explanation of how it works, not some anonymous forum post that attempts to explain something.

While a working search may go a long way toward alleviating the problems, I don't see that as the major issue.  The major issue, to me, is that almost everything is auto-generated from the source (at least the reference material, not necessarily the tutorials).  There needs to be more organizational design, more thought put into useful indices.  You can't auto-generate a *good* index.  There are cross-references and relational inferences that only a human can make.  For a project of this size, that may be too much for a single person, no matter how good they are.  So, I guess the bottom line is that fpc needs more documentation contributors.  And that requires people with a thorough knowledge of the material, not newcomers.

Contributions or ideas welcome.

That's what I'm trying to do.  And, thank you for your own heroic contributions!

-Jim

I've now added a TObject constraint to TFPGObjectList and the TData parameter of TFPGObjectMap. Sadly I can't add it to TFPGInterfacedObjectList or TFPGInterfacedObjectMap as currently both interfaces and objects implementing interfaces can be used.

Regards,
Sven