Hi,

I am just playing around with the CodeTools and have the following:

P := node.CodePos.P; Tool := TStandardCodeTool(CodeToolBoss.GetCodeToolForSource(node.CodePos.Code, false, false)); debugln(['identifier: ', Tool.ExtractIdentifier(P)]);

where node.CodePos.Code is the code of a unit and P is the starting position of something like

function foo(foobar: TSomething): TSomethingElse;

Tool.ExtractIdentifier() will now return the string "function". Isn't function a reserved word and not an Identifier? What is the codetool's understanding of an identifier and how do I search for the identifier?

The end result I am trying to achieve is to point it to somewhere and then have it return the *position* of the next Identifier (not just the next word). I know that ExtractIdentifier() won't help me with the position but I am just in the phase of some initial experimenting with it and it makes me wonder what the codetool's understanding of an "identifier" is and how to tell it that i am interested in the identifier of the declaration and not just the next word in the code.

That should be

Tool :=
TStandardCodeTool(CodeToolBoss.GetCodeToolForSource(node.CodePos.Code,
true, false));
if Tool=nil then exit;

An "identifier" is simply a valid pascal identifier (starting with a letter, followed by some letters and numbers). A 'keyword' is context sensitive.
For example when doing identifier completion the following 'function' is an identifier:

i:=function|

I wrote an example in codebrowser.pas:
function TCodeBrowserView.GetCodeHelp

Mattias

Mattias Gaertner wrote

I wrote an example in codebrowser.pas: function TCodeBrowserView.GetCodeHelp

Thank you for this example. Now you effectively implemented almost all the things I wanted to do.

My latest (not working) version before I read your answer was using Tool.BuildSubTreeAndFindDeepestNodeAtPos() and
Tool.MoveCursorToProcName() and Tool.MoveCursorToPropName() and *almost* worked (worked in some cases but not always), sometimes the node that it returned was correct, sometimes it did not make any sense at all and began and ended at some completely wrong positions and had a completely wrong Desc. (btw: what is a "clean" position and what is the difference between just the absolute position and the clean position? Is this important or is there no difference at all? At one Point I tried a method that looked like it was supposed to convert column and line to a "cleanpos" but it returned a number three times as large as the entire file size)

Please don't remove your comments in this example again, although they might seem obvious to anybody who knows the codetools, they are very helpful and explain things about the codetools that are not obvious from looking at the code alone. I would have for example never found the method CodeToolBoss.Explore() or known that I needed to call it (I have seen some of the examples but I didn't notice this method call before, probably because it is implicitely called by something else) and also I don't know what exactly LazarusIDE.BeginCodeTools() does.

Because I felt that the reason for BuildSubTreeAndFindDeepestNodeAtPos() sometimes but not always returning anything useful was that the tool itself was not properly initialized, I was looking for something that would tell the tool to properly initialize itself and parse the whole file but I looked for a method of the codetool class and not a method of the codetoolboss. I tried Tool.BuildTree() but it didn't seem to help.

The understanding of the codetools seems to be essential for doing any non-trivial thing inside the IDE. We need a document that describes the architecture of the codetools, the lifespan of such a tool, when and how to create it or where to find an instance of an already initialized tool, how to properly initialize it, what does happen internally when initializing it, generally what is going on inside these tool objects, what is needed for them to operate, what should be avoided, what can go wrong when using them, what is the codetoolboss and what is its role in this whole thing (I have now seen at least 3 or 4 ways of how and from where to get such a codetool instance and they all look slightly different), what steps are done by the boss and what steps are done by the tool, How does the whole containment hierarchy of these objects that are found "inside" (or belong to) other objects look like (maybe visualize it with a diagram), etc.

Bernd

Bernd Kreuss wrote

The understanding of the codetools seems to be essential for doing any non-trivial thing inside the IDE.

For example as an application of the codtools I have a tool in mind that would help me exploring a big codebase with a lot of calls through different abstraction layers that can easily confuse me when trying to read the code and follow wrapper calls 3 or 4 levels deep and still remember where I came from. I will soon have 5 editor windows open and some of then are scrolled to a different position because the calls go back and forth throughout all these files.

This tool would help me build a tree starting with the current method, listing all calls to other methods inside the method body as subnodes of this node and when I click on one of the subnodes it will again list all method calls inside the called method as subnodes of *this* node, letting me grow this tree only into the branches that are of interest to me. Whenever I click a node the editor window will immediately open/scroll directly to the *implementation* so I can always see the actual code and can quickly jump around in what could be described as the "callstack" of the simulation in my brain, without ever losing track of where I came from or where I have been already.

For implementing this tool I need a deep understanding of the codetools and what can be done with them and how they are used to their full potential.

Bernd

Bernd Kreuss wrote

Bernd Kreuss wrote > The understanding of the codetools seems to be essential for doing any > non-trivial thing inside the IDE. >

Code-exploration maybe.... There is lot of other stuff in the IDE...

...
Nice Idea, certainly helpful.

It has limits too (that shouldn't mean not to do it...):

Simple concepts like a Visitor (or anything where the method is based on inheritance) simply can not be resolved => codetools will always jump to the base class.

procedure TOnlineState.SwitchStateToOffline( Connection: TBaseConnection);
begin
   Connection.SayGoodBye;
   Connection.GoOffline;
end;

The code can have many different Connection classes, each having it's own SayGoodBye and GoOffline method.
Only at runtime, only if you know the real class, only then you can know where the code will go...

On the other hand, you can pick simplier tasks => there is a window that can show your jump history => so you don't need all windows open. That List is very basic => it could get many improvements (for example showing not only the line of a history point but also the procedure in which the line is (ok that is codetools, but less of it)...

I guess you probably rather go for your solution, and yours is better => but I just wanted to make an example, that the amount of "learning existing ide" and "which parts to learn" depends on the task you choose.

Martin

Zitat von Bernd Kreuss :

Sometimes it is easier to implement than to explain it in English. And I think in this case the implementation explains it better.

Maybe you had edited in the source editor and not updated the code browser. The code browser is meant to search hundreds of units which takes some time to scan. It would slow down the IDE quite a bit if it would update constantly.

See here
http://wiki.lazarus.freepascal.org/Codetools#How_the_codetools_parses_sources.2C_difference_to_a_compiler

Every function has its pros and cons. Explore is great for the code explorer, but is often too much for simple tools. Doing too much means more things can go wrong. For example it can stop on syntax errors, that might be irrelevant for the intended function. And of course it can be too slow, for example when invoking it on every compiler hint/error/warning.

and also I don't know what exactly LazarusIDE.BeginCodeTools() does.

It does some basic checks and commits the changes of the source editor to the codetools file caches.

BuildTree is often enough.
CodeToolBoss is the next level and handles multiple units. For example it checks if the main source has changed.

Another day.

Mattias

Zitat von Bernd Kreuss :

That's what the jump history and bookmarks are good for. And the ability to open the same source multiple times.

Keep in mind that methods are often overridden. And properties can be function calls and can be overridden too. And often there are many implicit function calls.
To be honest I doubt that showing *all* calls of a function is helpful, because the list is too big big to show in one tool window and because it does not show the really called overrides. IMO what is really needed is a tool to find the overrides of a call and show the list sorted for relationship, and to jump to them easily. I'm writing a tool to find overrides and sorting them. But that requires some extensions to the codetools.

For implementing this tool I need a deep understanding of the codetools and what can be done with them and how they are used to their full potential.

Mattias

Mattias Gaertner wrote

I wrote an example in codebrowser.pas: function TCodeBrowserView.GetCodeHelp

I'm now trying your example and it seems to suffer from the same problem that I found with my experiments. I inserted some debugln into your method and look at stdout to see the problem. The following is your code and I only inserted the debugln() at this position:

Now I simply call your method right at the beginning of GetTVNodeHint() without any other code to easily experiment with it and see how it behaves in the original environment where it is supposed to work.

To test it I select LCL in the code browser and then I go to the very first unit i find there (ActnList) and hover the mouse over some nodes from the beginning of the list and everything still looks normal. (I insert it here as email quote to (hopefully) prevent line wrapping and also attach some output from my console as text document just in case it is not easily readable here):

Node 21 ApplicationActionComponent CTNode 21 ApplicationActionComponent text -->ApplicationActionComponent: TComponent = nil;<--

or the following is also ok although of course it won't show the correct identifier yet but the node is found correctly, it displays exactly as expected:

Node 60 CreateAction(TheOwner:TComponent;ActionClass:TBasicActionClass) CTNode 60 function text -->function CreateAction(TheOwner: TComponent; ActionClass: TBasicActionClass): TBasicAction;<--

and the following is an example of a class definition, everything is fine, exactly as expected:

Now i go down in the list to the unit "Forms" there are many more that won't work, Forms is just an example. I use the very first definition in this unit, it is var Application: TApplication; now see how CTNode looks like:

Node 21 Application CTNode 21 xceptionObject text --> Application: TApplication = ni<--

or a procedure:

Node 60 CreateWidgetset(AWidgetsetClass:TWidgetsetClass) CTNode 60 procedure text -->procedure CreateWidgetse<--

or a class:

I didn't make any edits in any of these units and have none of them open in the source editor, the whole project (except my little change in codebrowser.pas) is right from the svn server and immediately before this experiment did a clean compile of everything and didn't change anything.

These are the exact same results that I had when doing it with BuildTree() and BuildSubTreeAndFindDeepestNodeAtPos(). Can you reproduce this? If not then what else should I debugln() here to narrow it down?

Bernd
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed... Name: some_more_examples_of_the_output.txt
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100728/4c35a120/attachment.txt>

I didn't test the code much.
I just saw that codebrowser stores file positions, not clean positions. Makes sense.
I fixed GetCodeHelp.

[...]

Mattias

It would show like this:

- TOnlineState.SwitchStateToOffline( Connection: TBaseConnection)
+ Connection.SayGoodBye
+ Connection.GoOffline

and then when I click on one of them It would scan the whole project and then pop up a list of all possible overrides that looks like:

TBaseConnection.SayGoodBye
TFooConnection.SayGoodBye
TBarConnection.SayGoodBye
TFooBarConnection.SayGoodBye

and because I know that I am interested in what it does with a TFooBarConnection (because somehow I know (from understanding the code) that I *now* want to know what will happen with a TFooBarConnection) I click on TFooBarConnection and then my Tree looks something like this:

- TOnlineState.SwitchStateToOffline( Connection: TBaseConnection)
- TFooBarConnection(Connection).SayGoodBye
+ MySocket.Send('quit' + #13)
+ SomethingElse.DoSomething()
+ Connection.GoOffline

Later I might click on it again, delete the subtree and explore a new one with a different override.

[sending this a second time because I got a strange error from the list server]

Mattias Gaertner wrote

I didn't test the code much. I just saw that codebrowser stores file positions, not clean positions. Makes sense. I fixed GetCodeHelp.

Ok, It works now. BTW: The explanation about the codetools in the wiki is already quite helpful, I don't know why I didn't find this text earlier, It would have saved me some time.

The GetCodeHelp() function is now working as expected but I discovered something else and this is currently also visible in the code editor in the svn version:

consider the following small program:

program foo;
type
  TAnswer = Integer; // represents an answer
var
  Answer : TAnswer; // in this global variable we store the answer
{call this to find the answer for everything}
procedure FindAnswer;
begin
  Answer := 42;
end;
begin
  FindAnswer;
  Writeln(Answer);
end.

if you have lazarus running from the console so that you can see the debug messages and move the mouse around in the above program over the various occurrences of the identifiers you will observe that whenever you are over the *declaration* it will raise (and gracefully catch) an exception and at the same time the hint window will miss the part for chhoSmartHint (the part with full declaration, filename and line/column) and only show the comment help (and fpdoc if available).

If you move the mouse over the actual usage of these names then it will show all 3 parts of the hint and not throw the exception. The Exception happens somewhere deep within TFindDeclarationTool.FindSmartHint() or .FindDeclaration() and is probably because it is already sitting at the declaration and cant find a declaration for the declaration itself.

This impacts the use in the CodeBrowser since we point them all directly at the declarations and so it will never give a smarthint and cannot fully replace the old default hint from the code browser. I'm sure this is only a little forgotten IF ELSE somewhere in FindSmartHint() that would do the trick but I can't easily find the spot where (how) to do this.

Bernd

Fixed in svn rev 26882.

Mattias

probably because it wasn't there until your previous post(s)?? ;)

Mattias Gaertner wrote

Fixed in svn rev 26882.

OK. Here is the rest to activate the new hint windows. I can't find any unwanted or faulty behavior anymore.

Bernd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: better-hints-codebrowser-v3.diff
Type: text/x-diff
Size: 4581 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100729/8a15c4c1/attachment.bin>

Thanks. Applied.

Mattias

Mattias Gaertner wrote

Thanks. Applied.

Thanks.

I have noticed that the html hints (at least on GTK2) look quite ugly: They have a white background and absolutely no border.

The following is a hack to make them look like they should at least on GTK2 (I have *not* tested this on windows and or any other widgetset). I don't even know whether this (or a similar) hack can fit into the bigger concept of the hint provider thing, but it fixes the display problems for me. (on GTK2)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: html-hint-color-and-border.diff
Type: text/x-diff
Size: 1172 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100730/4d896086/attachment.bin>

They are painted with your theme.
Here they look exactly like hints from other applications. Yellowish background.
Hints have no border.

Mattias

Mattias Gaertner wrote

They are painted with your theme. Here they look exactly like hints from other applications. Yellowish background. Hints have no border.

The nomal hints all look ok and have the yellowish color of clInfoBk and a thin border of 2 pixel (bsSingle?) around them and there is a spacing of a few pixels betwen the text and the border, only when it uses the html control from turbopoweripro then the hint is a white rectangle and has absolutely no spacing between the border and the text.

See the screenshots:
html-hint without borders: http://imagebin.org/107389 all other hints look ok: http://imagebin.org/107390

setting aHintWindow.BorderWidth to anything bigger than 0 will make the thin borer line visible again and make this amount of padding between the border and the html control.

http://imagebin.org/107395

and explicitely setting BGCOLOR in the BODY tag to clInfoBk will also fix the background color:

http://imagebin.org/107397

I have tried to explicitly set the default background color to clInfoBk and a few pixels padding directly in the html control in the turbopoweripro package but it will simply ignore it. Maybe it is a bug in turbopoweripro and happens only on GTK. At the weekend I have some time at my windows PC and see how it looks there.

Also I just remember I have recently set my color depth to 16 bits only because of problems with another application, maybe this is what triggers this behavior. Also I have noticed that the turbopoweripro will throw an exception everytime I use a <table> inside the HTML and it says something with GTK2 and GetSysColor() and it received 32 while only 0..31 are allowed or something like that.

It is not so important, it is just that I pay attention to every little detail and this one annoyed me a bit. If it is only because of my odd color depth that almost nobody else uses then i understand that this has no priority. I can live with it, the most important thing is that the Information itself is displayed somehow and this is the case.

Maybe I should prepare a minimal example if i have some time and some more information about what exactly it is that does not behave how it should (or how it is expected to behave) and file a bug against the turbopoweripro package.

Yes, you are right, the turbopower ipro needs improvement. And the produced html itself needs improvement too.

Mattias