I am new to Pascal programming and since I like using Emacs, I wanted to try to improve its support for editing Pascal code. I decided to implement an LSP server which text editors can use as back-end for providing IDE features. The server is implemented in Free Pascal and uses the CodeTools package for doing the actual work.

The implementation is still incomplete and unstable but so far I managed to get code completion working. Apart from improving stability and adding features, I am sure there are more possible improvements, since this is my first Pascal project. The project can be found at https://github.com/arjanadriaanse/pascal-language-server [1], any help and feedback is welcome.

A module is provided for using the server from Emacs, but it should be straightforward to add support for other LSP clients such as Visual Studio Code.

Regards,

Arjan

[1] https://github.com/arjanadriaanse/pascal-language-server

yes! Pascal *badly* needs this to be productive outside of Lazarus. I use Sublime Text and frequently now VSCode but not having good code tools is getting painful.

In fact I already started a language server some months ago (and since been busy with out stuff) but I think you've made more progress than me so I'll help to develop yours if you would like. I was even asking on the list about Lazarus code tools and was showed how to FPC's official parser works.

Please let me know what else needs to be done and I'll see if I can contribute.

Thanks, really.

Regards,
Ryan Joseph

Ryan Joseph wrote

yes! Pascal *badly* needs this to be productive outside of Lazarus. I use Sublime Text and frequently now VSCode but not having good code tools is getting painful.

VSCode has omnipascal ?
(The TMS Web core VS Code studio plugin uses this, and it seems pretty complete)

In fact I already started a language server some months ago (and since been busy with out stuff) but I think you've made more progress than me so I'll help to develop yours if you would like. I was even asking on the list about Lazarus code tools and was showed how to FPC's official parser works. Please let me know what else needs to be done and I'll see if I can contribute.

Very nice !

I suppose this can be used as a back-end for Atom and VS Code as well then ? That would be really cool !

A good excuse to dive into Atom package development using pas2js :-)

Michael.

On Apr 23, 2020, at 2:08 PM, Michael Van Canneyt wrote: VSCode has omnipascal ? (The TMS Web core VS Code studio plugin uses this, and it seems pretty complete)

yes it does but they don't support ObjFPC mode syntax very well which means things like the "generic" keyword and "is nested" to name a couple, break the parser and render the entire document invalid.

It's also closed source and likely to be for profit eventually which is a deal break imo. A language server for FPC is too important to be allowed to be within a single companies control.

yes exactly. See the docs https://microsoft.github.io/language-server-protocol/specifications/specification-3-14. I think currently he supports just code completion but there's a whole lot more.

LSP is a good idea and thankfully backed by a large company but it's painfully slow because the client needs to send the entire document in plain text to the server which then needs to be parse the entire thing, and this happens every time the document means with every keystroke the document is parsed and analyzed. The FPC parser doesn't support this anyways but the protocol doesn't even support sending partial changed text offsets. I talked in length with the LSP people at Sublime Text and this is just how it's going to be for now.

LSP is very new however so expect it to improve in the future.

A good excuse to dive into Atom package development using pas2js :-)

Regards,
Ryan Joseph

Arjan, I briefly skimmed over your code and it looks like instead of using the "DocParser" unit that I was using the parse the files, you're directly using the Lazarus "CodeToolBoss" which I don't have any experience with.

I think it was Michael actually who instructed me to use this parser but I also think he said the Lazarus code tools use this under the hood.

What are the implications of using the code tools as opposed to implementing the parser directly? If you have documents you could direct me to about the code tool units that would be helpful. I have an ok understand of LSP now and I want to make sure the code tools is able to supply everything we need.

Regards,
Ryan Joseph

Ryan Joseph wrote

Arjan, I briefly skimmed over your code and it looks like instead of using the "DocParser" unit that I was using the parse the files, you're directly using the Lazarus "CodeToolBoss" which I don't have any experience with. I think it was Michael actually who instructed me to use this parser but I also think he said the Lazarus code tools use this under the hood.

I don't think I did the latter part, since the codetoolboss is not using the fcl-passrc.

There are several parsers:

* docparser - not linked to FPC/Lazarus.
(I can't comment on the completeness, but I would suppose it's very complete)

* fcl-passrc - used in pas2js so very complete.

* I suppose the JCF has its own parser.

* lazarus codetoolboss - used in Lazarus IDE.

Probably the lazarus codetoolboss is best suited for the job since it is designed to work with incomplete sources. The others are all designed to work with completed and known-to-be-correct sources.
e.g. fcl-passrc normally stops on the first error.

I think the decision to use the codetoolboss is the best, see above as to why.

Michael.

I also noticed this !

This is exactly why I prefer not to use it, and why I applaud this native implementation :-)

As for the missing ability to send partial updates: that's up to the "big company" to fix.

Come to think of it, I suppose this is one more reason why 'modern' web development consists out of 1000-s of small files with essentially 1 function... To keep LSP happy :)

Michael.

On Apr 23, 2020, at 3:00 PM, Michael Van Canneyt wrote: I think the decision to use the codetoolboss is the best, see above as to why.

Very well. I assume the code tools parses the entire project and keeps the symbols in a database? That was something I was going to have to implement myself so it's good to not have to now.

I'm using the parser with "TPasElement" so I think that's fcl-passrc. It's a shame to have so many competing parsers but yours looked really solid to me.

Here the basics which code tools needs needs to provide:

1) Basic identifier completions like function, class, types names etc... these should be scope sensitive based on the document offset we receive from LSP.

2) Function parameter completions.

3) Class/record member completions.

4) Identifier references, i.e return all locations "DoThis" was called in the current workspace. (Workspace is the LSP term for the project). Code tools thus needs to parse all files in the workspace upon initialization.

5) Go to definition/declaration in workspace.

I would like to implement "code lens" which shows the context of a symbol in a popup window but I think only VScode supports this feature and it's no common so code tools may not be able to retrieve.

Regards,
Ryan Joseph

Well, it is solid, but its target is to compile.

Michael.

These parsers are optimized for different purposes. fcl-passrc is for parsing like a compiler. It parses everything everytime and has neither functions to alter code nor code completion. codetools is for parsing only needed unit interfaces/implementations and when a file changes, parse only things that have changed. And it provides many code completions.

Here the basics which code tools needs needs to provide: 1) Basic identifier completions like function, class, types names etc... these should be scope sensitive based on the document offset we receive from LSP.

IdentifierCompletion

2) Function parameter completions.

FindCodeContext
see lazarus/ide/codecontextform.pas for an example.

3) Class/record member completions.

I guess you mean identifier completion after a dot.

4) Identifier references, i.e return all locations "DoThis" was called in the current workspace. (Workspace is the LSP term for the project). Code tools thus needs to parse all files in the workspace upon initialization.

FindReferences

5) Go to definition/declaration in workspace.

FindDeclaration

I would like to implement "code lens" which shows the context of a symbol in a popup window but I think only VScode supports this feature and it's no common so code tools may not be able to retrieve.

What do you mean with "context"? Codetools can give you all kind of context.

Mattias

Exactly. Fields or methods and helpers even if possible. I assume that exists.

Btw, the names you're giving like "FindDeclaration" where can I find these? Maybe there's a wiki or unit header I can look at.

> I would like to implement "code lens" which shows the context of a symbol in a popup window but I think only VScode supports this feature and it's no common so code tools may not be able to retrieve. What do you mean with "context"? Codetools can give you all kind of context.

For example VScode lets you hover over a symbol name like TObject and a window will popup that shows the declaration in code, like a mini-editor that goes to the line where the declaration starts in that file. It's basically the same as opening a new tab/window but it's more for really quick references. I'm not sure how it's implemented in LSP but maybe you could return the line number and file name and the client does the rest.

Regards,
Ryan Joseph

https://wiki.freepascal.org/Codetools
And then look at the examples in lazarus/components/codetools/examples.

> > I would like to implement "code lens" which shows the context of a symbol in a popup window but I think only VScode supports this feature and it's no common so code tools may not be able to retrieve. > What do you mean with "context"? Codetools can give you all kind of context. For example VScode lets you hover over a symbol name like TObject and a window will popup that shows the declaration in code, like a mini-editor that goes to the line where the declaration starts in that file. It's basically the same as opening a new tab/window but it's more for really quick references. I'm not sure how it's implemented in LSP but maybe you could return the line number and file name and the client does the rest.

FindDeclaration gives you the line and column. You can also get the source span of the declaration. And there is FindSmartHint, which returns a brief text for a declaration including some context like type/var/const, private/public, parent class, etc.

Mattias

Seems like exactly the code lens functionality :-)

Michael.

El 23/4/20 a les 10:05, Michael Van Canneyt ha escrit:

Unless it's java, then every one of those 1000 files will be 2000 lines long just to spit out an "Hello world" :-D

SCNR (I've been assigned a project java related).

Bye

Glad to know this is needed and thank you for your interest.

I made some initial issues I could think of on the repository page at https://github.com/arjanadriaanse/pascal-language-server/issues [1]. Anyone is free to pick these up or create new issues, for example feature requests. Regardless of these issues, any pull requests are welcome and you can also email me patches.

Apart from that, having a client for Visual Studio Code is useful for many people, which would be a separate project.

Also, any feedback on my existing code is welcome, especially since this is my first Pascal project.

Regards,

Arjan

[1] https://github.com/arjanadriaanse/pascal-language-server/issues

Arjan Adriaanse schrieb am Do., 23. Apr. 2020, 08:33:

I had already written this in the forum, but I definitely welcome this. Good work! Hopefully we can get it stable and complete.

What might be interesting as well for Lazarus might be LSP client support. Not for Pascal, but this way we could make us of other language's LSPs to provide code completion for it in Lazarus as well. :)

Regards,
Sven

Good idea. Saves Mattias some work ;-)

Michael.

On Apr 23, 2020, at 6:29 PM, Arjan Adriaanse wrote: Also, any feedback on my existing code is welcome, especially since this is my first Pascal project.

You did a nice job structuring it I think and providing a template to scale up from. What language(s) are you coming from and why did you decide to make a language server for Pascal if you're not a Pascal user? Just curious.

I wrote my list of missing protocol implementations so that is what I'll be focusing on initially.

I'm trying to build this now using lazbuild on macOS and get some strange errors including:

lsp.pas:154:12: error: Identifier not found "specialize"

How could this be? {$mode objfpc} is declared right at the top.

Another curios one is:

lsp.pas:134:33: error: Incompatible types: got "<class method type of procedure(TObject;TObject;PPropInfo;TJSONObject;UTF8String;var TObject) of object;Register>" expected "<procedure variable type of procedure(TObject;TObject;PPropInfo;TJSONObject;UTF8String;var TObject);Register>"

I'm not usually a Lazarus user but I think lazbuild should work. Here's the command I use:

lazbuild "./pasls.lpi" --cpu=x86_64 --widgetset=cocoa

I use the cocoa widgetset on Mac because I get errors otherwise. That shouldn't be related though.

Regards,
Ryan Joseph

You did a nice job structuring it I think and providing a template to scale up from. What language(s) are you coming from and why did you decide to make a language server for Pascal if you're not a Pascal user? Just curious.

Thank you. I am coming from Java/.NET/Python/Haskell and was introduced to Free Pascal when porting this Delphi game to Linux: https://github.com/arjanadriaanse/Lemmix [1]

The project was started mainly to get more familiar with the language and to eventually be less restricted to the use of the Lazarus IDE.

Perhaps not all the used generics features are available yet in older FPC versions. I use the most recent release candidate from ftp://ftp.freepascal.org/pub/fpc/beta/3.2.0-rc1/ [2], but will investigate if older versions also work.

Regards,

Arjan

[1] https://github.com/arjanadriaanse/Lemmix
[2] ftp://ftp.freepascal.org/pub/fpc/beta/3.2.0-rc1/

Lazbuild used FPC 3.0.4. How do I make lazbuild choose the trunk instead of 3.0.4?

The first thing to address is TCompletion.Process and how it interacts with code tools. For example:

    URI := ParseURI(textDocument.uri);
    Code := CodeToolBoss.FindFile(URI.Path + URI.Document);
    X := position.character;
    Y := position.line;
    Line := Code.GetLine(Y);
    GetIdentStartEndAtPosition(Line, X + 1, PStart, PEnd);
    CodeToolBoss.IdentifierList.Prefix := Copy(Line, PStart, PEnd - PStart);

How LSP is intended to work is that a text buffer which contains the current state of the file is sent to the server and then this buffer is parsed. I think what the code above does is only operate on the file on disk. This maybe be fine for certain completions but it will be static until the user saves the file in the client. Worse yet is that the text offset we receive from the client is going to be out of sync.

Here's an example request:

{
"method" : "textDocument/didChange",
"jsonrpc" : "2.0",
"params" : {
"textDocument" : {
"uri" : "file:///Users/ryanjoseph/Desktop/FPCLS-Test/parser_test.pas", "version" : 1
},
"contentChanges" : [
{
"text" : "..."
}
]
}
}

and a hover request which shows the problem. Line 10 is from the didChange request, not line 10 from the file on disk.

{
"method" : "textDocument/hover",
"jsonrpc" : "2.0",
"id" : 3,
"params" : {
"textDocument" : {
"uri" : "file:///Users/ryanjoseph/Desktop/FPCLS-Test/parser_test.pas" },
"position" : {
"line" : 10,
"character" : 13
}
}
}

Regards,
Ryan Joseph

Use the compiler option, for example:

lazbuild --compiler=/usr/lib/fpc/3.2.0/ppcx64 pasls.lpi

The CodeToolBoss actually keeps a cache of files which are loaded at the 'textDocument/didOpen' notification. When the user changes text at the client side, even without saving, a 'textDocument/didChange' notification is sent which is used to update the cache. The CodeToolBoss seems to only use the file path as identifier, but the contents of the files and changes are provided by the client.

Regards,

Arjan

Help:
lazbuild -h

lazbuild --compiler=/usr/lib/fpc/3.3.1/ppcx64

Codetools checks used files and automatically reloads them.

If LSP updates parser_test.pas, then codetools will notice.

If LSP only sends changes, then you have to apply these changes to codetools codebuffers:
Code.Insert/Delete/Replace/Move

Modified codebuffers are not auto updated from disk, until you Revert or Save.

Mattias

This is what we need to do then otherwise nothing will work. LSP for some insane reason doesn't even provide a changed text, just the full document every time. I guess we need to use Replace every time. I talked with the Sublime Text people and they since isn't even supported from their API so even if LSP supported it there will be clients that don't provide the information anyways.

Regards,
Ryan Joseph

Btw, I guess you mean LoadFile, not FindFile.

Mattias

Excellent, I didn't see that yet.

Ugh, I rebuild using trunk 3.3.1 and now get this error:

/Users/ryanjoseph/Developer/lazarus/lcl/./include/custompage.inc:139:34: error: Operator is not overloaded: "Class Of TCustomTabControl" and "Boolean"

I guess we need to use 3.2 and nothing else. I didn't even know 3.2 existed so I need to download it.

Regards,
Ryan Joseph

On Apr 23, 2020, at 9:11 PM, Arjan Adriaanse wrote: lazbuild --compiler=/usr/lib/fpc/3.2.0/ppcx64 pasls.lpi

is there a macOS installer for 3.2? I found a link to a FTP server but I don't see this exists. Maybe fix that bug in trunk is the better idea. My lazarus sources are old also so perhaps this was fixed?

Regards,
Ryan Joseph

Not sure about an installer for macOS, but you should be able to build FPC 3.2.0 from the source found here
ftp://ftp.freepascal.org/pub/fpc/beta/3.2.0-rc1/source/ [1] using your older compiler. Also note that I use the latest Lazarus release 2.0.8, because older versions did not seem to build using this FPC version.

Regards,

Arjan

[1] ftp://ftp.freepascal.org/pub/fpc/beta/3.2.0-rc1/source/

I just tried update my lazarus sources using "svn up" and now I'm getting another strange error when using lazbuild.

/Users/ryanjoseph/Developer/lazarus/components/ideintf/ideintf.pas:11:3: fatal: Can't find unit LazarusPackageIntf used by IDEIntf

Do I need to do something else to get the sources updated?

Regards,
Ryan Joseph

You also need to update your Lazarus version:
https://github.com/graemeg/lazarus/commit/0bf882fb5b71fb3288ef5571998617fe8a5d3cd0

Am Do., 23. Apr. 2020 um 15:22 Uhr schrieb Ryan Joseph via fpc-pascal <

You have old ppu files.

For example:
make distclean all

Mattias

ok, did this and now getting:

/Users/ryanjoseph/Developer/lazarus/components/lazdebuggergdbmi/gdbmidebugger.pp(63,16) Fatal: (10022) Can't find unit PropEdits used by GDBMIDebugger

svn up says "At revision 63051."

Never an easy time building lazarus for me. :)

Regards,
Ryan Joseph

Please post the last couple of lines, e.g. starting at the last ppcx64 call.

svn up says "At revision 63051." Never an easy time building lazarus for me. :)

Do you have local modifications?

Mattias

/bin/cp -f Makefile.compiled units/x86_64-darwin/nogui/IDEIntf.compiled /Applications/Xcode.app/Contents/Developer/usr/bin/make -C components/lazdebuggergdbmi LCL_PLATFORM=nogui /bin/rm -f units/x86_64-darwin/nogui/lazdebuggergdbmi.ppu /usr/local/bin/ppcx64 -MObjFPC -Scghi -O1 -g -gl -l -vewnhibq -dLCL -dLCLnogui -gw -Fu../../packager/units/x86_64-darwin -Fu../lazutils/lib/x86_64-darwin -Fu../buildintf/units/x86_64-darwin -Fu../../lcl/units/x86_64-darwin -Fu../debuggerintf/lib/x86_64-darwin -Fu../../lcl/units/x86_64-darwin/nogui -Fu../lazcontrols/lib/x86_64-darwin/nogui -Fu../ideintf/units/x86_64-darwin/nogui -Fu. -Fu/usr/local/lib/fpc/3.0.4/units/x86_64-darwin/rtl -FE. -FUunits/x86_64-darwin/nogui -dx86_64 lazdebuggergdbmi.pas Hint: (11030) Start of reading config file /etc/fpc.cfg Hint: (11031) End of reading config file /etc/fpc.cfg Free Pascal Compiler version 3.0.4 [2018/09/30] for x86_64 Copyright (c) 1993-2017 by Florian Klaempfl and others (1002) Target OS: Darwin for x86_64
(3104) Compiling lazdebuggergdbmi.pas
(3104) Compiling gdbmidebugger.pp
/Users/ryanjoseph/Developer/lazarus/components/lazdebuggergdbmi/gdbmidebugger.pp(63,16) Fatal: (10022) Can't find unit PropEdits used by GDBMIDebugger Fatal: (1018) Compilation aborted
make[1]: *** [lazdebuggergdbmi.ppu] Error 1
make: *** [lazbuild] Error 2

I did have some but I thought I got them merged in correctly. Perhaps not? I see the missing filling sitting right there:

/Users/ryanjoseph/Developer/lazarus/components/ideintf/propedits.pp

Regards,
Ryan Joseph

You can find the macOS installer here. Despite it's name it also contains the x86_64-darwin one.

Regards,
Sven

On Thu, 23 Apr 2020, 21:43 Ryan Joseph via fpc-pascal, <

I usually revert to brute force troubleshooting in these instances: pass - va to the compiler. A lot of information, but usually the problem is close to the end so scanning the output in reverse should often reveal what is wrong.

OK, I just downloaded the laz sources and rebuilt from scratch (as usual) and it's building with the 3.3.1 now.

I'm testing the server in Sublime Text and first thing I get is:

LSP: invalid response payload {'id': 1, 'result': {'capabilities': {'completionProvider': {'triggerCharacters': None, 'allCommitCharacters': None, 'resolveProvider': False}, 'textDocumentSync': {'openClose': True, 'change': 1}}}, 'jsonrpc': '2.0', 'error': None}

I assume this is the response from the initialize message being malformed. What client were you testing in when developing?

Regards,
Ryan Joseph

On Apr 24, 2020, at 9:52 AM, Ryan Joseph wrote: LSP: invalid response payload {'id': 1, 'result': {'capabilities': {'completionProvider': {'triggerCharacters': None, 'allCommitCharacters': None, 'resolveProvider': False}, 'textDocumentSync': {'openClose': True, 'change': 1}}}, 'jsonrpc': '2.0', 'error': None}

Just tested on my language server and the problem is you provided a null error key, which is not in the protocol.

I've never seen the fpjsonrpc unit before so I don't know where this got inserted but it needs to be removed.

Regards,
Ryan Joseph

I think we need to override a method to remove the null error, like this:

function TLSPDispatcher.FormatResult(Const AClassName, AMethodName: TJSONStringType;
Const Params,ID, Return : TJSONData) : TJSONData;
begin
  Result:=TJSONObject.Create([
    'result',Return,
    TransactionProperty,ID.Clone
    ]);
  if jdoJSONRPC2 in options then
    TJSONObject(Result).Add('jsonrpc','2.0');
end;

but I'm still not getting any results from Sublime Text. The bigger problem now however is that I'm not seeing stderr message in Sublime Text so I have no idea what's going on. There's apparently a new discord server for the LSP package so I'm going to ask questions here if you want to follow along (see link below). So anyways, I give up until I can get error messages. :) My server is socket based so I could run it in an terminal to see the output but I have no experience with stdio servers. If you have any means try Sublime Text for yourself and see if you can figure it out. The editor your using must be working though....

https://discordapp.com/channels/280102180189634562/645268178397560865

Regards,
Ryan Joseph

I believe the OP was using Emacs.

Michael.

I fixed TCustomJSONRPCDispatcher not to add a Null in case of V2. But V1 needs it. see https://www.jsonrpc.org/specification_v1

Committed in revision 45047.

Michael.

Nice, it was a bug. :)

So the Sublime Text people explained there was a special panel to get errors so I am seeing them now. Also it's important to flush the stderr pipe after writing to it or the messages will come out of order.

However, I'm still not able to get completions despite this being the one feature which is claimed to be supported. I don't know if this is because of more errors or the completions are not what I think they are. Here is the code tools related function which gets completions at the current text offset.

What does this do exactly? If there is an identifier "DoThis" in the current scope (say a function name) does this return "DoThis" if the cursor is after the word "Do"? Previously it was said "IdentifierCompletion" does this but I haven't looked into the code tools yet.

function TCompletion.Process(var Params: TCompletionParams): TCompletionList;
var
  URI: TURI;
  Code: TCodeBuffer;
  X, Y, PStart, PEnd, Count, I: Integer;
  Line: string;
  Completions: TCompletionItems;
  Identifier: TIdentifierListItem;
  Completion: TCompletionItem;
begin with Params do
  begin
    URI := ParseURI(textDocument.uri);
    Code := CodeToolBoss.FindFile(URI.Path + URI.Document);
    X := position.character;
    Y := position.line;
    Line := Code.GetLine(Y);
    GetIdentStartEndAtPosition(Line, X + 1, PStart, PEnd);
    CodeToolBoss.IdentifierList.Prefix := Copy(Line, PStart, PEnd - PStart);
    Completions := TCompletionItems.Create;
    if CodeToolBoss.GatherIdentifiers(Code,X + 1,Y + 1) then
    begin
      Count := CodeToolBoss.IdentifierList.GetFilteredCount;
      for I := 0 to Count - 1 do
      begin
        Identifier := CodeToolBoss.IdentifierList.FilteredItems[I];
        Completion := TCompletionItem(Completions.Add);
        Completion.insertText := Identifier.Identifier;
        Completion.detail:=Identifier.Node.DescAsString;
        Completion.insertTextFormat := TInsertTextFormat.PlainText;
      end;
    end else begin
      if CodeToolBoss.ErrorMessage'' then
        writeln(stderr, 'Parse error: ',CodeToolBoss.ErrorMessage)
      else
        writeln(stderr, 'Error: no context');
    end;
    Result := TCompletionList.Create;
    Result.items := Completions;
  end;
end;

Regards,
Ryan Joseph

In my test I'm getting an error at this block, "Parse error: unit not found: fpextres".

Does code tools need access to this unit? It's not part of the source I've supplied it so I don't know why it's looking for this.

Regards,
Ryan Joseph

This unit is responsible for reading resources and is part of FPC. It's probably used by one of the units you use.

Michael.

fpextres.ppu is installed by default on darwin.
Maybe you didn't setup codetools for your fpc units. Because of the vast amount of fpc units, codetools needs to scan them once. There is a simple init function, that uses the common environment variables PP, FPCDIR and PATH to find the fpc executable and the fpc sources. It creates a map ppu file to unit source and stores the result in the ConfigFilename. So next time it can quickly read it.

CodeToolBoss.SimpleInit('~/.ChooseACacheFile');

If you need more control, look into the SimpleInit function.

Keep in mind that most codetools are not threadsafe.

Mattias

I don't use Lazarus and just killed my entire install last night when I tried to rebuild. I guess I can do this in Lazarus so I'll check now.

Can this not be omitted? For non-Lazarus users it's going to be a problem. That means the language servers requires Lazarus to very good if we an avoid this.

Regards,
Ryan Joseph

What do you want to omit exactly ?

For non-Lazarus users it's going to be a problem. That means the language servers requires Lazarus to very good if we an avoid this.

It does not require lazarus, but it does require the FPC source tree.

Michael.

On Apr 24, 2020, at 2:54 PM, Michael Van Canneyt wrote: What do you want to omit exactly ?

For example, I'm not a lazarus user but I'm getting strange errors that don't relate to my single file program. That's going to confuse users like it did me. I'd like to omit the need to configure anything in Lazarus if possible.

> For non-Lazarus users it's going to be a problem. That means the language servers requires Lazarus to very good if we an avoid this. It does not require lazarus, but it does require the FPC source tree.

Ok then. How do I setup code tools on the FPC source tree?

Regards,
Ryan Joseph

These are the right paths. There should be a propedits.ppu in ideintf/units/x86_64-darwin/nogui

Are there any ppu files left after "make distclean"?

Mattias

On Apr 24, 2020, at 3:32 PM, Mattias Gaertner via fpc-pascal wrote: Are there any ppu files left after "make distclean"?

I just trashed the entire thing and started over. It was easier that way, much easier. It's working now so I'm happy. :)

I'm still getting that code tools error though so I wonder if it's related.

Regards,
Ryan Joseph

The options for initializing CodeToolsBoss are passed through environment variables as described at
https://wiki.lazarus.freepascal.org/Codetools [1].

For example, mine are set up as follows through the Emacs client settings.

FPCDIR="/usr/lib/fpc/3.2.0/source"
LAZARUSDIR="/usr/share/lazarus"

However, specifying the Lazarus source path does not seem to suffice to find the units of a component like CodeTools. Therefore I would recommend testing on simple code without many dependencies for now.

Regards,

Arjan

[1] https://wiki.lazarus.freepascal.org/Codetools

On Apr 24, 2020, at 3:59 PM, Arjan Adriaanse wrote: FPCDIR="/usr/lib/fpc/3.2.0/source" LAZARUSDIR="/usr/share/lazarus"

I see now! I didn't know that but it's easy to document once we get that far.

btw, I'm having a problem with the fact that published un-set properties are being set with null values. Some of the fields are marked as optional in the protocol and returning a NULL value could be an error which crashes the client. I don't have much experience with streaming RTTI so I don't know how to resolve this.

For example this "documentation" property will always be set to null but really what want is to omit the key entirely.

 published
    property documentation: TMarkupContent read fDocumentation write fDocumentation;

Regards,
Ryan Joseph

Indeed with the current way JSON streaming is handled, optional values are not considered. This needs more work and the issue is tracked here https://github.com/arjanadriaanse/pascal-language-server/issues/7 [1].

Regards,

Arjan

[1] https://github.com/arjanadriaanse/pascal-language-server/issues/7

You don't need to configure lazarus ?

You must configure the codetools used in the LSP.

This is something Mattias needs to answer.

Michael.

We may need to rip up your nice streaming system and iterate over properties then. Sven recently added custom attributes in the trunk which we could use to mark the published properties (I think) but then we're inside the JSON RPC library and we may lose control over this process.

Regards,
Ryan Joseph

The LAZARUSDIR merely sets a macro. There is no currently no code in codetools that scans Lazarus packages.

In Short: You can omit LAZARUSDIR.

Setting PP might be useful.

You can add unit and include paths:
https://wiki.freepascal.org/Codetools#Defining_search_paths_and_macros

Mattias

See my mail about SimpleInit.

Mattias

When I hover over a function in Lazarus I get a function definition and unit location. Is that coming from TCodeManager.FindSmartHint?

Regards,
Ryan Joseph

Setting FPCDIR from the editor fixed the problem. I'm not crazy about getting tons of completions from the System module though. Lots of clutter for things I never use. Is there a way to hide those?

Regards,
Ryan Joseph

When I hover over a function in Lazarus I get a function definition and unit location. Is that coming from TCodeManager.FindSmartHint?

Yes

Mattias

You'd need nullable types for 'optional' values.

Your streaming library needs to be able to decide whether or not to stream an integer. AFAIK you could do that based on an attribute indicating a sentinel value which should not be streamed.
You can do this with today's implementation with the OnStreamProperty event and some dictionary of 'optional' properties.
TMyOptionals = specialize TDictionary<String,String>; and as key use ClassName.PropName, and as value the sentinel value not to stream. Bit of a roundabout way, but at least can be done with the released FPC.

I don't think that requiring trunk would be a good idea for this project.

Michael.

I'm not able to get this one working then. Is the buffer the current file? This is what I'm doing:

    URI := ParseURI(textDocument.uri);
    Code := CodeToolBoss.FindFile(URI.Path + URI.Document);
    X := position.character;
    Y := position.line;
    hint := CodeToolBoss.FindSmartHint(Code, X, Y);

Just returns an empty string but maybe the code buffer is not correct.

Regards,
Ryan Joseph

In your loop
Completion.detail:=Identifier.Node.Desc...

Note: Always check if Nodenil.

You can check the unit via

Tool:=Identifier.Tool;
If Toolnil then
  if sametext(ExtractFilenameOnly(Tool.MainFilename),'system') then
    ...hide...

Mattias

Op 24-04-2020 om 11:25 schreef Arjan Adriaanse:

Indeed with the current way JSON streaming is handled, optional values are not considered. This needs more work and the issue is tracked here https://github.com/arjanadriaanse/pascal-language-server/issues/7 [1].

I did not have a look at your code, but if you have serialization-issues, you might consider looking at Cerialization. (https://gitlab.freepascal.org/Joost/cerialization)

It is what I use for my debugging-project. And it easier to adapt and extend as the json-streaming classes that come with fpc. (Because you always have to wait for the next fpc-version)

Regards,

Joost.

I just barely understand the FPC serialization so I'll let Arjan decide what he wants to do on that. I does need to be resolved though because I've already had bugs appear in Sublime Text but passing in optional values as null objects.

Regards,
Ryan Joseph

Never mind, the line offsets are zero based so I needed to + 1 for code manager. It's working now.

Regards,
Ryan Joseph

Still FindFile instead of LoadFile?

Do you get all mails from this list?

Mattias

Is there a demo somewhere?

Mattias

I don't know what the code buffer technically is but FindFile works after the source was set previously. I assumed it was just a reference to the current file which corresponds to the line/column numbers.

Sorry, what do you want to see a demo of? The language server?

Regards,
Ryan Joseph

Ah. I didn't see any source set. ok. Thanks for the explanation.

Yes.

Mattias

Op 24-04-2020 om 16:44 schreef Mattias Gaertner via fpc-pascal:

I don't know if Ryan has some changes of his own, but the project can be found here:

https://github.com/arjanadriaanse/pascal-language-server Regards,

Joost.

The first message has the GitHub page. Of course the bugs and changes I've made are not included but I will put them up on GitHub over the weekend. What editor do you want to test on? I only know Sublime Text and Arjan can help with emacs (which i never used even once!). I'll learn VSCode eventually but its been enough to get this working with Sublime Text.

Regards,
Ryan Joseph

In the FPC streaming system with published properties, what type do I use for an array of strings? I've tried TStringList and TCollection but they give me errors like:

:: <~~ pascal-language-server 1: {'message': ' : Unsupported JSON type for object property: "jtArray"', 'code': -32603}

"initializationOptions" is a JSON array of strings, i.e. 'initializationOptions': ['-Fu/path/to']

  TInitializeParams = class(TPersistent)
  private
    fRootUri: string;
    fCapabilities: TClientCapabilities;
    fInitializationOptions: TCollection;
  published
    property rootUri: string read fRootUri write fRootUri;
    property capabilities: TClientCapabilities read fCapabilities write fCapabilities;
    property initializationOptions: TCollection read fInitializationOptions write fInitializationOptions;
  end;

Regards,
Ryan Joseph

I have FindCodeContext working correctly I believe because ProcName from TCodeContextInfo returns the correct name of the function. In TCodeContextInfoItem the params string list is nil though. Shouldn't that contain the list of parameters? Not sure how this intended to be used if that list of empty.

Regards,
Ryan Joseph

Is X,Y inside the brackets?

Mattias

You mean ()? Yes it is. ParameterIndex also increments so for example

DoThis(a,b,c,^)

will give ParameterIndex 4 if the cursor is at ^.

Is Params supposed to be a list of the parameters in the format "name: type"? If so that's what I need to return along with the parameter index. See

https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocument_signatureHelp

Regards,
Ryan Joseph

The TCodeContextInfoItem.ProcName is set only for compiler built-in functions.
Normal functions have an Exprnil.

See procedure TCodeContextFrm.CreateHints in
lazarus/ide/codecontextform.pas

Mattias

Op 25-04-2020 om 05:53 schreef Ryan Joseph via fpc-pascal

As I said before, the Cerialization-library is written with this kind of use-cases in mind. (https://gitlab.freepascal.org/Joost/cerialization)

Although that what you want should be possible with the default libraries using TStringList and some events...

Regards.

Joost.

Hmm. Maybe "make distclean" does not remove all files. I wish, you had kept a copy.

Mattias

On Apr 25, 2020, at 6:51 PM, Mattias Gaertner via fpc-pascal wrote: Hmm. Maybe "make distclean" does not remove all files. I wish, you had kept a copy.

It'll probably happen again. ;)

Since we're so invested in code tools now what is actually required for users in terms of Lazarus? Can we just package a single binary which links to code tools or do users need to install an entire Lazarus setup?

Regards,
Ryan Joseph

Is the following line something which can be cached? Currently this is going to be called very often but is it an expensive operation? Not sure what it does exactly....

Code := CodeToolBoss.FindFile(URI.Path + URI.Document);

Regards,
Ryan Joseph

It's gpl/lgpl. Feel free to use parts.

Mattias

It does a binary search on the cached files. Should be barely measurable.

Mattias

On Apr 25, 2020, at 7:48 PM, Mattias Gaertner via fpc-pascal wrote: It's gpl/lgpl. Feel free to use parts.

sure, but does it does it depend on other systems which are installed from Lazarus or will a stand-alone binary run on a system that never installed Lazarus?

Regards,
Ryan Joseph

It will run on a server that never installed lazarus.

Michael.

On Apr 25, 2020, at 8:49 PM, Michael Van Canneyt wrote: It will run on a server that never installed lazarus.

Excellent. Where are the caches kept btw? I thought they were in the Lazarus directory but I guess not.

Regards,
Ryan Joseph

I suspect in a file in ~/.lazarus or ~/.yourprojectname, probably the latter.

Michael.

Memory.

Mattias

I'm still stuck on this problem. I can get the full proc head using CodeTool.ExtractProcHead but I can't get the name/type parameter pairs in a list (unless I do something stupid like parse the string from ExtractProcHead). I need to return the parameters in discrete pairs in text that matches the format of the header. I know ExtractProcHead finds this information but it's lost in the function before it returns.

function ExtractContext(CurContext: TCodeContextInfoItem; var ParamPairs: TStringList): String;
  var
    Code: String;
    CurExprType: TExpressionType;
    CodeNode, ChildNode: TCodeTreeNode;
    CodeTool: TFindDeclarationTool;
    ParamList: TCodeTreeNode;
  begin
    CurExprType:=CurContext.Expr;
    Code:=ExpressionTypeDescNames[CurExprType.Desc];
    if CurExprType.Context.Nodenil then
      begin
        CodeNode:=CurExprType.Context.Node;
        CodeTool:=CurExprType.Context.Tool;
        case CodeNode.Desc of
          ctnProcedure:
            begin
              Code:=CodeTool.ExtractProcHead(CodeNode,
                  [phpWithVarModifiers,phpWithParameterNames,phpWithDefaultValues,
                   phpWithResultType]);
              // TODO: get param name/type pairs
              ParamList:=CodeTool.GetProcParamList(CodeNode);
              if ParamList  nil then
                begin
                  ChildNode:=ParamList.FirstChild;
                  while ChildNodenil do
                    begin
                      Code += ChildNode.DescAsString+' ';
                      ChildNode:=ChildNode.NextBrother;
                    end;
                end;
            end;
        end;
      end;
    Result := Code;
  end;

Regards,
Ryan Joseph

The other thing I'm stuck on with code tools. In TCodeManagerTool:

    function FindReferences(IdentifierCode: TCodeBuffer;
          X, Y: integer; SearchInCode: TCodeBuffer; SkipComments: boolean;
          var ListOfPCodeXYPosition: TFPList;
          var Cache: TFindIdentifierReferenceCache  // you must free Cache
          ): boolean;

What do I supply for "SearchInCode" so that it searches the entire project (or entire folders)?

Regards,
Ryan Joseph

Any word on this? Is it even possible to find all references in a project or do we need to extend code tools for this to work?

Regards,
Ryan Joseph

It is definitely possible.

Check lazarus sources, directory ide, unit findrenameidentifier.pas:

Around line 302:

    // add packages, projects
     case Options.Scope of
     frProject:
       begin
         OwnerList:=TFPList.Create;
         OwnerList.Add(LazarusIDE.ActiveProject);
       end;

That should give you a start.

Michael.

No, codetools have everything needed. FindReferences is merely one part. You need a combination of several tools. Depending on your needs these may vary.

I added an example for FindReferences:
lazarus/components/codetools/example/findreferences.lpi

Mattias

Ryan,

Do you have a link to the LSP discord channel?  I tried following the one you posted Friday, but I got an error that I either didn't have access to any text channels or there weren't any on the server.

Thanks!

Here's the LSP GitHub page. Scroll to "Getting help" for the link.

https://github.com/sublimelsp/LSP

Regards,
Ryan Joseph

On Apr 28, 2020, at 4:18 AM, Mattias Gaertner via fpc-pascal wrote: I added an example for FindReferences: lazarus/components/codetools/example/findreferences.lpi

I see now, there are more steps involved than I thought.

LSP doesn't have a concept of the "main file" but this seems to require the main program file to start the search from. All we get from LSP is the workspace root directory and I've passed in some -Fu paths via FPCOptions.

Is there a workaround to this or can we get the main program from code tools?

Regards,
Ryan Joseph

I've learned how to use FindBlockCounterPart but it would be preferable to get the ranges which contain the identifiers that are the start/end of the blocks. For example the ranges of "begin" and "end" instead of the entire range which spans from the cursor to the end of the block.

Any suggestions?

Regards,
Ryan Joseph

You can add any number of files as start. Just call AddStartUnit for every starting module.
For example add all Pascal modules of the workspace directory.

Mattias

For clarity: a module is a program or library or lazarus package.

That means you'll need to scan for it, or add some special file.

TMS Software's VS Code studio plugin also requires the presence of 1 .dproj file in the top-level directory. Similar here.

Michael.

In TCodeToolsOptions we provide FPCOptions and ProjectDir so code tools should have a number of directories to search for. At startup does it parse these files automatically or otherwise scan for pascal files? I can rescan myself but I thought code tools may already have this loaded up and ready to us.

Regards,
Ryan Joseph

That's a wrong approach. Pascal units are in a uses-tree. You need a starting point, the top of the tree: a single module.

Michael.

On Apr 28, 2020, at 3:26 PM, Michael Van Canneyt wrote: That's a wrong approach. Pascal units are in a uses-tree. You need a starting point, the top of the tree: a single module.

I can make the user provide the program file and start from there but then the references in other units not related to the main program will not be found.

Mattias says we can call AddStartUnit multiple times but there's no logic I can use besides adding all files in the provided directories.

Here's what I'm trying to implement:

https://microsoft.github.io/language-server-protocol/specifications/specification-3-14/#textDocument_references

It's just a project wide search so what can we do besides just add all files found in the project?

Regards,
Ryan Joseph

Why do you think so ?
The codetools will find all referenced units that it finds in the unit search path.

Mattias says we can call AddStartUnit multiple times but there's no logic I can use besides adding all files in the provided directories.

You don't need all files. You just need a starting point: the project.

You don't need all files, as far as I can see.

"The references request is sent from the client to the server to resolve project-wide references for the symbol denoted by the given text document position."

As far as I can tell this means:

You don't need to do anything except add the project file.

The codetools will use that as the starting point and that's it.

Michael.

Correct. As it should be. Why should it find unrelated units?

Mattias says we can call AddStartUnit multiple times but there's no logic I can use besides adding all files in the provided directories.

If you really don't know the program file, and have no meta information to find it, then blindly parsing every file is the way to go. If it is an include file, parsing will probably fail. If it is an unrelated file needing other compile flag, parsing might fail too. So basically you have to blindly open all these files, use CodetoolsBoss.Explore and see if parsing works. You can check if it is a program and add it via the AddStartUnit.
Or if you really want all unrelated files too, then you can skip the graph and simply feed all found, parseable files to the FindReferences loop.

Since most compilers require a start file, maybe you can look at other LSP, how they determine the start file.

Mattias

On Apr 28, 2020, at 3:45 PM, Michael Van Canneyt wrote: Why do you think so ? The codetools will find all referenced units that it finds in the unit search path.

You're right. I was thinking there are going to be units that aren't referenced by the main program but why do you really want to search those anyways as they aren't actually part of the program.

I'll see how it works to provide a main program file. I guess it can search for lazarus projects to insepect also but I'll leave that for later. My problems are mainly related to not having a notion of a project file like in lazarus that we can read paths from.

Regards,
Ryan Joseph

You can do what most VSCode/Atom project managers seem to, add package.json or somesuch:

{
"pascal-lsp" : {
"projectfile" : "yourproject.pas",
"searchpath" : ["a","b"]
}
}

And use that as a starting point. As a fallback you can scan for lpi/lpr (would be helpful for lazarus users)

Most plugins I saw till now use a kind of helper file in the project dir.

Michael.

Yes, that's what the plugin for Sublime Text does also. Users will have to provide a "initializationOptions" to the language server which contains extra information. It would also be possible to add a lazarus project file and get the options from there but I'll let a lazarus user implement that later.

Here's what I have now from a ST project file (which is plain JSON):

"settings":
{
"LSP":
{
"pascal-language-server":
{
"enabled": true,
"initializationOptions":
{
"FPCOptions": [
"-Fu./sources"
],
"program": "./Main.pas"
}
}
},
....

Regards,
Ryan Joseph

Great !

Incredibly looking forward to test-driving this in Atom... :-)

Michael.

On Apr 28, 2020, at 7:47 PM, Michael Van Canneyt wrote: Incredibly looking forward to test-driving this in Atom... :-)

The biggest left is getting diagnostics working so we can see errors that popup.

Speaking of that is there a way in code tools to get a list of parsing errors that occurred for a given file? It's possible that parsing fails but I don't know where errors occurred until I make a query and comes back with an error. I'd like to get this up front so I can use LSP diagnostic feature that shows inline errors or in the gutter etc...

Regards,
Ryan Joseph

Codetoolboss.Error*

Mattias

On Apr 28, 2020, at 8:52 PM, Mattias Gaertner via fpc-pascal wrote: Codetoolboss.Error*

Not sure I'm understand the usage. When do I check for errors and are there ever more than 1 or just the last error? I'm guessing maybe after:

    URI := ParseURI(textDocument.uri);
    Code := CodeToolBoss.FindFile(URI.Path + URI.Document);
    for Change in contentChanges do
    begin
      Code.Source := TTextDocumentContentChangeEvent(Change).text;
      // .... check for error here? ....
    end;
    // exception handling
    procedure ClearError;
    function HandleException(AnException: Exception): boolean;
    procedure SetError(Id: int64; Code: TCodeBuffer; Line, Column: integer;
                       const TheMessage: string);
    property CatchExceptions: boolean
                                   read FCatchExceptions write FCatchExceptions;
    property WriteExceptions: boolean
                                   read FWriteExceptions write FWriteExceptions;
    property ErrorCode: TCodeBuffer read fErrorCode;
    property ErrorColumn: integer read fErrorColumn;
    property ErrorLine: integer read fErrorLine;
    property ErrorMessage: string read fErrorMsg;
    property ErrorId: int64 read FErrorId;
    property ErrorTopLine: integer read fErrorTopLine;
    property ErrorDbgMsg: string read FErrorDbgMsg;
    property Abortable: boolean read FAbortable write SetAbortable;
    property OnCheckAbort: TOnCodeToolCheckAbort
                                         read FOnCheckAbort write FOnCheckAbort;

Regards,
Ryan Joseph

Only one.
Check when a function returns false.

Yes

The above should be self explanatory.

Mattias

On Apr 28, 2020, at 7:47 PM, Michael Van Canneyt wrote: Incredibly looking forward to test-driving this in Atom... :-)

Michael, do you actually know how to setup a language server in Atom? I've only tested on Sublime Text but I do plan to learn how to setup VSCode also when I find some time.

You should be able to build this and do some testing even if it's not finished and we'd like to know if it works on other editors.

https://github.com/genericptr/pascal-language-server

Regards,
Ryan Joseph

On Apr 28, 2020, at 10:01 PM, Mattias Gaertner via fpc-pascal wrote: Yes

This wasn't doing what I thought. It still doesn't give an error until I query at another point in the unit. I thought it would pre-parse or something but it seems to be on demand. I may have to handle the errors in another way.

    Code := CodeToolBoss.FindFile(URI.Path + URI.Document);
    for Change in contentChanges do
    begin
      Code.Source := TTextDocumentContentChangeEvent(Change).text;
      if CodeToolBoss.ErrorMessage  '' then
        begin
          writeln(stderr, 'Document Parse error: ',
                          CodeToolBoss.ErrorMessage,
                          CodeToolBoss.ErrorLine, ':', CodeToolBoss.ErrorColumn);
          flush(stderr);
        end;
    end;

Where can I know which file it was from? I've got error message from other units than the current one but I don't see a way to get the name of that file.

Regards,
Ryan Joseph

https://github.com/atom/atom-languageclient

and then it should be a matter of creating a descendent of the language client. The C# one seems easy enough to convert :-)

You should be able to build this and do some testing even if it's not finished and we'd like to know if it works on other editors. https://github.com/genericptr/pascal-language-server

I won't have time to test till the weekend, but during the weekend I will test and report. Don't let me keep you from adding features in the meantime, though :-)

Michael.

On Apr 29, 2020, at 3:23 PM, Michael Van Canneyt wrote: and then it should be a matter of creating a descendent of the language client. The C# one seems easy enough to convert :-)

Let me know if you can figure it out.

VSCode looks pretty difficult:

https://github.com/donaldpipowitch/how-to-create-a-language-server-and-vscode-extension

Sublime text was pretty easy (just install the LSP plugin and fill out the JSON settings).

Regards,
Ryan Joseph

Will do.

VSCode looks pretty difficult: https://github.com/donaldpipowitch/how-to-create-a-language-server-and-vscode-extension

I think the client-side is relatively simple ?

Sublime text was pretty easy (just install the LSP plugin and fill out the JSON settings).

Yes, they did the heavy lifting for their editor. A good initiative.

Well...
if the Lazarus team can implement a LSP plugin for the editor, then we'd have instant language support for a whole bunch of languages. That would be awesome...

Michael.

On Apr 29, 2020, at 3:37 PM, Michael Van Canneyt wrote: if the Lazarus team can implement a LSP plugin for the editor, then we'd have instant language support for a whole bunch of languages. That would be awesome...

but language servers don't do any syntax styling so they're only half the solution. I was also disappointed to learn they don't allow arbitrary commands that apply text ranges to the editor. That would mean a unified interface for the countless scripts people write for editors like Sublime Text. Worst is the "Class Completion" feature in CodeTools can't be implemented in editors unless Microsoft extends the protocol.

Regards,
Ryan Joseph

So much for the 'One editor to rule them all' advocates :-)

Michael.

Testing this more I'm having a problem getting overwhelmed by completions for classes because of methods from TObject (and other parents) so I'd like to add some information about this.

Using this method:

    if CodeToolBoss.GatherIdentifiers(Code,X + 1,Y + 1) then
    begin
      Count := CodeToolBoss.IdentifierList.GetFilteredCount;
      for I := 0 to Count - 1 do
      begin
        Identifier := CodeToolBoss.IdentifierList.FilteredItems[I];

can I find the parent class of "Identifier"(TIdentifierListItem)? There's a TCodeTreeNode field that maybe I could walk up to find what I need?

Regards,
Ryan Joseph

I found the "Explore" method which appears to do what I wanted. There's also CheckSyntax which appears to do basically the same thing.

Regards,
Ryan Joseph

Stuck on another code tools problem.

I've been using code like this to find words at node offsets and this works well until there are $ifdefs in the file and then the text position getting offset and fail to return the correct identifiers.

GetIdentifier(@Code.Source[Node.StartPos]);

How should this be resolved? I don't see there is a copy of the actual text parsed (without ifdefs) so I don't know how to reference the text that the parser actually used.

Regards,
Ryan Joseph

The Node.StartPos are referring to the Tool.Scanner.CleanedSrc, which is the code stripped of skipped $ifdef ranges and include files included.

How should this be resolved? I don't see there is a copy of the actual text parsed (without ifdefs) so I don't know how to reference the text that the parser actually used.

You can convert the cleanpos to actual file position using

Tool.CleanPosToCodePos

Mattias

On May 1, 2020, at 2:42 PM, Mattias Gaertner via fpc-pascal wrote: The Node.StartPos are referring to the Tool.Scanner.CleanedSrc, which is the code stripped of skipped $ifdef ranges and include files included.

That's what I was missing. Thanks again.

Regards,
Ryan Joseph

Ok, got things working now. Here's the basic pattern I'm using. I call explore and then walk the root tree node to find what I need but i've noticed the 2nd time I do this I get less results than the first time! What is happening? I assume it's caching something but do I need to reference the cache myself?

  Code := CodeToolBoss.LoadFile(Path,false,false);
  if not CodeToolBoss.Explore(Code,Tool,false,false) then
    exit;
  WalkTree(Tool.Tree.Root);

Regards,
Ryan Joseph

This thread is getting too long.

Can you please create new threads for new topics?

Ok, got things working now. Here's the basic pattern I'm using. I call explore and then walk the root tree node to find what I need but i've noticed the 2nd time I do this I get less results than the first time! What is happening?

What are you doing between calls of Explore?

Mattias

Testing this more I'm having a problem getting overwhelmed by completions for classes because of methods from TObject (and other parents)

The RTL is powerful.
That's why I prefer to put recently used identifiers at top of the completion box.

Sure.
Something like:
while Nodenil do begin
  if (Node.Desc=ctnClass) then begin
    if Tool.ExtractClassName(Node),false)='TObject' then ...
    break;
  end else
    Node:=Node.Parent;
end;

Mattias

Sure. I'll respond now on this last question but I'll make new ones later if that's better for the list.

I'm walking the node tree and then using the Start/EndPos to read into the source text so that I can extract identifier. It's just: LoadFile, Explore, WalkTree. I didn't step through the debugger or anything but I assume the actual node structure starting from Tool.Tree.Root changed.

If that's not normal behavior I'll look harder and get into the debugger.

Regards,
Ryan Joseph

"Explore" checks if some files have changed and if so rebuilds the nodes.

If that's not normal behavior I'll look harder and get into the debugger.

Mattias

On May 1, 2020, at 9:28 PM, Mattias Gaertner via fpc-pascal wrote: "Explore" checks if some files have changed and if so rebuilds the nodes.

so I missed something fundamental! I used Explore because I noticed that I was able to get parser errors after calling it. When does the actual node tree get built then? I just want to load the file and step through the nodes like I explained earlier.

Regards,
Ryan Joseph

By many functions in CodetoolBoss. Explore, FindDeclaration, FindReferences, GatherIdentifiers. All of them checks if a file have changed and rebuilds nodes. If no file/setting has changed, nodes are kept.

I just want to load the file and step through the nodes like I explained earlier.

Maybe I misunderstand what "like I explained earlier" means. The thread was pretty long.

Mattias

Found the problem. I overwrote "FirstChild" of TCodeTreeNode using a var param. oops! I still thought LoadFile should have rebuilt the node tree but it doesn't seem to do this (even if UpdateFromDisk and Revert params are set to true).

I'll post any new code tools questions to a new thread now and keep this one for the language server.

Regards,
Ryan Joseph

LoadFile merely loads the file.

I'll post any new code tools questions to a new thread now and keep this one for the language server.

Thank You.

Mattias