Hi,

[My first post here; please be forgiving.]

Quite sometime ago I worked with Anthony (wasn't much, mainly testing and code suggestions etc.) for a while (2002-2003) on JEDI Code Format (JCF).

At the time, especially when compared to DelForEx, JCF was noticeably slow --basically because of internal structure design choices.

A rewrite (or, rather, a major overhaul) was needed, but now that I took a source, it seems Anthony never got round to it.

Don't get me wrong; I do understand his reasons: I, for one, didn't have a fraction of his patience or stamina.

But, seeing the above threads about 'case higlighting' and 'using regexs for syntax higlighting', I remebered my main complaints about JCF --that not only was it slower than necessary, much of it was also a duplication of an already acomplished job: Namely, duplicating what FPC already does --parsing and tokenizing.

Given that I have some time to spare now, I would like to tackle using FPC's own parser/tokenizer engine for a code formatter.

Trouble is, I am not familiar enough with the internals of FPC source tree; meaning I don't even know where to begin looking for the relevant files in the FPC tree.

Could someone point me in the right direction please.

I'd also like to hear opinions whether I should spend time on this, or even whether it is a good idea to use FPC's parser/tokenizer engine for code formatter.

Hi!

I don't think that it would be a wise idea to use the compiler's parser/tokenizer for this. It is too much trimed for its main purpose: compiling.
But you might try the parser in packages/fcl-passrc which is used inside the fpdoc tool. It also has it's own (small) wiki page here: http://wiki.freepascal.org/fcl-passrc
As far as I understood it, it was extended to also parse implementation sections (for documentation only interface sections are needed). It still has some bugs (see BugTracker), but it seems to work rather well.

Regards,
Sven

IMHO: No.
The FPC parser/tokenizer is not very forgiving about errors in the code. That means that as soon as an error is encountered, the parser stops parsing. That in turn means that everything below the error will not be formatted.

Michael.

Yes, it can already parse all fcl sources.

Adem,
What I miss in all code formatters is formatting fragments. For example when the codetools adds missing procedures or when a code template is inserted or when some fragment is pasted from clipboard the IDE should format this fragment with the user settings. For performance reasons the formatter should not rescan the whole unit every time a fragment is inserted.

Mattias

Yes, and this is frustrating - current JCF (integrated in Lazarus) does not understand few things (generics specialization for example) and it fails to format everything.

Regards
?ilvinas

2010/5/28 ?ilvinas Ledas :

Yes, and this is frustrating - current JCF (integrated in Lazarus) does not

And the current JCF (integrated in Lazarus) doesn't format the code exactly the same as the original JCF even with the exact same settings file. :-(

This is also what I would like to see.

Indeed, I get confused when I see so many options for code formatters without visual feedback.

What I really would like to see is, when I do change an option, what the code will be like.

IOW, code formatter should be able to act upon code snippets too.

I am aware of that.

But, I am hoping to modify the FPC parser/tokenizer so that instead of it deciding whether to stop, it should do a callback and let the caller decide whether it should stop.

Is this likely to be unworthy of the effort?

[Sent this as private email, by mistake. Sorry, Michael. And, I hate Thunderbird :) ]

No, but it is hard to do in a meaningful way.

Michael.

Could you (or someone else) point me to the source files of FPC parser/tokenizer please.

I can guess that it won't be easy; but if it can be done, I think it should be.

It is the only way the code formatter can keep up with new features of FPC.

Most units that are related to parsing are inside "compiler" directory and prefixed with a "p", e.g. pdecl.pas and pdecsub.pas (but you still need to find the relations between those units yourself - I myself only find these "on demand" ^^)

I don't think that it will be that easy. You might need to do so many changes that you can't simply copy new features from the compiler to your formatter.

You might be better of playing around and modifying the parser in fcl-passrc - at least that one was developed for simply parsing Pascal code. You might be able to extend it with better error recovery and fragment parsing. (Before you start yet another Pascal parser, because a "forked" FPC parser might become to different to be recognized as FPC's parser any more)

Regards,
Sven

fcl/packages/fcl-passrc.

Note that this is NOT the scanner/parser used by the compiler itself, this one is used for the documentation generation and other tasks.

Using the parser/scanner from the compiler is an impossible task, they are impossible to separate.

Michael.

Thank you.

Hmmm..

Not really what I wanted to hear ;)

Is there a bounty sort of mechanism in this community?

Maybe someone would step in to modularize the scanner and the parser functionalities (maybe the compiler too).

That sort of thing would not only make code formatting a breeze, it would also help all sorts of things --code analysis, refactoring etc.

I'd be glat to put up a bounty.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100528/7da5272b/attachment.html>

It might be possible but the fpc scanner/parser are rather interwinded with other parts of the compiler, just to name an example: pmodules.pas parses units and programs but controls also code generation and handles symtables etc. for them.

It is the only way the code formatter can keep up with new features of FPC.

FPC isn't moving that fast nowadays regarding syntax so keeping up shouldn't be that hard.

> It is the only way the code formatter can keep up with new features of FPC. FPC isn't moving that fast nowadays regarding syntax so keeping up shouldn't be that hard.

Maybe not in the last month, but it definitely got in the last year a lot of new keywords and syntax. For example:
foreach, generics were extended, objcclass, objprotocol, objccategory, nested classes, some new modifiers were added

Mattias

Florian,

I can only --barely at that-- guess the amount of work and effort you have put into FPC and I appreciate it all immensely. [No, this isn't grovelling. It is sincere. But, I just wanted to say it, before moving on to the next paragraph.]

Would I have your blessing if I proposed a bounty to unentwine them so that each one of those major modules becomes objects in tehir own right --commnicating with one another through public/published events and properties.

I can guess that it would be hard to do, but I feel it will be absolutely impossible without your (and the rest of compiler team's) blessing.

Adem wrote

Is there a bounty sort of mechanism in this community?

http://wiki.lazarus.freepascal.org/Bounties

Vincent

Adem wrote

I can guess that it would be hard to do, but I feel it will be absolutely impossible without your (and the rest of compiler team's) blessing.

All I can say is: Good Luck!

I found a Oil Tanker more responsive and flexible than some of the developers here. Maybe your approach of flashing money around might win you some votes.

I found a Oil Tanker more responsive and flexible than some of the developers here.

I know, most git users prefer the development model "fix one, break one".

Graeme Geldenhuys wrote

I found a Oil Tanker more responsive and flexible than some of the developers here. Maybe your approach of flashing money around might win you some votes.

I've found an oil tanker, once, but it wasn't flexible.

I can see another benefit of separating parser/tokenizer from code generation. Is easier and better support of newer targets. I'm talking about long wished LLVM, and even possibly Java and/or .NET. (No need to support all possible features, but generate their native byte-code).
So far, the internal structure of the compiler is the **only** obstacle (caused by lack of manpower, actually)

thanks,
dmitry

Would I have your blessing if I proposed a bounty to unentwine them so that each one of those major modules becomes objects in tehir own right --commnicating with one another through public/published events and properties.

15 years ago I had similiar dreams. Within the last 15 years I learned that a compiler is something different: one does a clear design and then a border case pops up (followed by more) which kills this design. I wrote a lot of other software during this time but nothing is comparable in this regard with a compiler. I'am pretty sure not only FPC has this problem, gcc has it as well (just look at the sources) and considering the trouble Borland, CG etc. have to create a 64 bit delphi compiler hints that they might have the same problem. So I simply don't think that this can be achieved in a reasonable time. IMO the effort to do so should be spent in existing pascal parsers.

Separating code generation is easy. The bigger problem is the symtable information: fpc uses symbol information while parsing and generating the parse tree for better error checking and for simpler and faster parsing. Further, the compiler is a really complex piece of software so we don't want to be hindered by exposing functionally too much, this would make the oil tanker a continent.

BTW: We had this discussion ten years ago already when fpdoc was made and it was decided to write a separate parser after failed attempts to use fpc's frontend.

True, but the years before we were really slow ;)

Graeme,

With a comment like that, I am afraid you're actually forcing a negative response.

No, I wasn't flashing around money.

I have absolutely no financial gain whatsoever from whether or not FPC/Lazarus improves or gets more popular --that is, I haven't written a single line of code I charged for (not even indirectly); whatever I wrote I did so simply to scratch my own itch.

I am (have been) a Delphi user since time immemoriam; but I have also been lurking in the background in the devel list for a long time. I have seen how overworked the compiler team is.

So, instead of coming up as another "give me this or that" demander (to coin a new word), I thought it would be better to get someone else to refactor the already written code to be more usable in downstream projects.

This is basically why I proposed what I proposed.

So, your comment that I am flashing around money is not just unfair (to me and everyone else), but it hurts too.

I wish you would make more effort to contain any bitterness you may have and let other contrinute what they can.

I think I know what you mean.

But, there may be --just may be, I am hoping-- people (more like, individuals) in this community that can undertake such a task and not feel like doing it all over again.

Of course, this has been so. But, the upside of it all is that, much/most of all those corner cases have been dealt with by now. So, much of what needs to be done is (should be) to take the existing code and refactor it to enable some downstream projects that (even the need for them) did not exist those years ago.

What I had in mind was not to ask you (or the compiler team) to do this. That would not only be unfair to ask (on top of current burden) but would also be (most probably) totally boring/unexciting. But, both your blessing(s) and supervision are essential; otherwise it is a no go from the start.

I had the same question some time ago, and was told that the FPC parser is not suited for anything but compilation purposes. And to be honest, I couldn't even locate that parser code in the FPC sources :-(

I'd also like to hear opinions whether I should spend time on this, or even whether it is a good idea to use FPC's parser/tokenizer engine for code formatter.

I'm interested in essentially the same subject, since I tackled with PasDoc and its limited built-in parser. Another goal were a Modula/Oberon frontend for FPC, with improved syntax. Drop me a line if you think that we can cooperate on this subject.

DoDi

But you might try the parser in packages/fcl-passrc which is used inside the fpdoc tool. It also has it's own (small) wiki page here: http://wiki.freepascal.org/fcl-passrc

Is it a bug in the source or in the wiki engine, that the first essential links are for "action=edit"?

Can somebody fix that?

DoDi

It's not only a matter of making the parser proceed, it's also a matter of *how* to proceed, i.e. to find some point in the source code where parsing can continue. That code (to find an synchronization point) is not part of the existing parser. Some research has to be done whether it's possible to e.g. have the FPC parser as a base class, from which a fault-tolerant parser class can be derived.

DoDi

Using the parser/scanner from the compiler is an impossible task, they are impossible to separate.

There's nothing impossible in software ;-)

It's a matter of refactoring, and whether the refactored code will be accepted by the FPC team. First expected rejection argument: it slows down compilation...

A weak argument so far, perhaps we can make the parser even faster?

DoDi

Maybe someone would step in to modularize the scanner and the parser functionalities (maybe the compiler too).

It looks feasable to me, starting with fpc/compiler/parser.pas.

DoDi

I had similar problems with my general decompiler, in writing front-ends for various binary and textual formats. But with a restricted goal, covering Pascal-style languages in the first step, I think that a separation is feasable. Even a C frontend should be feasable, as I've demonstrated in ToPas, and Borland in CBuilder/Kylix; this would be interesting with regards to iPhone etc., where Apple requires applications written in (Objective) C.

A 64 bit compiler deserves a different back-end, and eventually appropriate intermediate objects (for storing 64 bit pointers). This is what FPC has since a long time, while the Delphi compiler was targetted at x86 for Win32 only.

So I simply don't think that this can be achieved in a reasonable time. IMO the effort to do so should be spent in existing pascal parsers.

Just the number of existing Pascal parsers suggests to me that time is spent *better* in a front-end unification, than in maintaining and synchronizing any number of independent parsers.

DoDi

BTW: We had this discussion ten years ago already when fpdoc was made and it was decided to write a separate parser after failed attempts to use fpc's frontend.

Just a question: AFAIR at that time the parser was an automaton, created by some lex/yacc port. At that time I also gave up to make the parser usable for documentation or other purposes. But when this is no more true, the discussion could be resurrected?

DoDi

I am no expert on either the parser/tokenizer or the compiler, so I am in no position to guesstimate what might happen in that area.

What I do have --as a gut feeling-- is that, even if the compilation did slow down a bit, the benefits from modularized/componentized back-end is very much likely to compensate for it in the agregate.

By 'agregate', I mean the whole process of development (or, 'experience' if you'd rather).

After all, we spend more than a fraction of time compiling a project compared to the time spent on writing and modifying it.

And, it is this 'development' process that I am expecting the biggest payoff from.

I am sure we all can see the benefits.

Trouble is, I cannot do it.

All I can do is to put a bounty on it to help it materialize; but without a go ahead from the compiler team, there's no point.

No. yacc was never used seriously, the usage of flex was removed mid 90's.

That is why FPC provides one in fpc-passrc: so other people will not try to re-invent the wheel.

It is currently used in 3 projects
- fpdoc
- WST to generate interface descriptions
- A project I am working on to convert pascal code to JavaScript. (see also the fcl-js sources)

This is also the reason I contracted Matthias to extend fcl-passrc.

All in all, I think it would be to everyones advantage if we could concentrate on fcl-passrc. It is by far the most compatible engine out there, and is sure to remain maintained (for fpdoc) and extended whenever the compiler is extended.

I think this is a far more realistic scheme than trying to extract the scanner/parser from the compiler.

Michael.

Yes, but within a reasonable time and without other penalties :)

It's a matter of refactoring, and whether the refactored code will be accepted by the FPC team.

If it's good I'am sure it will be accepted, but I expect this to be a work of years.

First expected rejection argument: it slows down compilation... A weak argument so far, perhaps we can make the parser even faster?

I doubt this, just one example: ptconst.pas: it parses typed constants. Due being tailored for the compiler, it reads the constants and writes them without any second pass to the output assembler list. A generic parser needs to create first some kind of parse tree and this parse tree needs to be outputted in a second pass. One does so and we release a compiler with this and then the people pop up, crying that their pascal files with millions of lines of typed constants don't compile anymore or compiles for ages because memory requirements of the compiler heavily increased.

Maybe it's just me, but I agree with Michael here. I think effort should be spent on improving the fcl-passrc parser instead. After all, it is included with FPC, so should take priority than other external (3rdParty) parsers. More effort should be placed on keeping it up to date with what FPC itself supports. This should also become the general purpose parser that could be used for all other projects build with FPC.

Hans-Peter Diettrich wrote

for documentation or other purposes. But when this is no more true, the discussion could be resurrected?

So what are the main reasons nobody wants to use the fcl-passrc parser? Except the obviously lack of some language syntax - which I think should be easier and quicker to add, than to try and modularize the compiler's parser.

I would also imagine the fcl-passrc parser should be easier to maintain that something that has deep ties in the compiler workings itself.

Graeme Geldenhuys wrote

So what are the main reasons nobody wants to use the fcl-passrc parser?

Lack of the expression parser?

thanks,
dmitry

Adem,

fpc-passrc was recommended by many. Is there some reason you don't want to use it? It can parse all fcl sources which proves it is good. Or, maybe I have missed something here...

There is one more parser around, in codetools. It is scanning and parsing code in the background when you use Lazarus. It is optimized for recovering from invalid code (which is being edited) and would suit perfectly for a code formatting as well. That is, if you want to integrate the formatting with Lazarus. It was not mentioned in your mail.

The problem is that codetools is a big and complicated piece of code. It seems only Mattias G?rtner really understands its internals. Still, I guess there must have been an idea to integrate code formatting with codetools. How is it, Mattias?
It would be logical because the parsing is done already and formatting would be quick. When you start an external JCF, it has to parse the code again.

So, you have at least 3 choices:

1. fpc-passrc, a parser "officially" supported by FPC project. Relatively easy to use I guess (although I haven't used it myself).

2. Codetools integration. More challenging but would be very cool in Lazarus. Besides, if you learned codetools internal code, it would make already 2 people who know it. :-)

3. Extract the parser somehow from FPC. IMO, it doesn't make sense. I don't know the compiler code much but I know it is a big interdependent chunk of code. Besides, your task is not really related to a compiler but to visual appearance, which is more in a domain of Lazarus.

Regards,
Juha

This is what virtual methods are good for.

DoDi

Hmm, actually getting the existing codebase compiled for those targets without requiring extensive changes seems a bigger challenge.

Florian already corrected this, but I think you are confused with h2pas which uses some lex/yacc port. (tply)

ptop has an own parser too btw. So that makes at least 4 or 5 parsers in the combined FPC/lazarus projects

- the compiler's
- fpdoc/fcl-passrc
- ptop
- lazarus codetools
- lazarus syntaxhighlight (different from codetools?)
- Jedi CodeFormat.

Because of the notion the compiler one is updated "for free".

But that is as always the attraction of oversimplification. If the compiler changed, it would probably break the tools depending on it too, and with current fcl-passrc/h2pas etc bugfixing speed, that would mean they would probably be out of order more often than not :-)

Personally I think the bounty is better spent on improving fcl-passrc and giving it some regular work and maintainance.

That doesn't mean the various parsers could be better synchronized, and I think something can be done there. But I think it would yield more to integrating fcl-passrc into the testsuite, and thus document the boundery conditions of implemented constructs.

Trolling again?

Joost.

Well, given the recent breakage because of the support for 'deprecated', I started on a test file, but I think that a better and more fundamental approach is needed. My idea was to create a large number of small files with code fragments. The test application reads the fragments and puts them in a unit (interface/implementation) and attempts to scan/parse it. After that, it's just a matter of creating the code fragments.

Michael.

Other than some historical accident, what can be the point of writing/maintaining various parsers while there is the true blood parser right inside the compiler engine.

I mean, if I'd suggested a slightly different version of TList, I am sure I'd be given thr tar and feathera treatment for attempting to duplicate an already existing class for no major gain. Yet, there seems to be more than one (each half-baked to various degrees) parsers lying around.

Code formatting isn't main and only place to use for the refactored FPC's parser, there are many more downstream areas it can be used for. I could go on about what sort of horizons it would open up downstream, but I am sure you all know about them better than I do.

Anyway, so far, all I have learned about FPC's tokenizer/parser/compiler is that it is near impossible for ordinary mortals to factorize them, plus it would take another ten years or so to stabilize.

Good job I wasn't the one suggesting FPC to be 64-bit compatible; judging by how long it is taking EmBorCoDero, I might have been given centuries :)

The point is that two separate parsers are probably easier and cheaper to maintain than one integrated one that has to forfill all _different_ requirements for the individual parsers.

Well, the problem is that anything can be made to look easier by layering one layer simplifications over the next.

If everything was as simple as the identifying the base principle, we would have fusion-power for about a century now. Unfortunately, often the implemnetation details make something non trivial.

A nice idea, but what about verification of the result? Not getting exceptions is not necessarily a sign of a properly working parser, with decent error recovery.

DoDi

This is even not guranteed for the compiler parser ;) I discovered recently that fpc eats the following nonsense:

label
  1234;
begin
  4321+1234:
end.

Different discussion. That is for new tests specific for fcl-passrc. I'm talking about stuffing the tests that we already have through it.

which has several parsers, depending on usage. directives, nodes, indentation and fragments.

- lazarus syntaxhighlight (different from codetools?)

Yes, the synedit highlighter TSynPasSyn parses pascal.

- Jedi CodeFormat.

Mattias

The expression parser of the codetools is quite independent (expreval.pas). It should be easy to adapt that code for the fcl parser.

Mattias

That is step 2. Step 1 is proper parsing. Can't do step 2 if you don't have step 1. You can't run before you can walk...

Michael.

Ah. I misunderstood that.

But still, the test-app I have in mind could be used for that: just a matter of configuring.

Michael.

Good to know, thank you :)

Michael.

Yes, one can always hack around such things but in general I'am very pessimistic that using the fpc parser is possible without too much drawbacks on the compiler and compiler development.

Considering the benefits, I still would like it done.

If only someone would come forward to name a price on it.

Adem-

I have great interest in a parser-based code formatting tool. However, I have doubt that using the compiler's parser is the best solution. So, I'd like to discuss some of the issues surrounding this decision.

I'd like to get your ideas on the following problem. There are basically two philosophies of code reformatting. These I'll call "reorganization" and "rebuild".

Using the reorganization approach, the source file is largely retained intact with some lines modified to enforce some formatting rules. The idea is to keep everything as it is except those things that have specifically been requested to be examined and modified. Minimal change is called for. Parts of the source file that the parser does not understand are left intact.

Using the rebuild approach, the source file is examined by the reformatter, digested for content, and completely rewritten by the reformatter according to its rules. Everything is re-written according to the formatting rules so nothing can be included that the reformatter does not understand. For example, any manually formatted details are lost. The reformatter must know about and understand how to retain all valuable content. The developer cannot add anything important that the reformatter does not know how to handle.

Which approach do you think should be used?

I prefer the reorganization approach because I view source code as carefully sculpted art. Others treat source code as essentially a machine-generated intermediate representation of some abstract concepts that can be transformed over and over again as needed. These people would be more comfortable with the rebuild approach.

I'm just curious how you look at the problem because it has a big impact on what parser is used. Of course, I'm also interested in everyone else's viewpoint as well.

On 2010-05-31 05:36, Doug Chamberlin wrote:

Doug,

I have great interest in a parser-based code formatting tool.

However, I have doubt that using the compiler's parser is the best solution.

It may not be the best solution basically bexause it was written solely for the compiler's needs. IOW, a very customized, special purpose one.

Yet, as far as parser goes, it is the best solution. Because it is the originial source of information. Any new additions as well as corrections go in there --if it doesn't work, nothing does. IOW, it is also the main specification.

At the moment, as I gather, either because the need has never really arisen to or because of the workload, that code (which is also the best specification) has not been modularized enough to be usable for other purposes. [by 'other purposes' I don't mean code formatting alone, there are others and more will doubtless come up in the future.]

So, I'd like to discuss some of the issues surrounding this decision. I'd like to get your ideas on the following problem. There are

basically two philosophies of code reformatting. These I'll call "reorganization" and "rebuild".

Using the reorganization approach, the source file is largely

retained intact with some lines modified to enforce some formatting rules. The idea is to keep everything as it is except those things that have specifically been requested to be examined and modified. Minimal change is called for. Parts of the source file that the parser does not understand are left intact.

Using the rebuild approach, the source file is examined by the

reformatter, digested for content, and completely rewritten by the reformatter according to its rules. Everything is re-written according to the formatting rules so nothing can be included that the reformatter does not understand. For example, any manually formatted details are lost. The reformatter must know about and understand how to retain all valuable content. The developer cannot add anything important that the reformatter does not know how to handle.

Which approach do you think should be used? I prefer the reorganization approach because I view source code as

carefully sculpted art. Others treat source code as essentially a machine-generated intermediate representation of some abstract concepts that can be transformed over and over again as needed. These people would be more comfortable with the rebuild approach.

I'm just curious how you look at the problem because it has a big

impact on what parser is used. Of course, I'm also interested in everyone else's viewpoint as well.

These, the following, are simply my own/personal opinions. IOW, I am not out to press them as precondition.

As far as I can see, JCF itself does (or, rather, can do) some of the latter example you gave. IOW, IIRC, it can add code (if opted for by the user) such as 'begin-end' stuff for 'if' blocks.

if something then something else somethingelse;

becomes

if something then begin
   something;
end else begin
   somethingelse;
end;

it can also reorganize some comments in such constructs.

While I do remember talking about 'un-with'ing (or, 'de-with'ing), I am not sure if Anthony did get around implementing it in JCF. This (admittedly an extreme one, one which I haven't even checked to see if it compiles) is the kind of stuff I am talking about:

var
   Box1: TRect;
   Box2: TRect;
begin
   With Box1, Box2 do begin
      Left := Left;
      Top := Top;
      Right := MyRight;
      Bottom := Bottom;
   end;
end;

I'd like something to take care of this sort of stuff; but I am not sure it has to be the code formatter --a refactoring tool sounds like a better one for the job definition.

If so, it brings us to the 'refactoring tool'; which --unless I am mistaken-- there isn't one for FPC/Lazarus/others. If such a thing doesn't exist, I can only guess it is simply because there isn't the infrastructure suitably ready for it --how many of us would undertake writing a fully-fledged parser engine before writing a refactoring tool (which is hard enough) for non-commercial puposes? Same goes for code analysis tools etc.

These are the reasons why I am pinning so much hope on modularizing tokenizer/parser/compiler trio. Once that done, someone else (singular or plural) --I am hoping-- will step in to write those tools which in return help developing all sorts of other tools better and faster.

But I digress.

Short answer to your question is, I don't have a condition carved in stone.

Zitat von Adem :

The infrastructure exists. I just didn't know that there is demand for this.

Mattias

Adem, did you see my other mail earlier on this thread? I suggested using codetools parser for the task and integrate the formatter in Lazarus.

I wrote:
There is one more parser around, in codetools. It is scanning and parsing code in the background when you use Lazarus. It is optimized for recovering from invalid code (which is being edited) and would suit perfectly for a code formatting as well. That is, if you want to integrate the formatting with Lazarus. It was not mentioned in your mail.

Codetools is a suitable infrastructure. It already implements some refactorings.
Learning it and integrating your formatter would be a challenging task but still more realistic and more sane than extracting the compiler's parser. And it would be really cool to have such feature integrated in Lazarus.

What do you think?

Regards,
Juha

This is not necessarily true. Someone might hack in support for a new procedure specifier. However, it is not guranteed that this one would be exposed in any way to an external interface.

The compiler simply discards comments and it has no infrastructure to link comments in any way to code.

But, the parser does produce a parse tree, doesn't it.

And, it emits (or, could emit) events telling the calling module what went wrong and where (and get feedback whether to stop or continue).

If it does (can do) these basic things, it is good enough for non-compiler use. Is it not?

Naturally, comments are no good for the compiler --so, I am assuming that the current parser does recognise comments but does not add them to the parse tree; if so, it is fine, all it will have to do is to add those nodes while the compiler still ignores them. Would that be too difficult to add?

FPC doesn't produce a parse tree during compilation. The parser puts declarations/definitions immediatly into the symtables, for code an intermediate representation is created which is indeed a tree but no more a parse tree because statements might already be transformed.

No idea, at least I expect a lot of pitfalls.

Zitat von Adem :

I think the biggest problems are the directives. The compiler parser simply skips ELSE code.
A code formatter should not ignore that parts.
How do you want to format an include file for another target? How to handle macros?

Mattias

So, if I understand this right, the code is parsed but instead of producing a tree, a more directly usable structure is created.

Now, the question is, can it be made to produce (in parallel with what it already does) a tree to be used elsewhere?

Pitfalls.. definitely --and probably more than what anyone can guess now.

That's why I'd like someone with the correct skill set to handle it.

At this moment, all I can utter will be theoretical stuff. Like, producing alternative branches for directives and formatting them. I have no idea to offer for macros though.

But, knowing that you have spent far^n more time on those issues than I am, I have a feeling (or, hope) you will have much more practicable suggestions (than mine) for those sorts of problems.

How would *you* deal with macros?

Of course this could be done but we are now at the point were the discussion started: if somebody adds new features and doesn't update this alternate path/code to produce this tree, the situation is the same as currently.

True. In order to avoid this sort of thing all together, the best solution would be making the compiler use a parse tree.

But, failing that, adding/altering the needed code in the parser still looks like a lot easier (less effort) than doing it with a completely separate parser.

Florian already explained at the start of the discussion that this is not feasable because of speed and memory penalties.

But, failing that, adding/altering the needed code in the parser still looks like a lot easier (less effort) than doing it with a completely separate parser.

Strange, I had come to the opposite conclusion based on all the replies you got.

Michael.

It probably won't be possible because (as noted by Mattias) compiler doesn't actually parse false-branch of $IFDEFs, and this cut-off is done before syntax check during source code preprocessing.

Well... "someone with the correct skill" already made suggestion to better invest time/energy/money in improving another parser (one from fcl, codetools, ...), and leave compiler's one intact. That is the best approach, I'd say.

Regards,
Aleksa

There are not much options here. It is text substitution based, so if you want to transform generic _source_ containing macros, you are pretty much limited to ascii transformations.

However FPC's macro use is extremely low, and usually it is only used to define a conditional calling convention. If you limit it to that, you only need a "custom" directive token, and some scanner hackery.

I am not a compiler author, so I do not wish to be arguing with experts; but, it's not as if we have both alternatives to measure and chose the best on the aggregate.

I can see that it is expected to be slower and will need more memory; but, we don't know is how much slower and how much more memory it will need. And, whether these will be noticeable/bearable considering what we gey out of it.

From the replies I got, all I can conclude is that it will (would) be a very hard task to do. That does not necessarily mean that --once it gets done-- adding changes or new stuff will also be that hard to that particular code path.

Zitat von Adem :

[...] How would *you* deal with macros?

I would not use a complete parser.
I would use a forgiving parser like the synedit highlighter or the indentation parser of the codetools.
It depends on the features your code formatter should support.

Mattias

To that we can reply now: No, it will not be bearable.

We get enough complaints about speed/memory use as it is.

Any additional slowdown is therefor simply not an option.

Michael.

Are we mixing what the compiler does with what the parser does here?

Personally, I am not bothered with what the compiler does internally; as long as the parser produces an alternative branch (that the compiler can choose to ignore), I am OK with it.

The question is, can the parser be made to do that?

Spending time/energy/money on improving another parser?

Yes, it does sound like a sensible thing --for now.

But, let's project a few years down the line, to a time when this other parser has reached a 'perfect' stage.

You'll have competing products one pulling one way, the other pulling another. You'll also have two main parties one favoring one over the other.

By then, both of those projects will have gained sufficient inertia to make it impossible to alter and/or combine?

It's like "if you don't like the plumbing in this kitchen here, go build your own house with a kitchen of your desires"..

Does that always have to be like that?

:)

You will always get complaints one way or another all the time, you know it as well as I do, even though/if your product reaches the speed of light. This is a fact of life --one which is usually responded to with "get a better hardware".

[See all those OSes that are getting more and more memory hungry and slower according to some, yet people seem to want even more features that add to all those 'bad aspects'.]

I too find it sensible/easier to tell people not to do anything --especially it may turn out to involve me mentally or physically-- different. So, I can understand that. But, then I usually tell myself to cool down a little (and be less fanatical so to speak) at least when discussing the idea.

Of course, I cannot expect everyone to do the same, but it would be nice if some could lend a little more benefit of doubt than others and say something like "go ahead and do it, we'll decide if it's good enough --then, not now"..

Hi lazarus list,
At the beginning of this discussion I also wondered if one should make a "general purpuse" parser out of the compiler's one, but now I think one should not.
Compiling and code formatting are different tasks. Both need to parse the code but if you want an optimized solution then the parsing has different characteristics in the two tasks.
These are better to be done by different parsers, the one thing that should be common is a clear specification of the language syntax (now it is called reference guide), highlighting the differences between versions. If we have a good (well designed) general purpuse parser then in case a new version is released it requires minimal effort to adopt new syntax (the sources may not even need to be changed just the used language description files) with the help of the specification. This parser can be used to different purposes (code formatting, translating, documentation, UML round-trip) and can have many features the compiler's parser doesn't need to have. I don't see this as a bad thing.
I think the most important points are:
- have 1 such ('canonized') parser, not many different projects, and make projects in need of a pascal parser use this one
- it must be well designed to be easy to keep up-to-date (follow changes in syntax)
- be well documented to be easy to maintain
(the last two are just trivial software engineering principles)

I don't think we should base this parser on the compiler's one, as some have already pointed out, there are other parsers out there more close to a general purpose one. Using one of those (fcl-passrc or codetools' parser) and developing it to meet the above requirements seem to me the most clear solution that also requires the least effort.

"You'll have competing products one pulling one way, the other pulling another."
I'm not sure I understand you correctly. If you were talking about the two parsers then this makes no sense to me as we are going to have two parsers both sticking to the same "official" free pascal specification, one optimized for high speed compilation the other for general purpose use. These are not competing products and even less pulling each other different ways.

regards
Kriszti?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100531/57cd3ca3/attachment.html>

If we had an comprehensive official specification or any chance of getting one in near future, I'd definitely agree with you.

But, there isn't one.

And, AFAIK, there has never been one (at least in the last 20 years) that has been adhered to in every detail. Each implementation has been a subset with more or less extensions. Source code has as usual been the only complete definition.

Could we use the parser(s) to produce specifications too, I wonder.

Why is this emphasis on the parser being '/forgiving/'?

Why wouldn't I want to know about the ugly truth that the code I just wrote will not compile?

Why should I have to wait till I compile the rubbish I produced to find out that it is indeed worthless?

Well, I don't have any more reasons to present.
Just tell me when you'll have finally decided about which parser to use as a base and if it is not the compiler's one (based on this discussion it seems too hard for me) I hereby offer my help in developing it. I can help in both design/coding or documenting. (after getting familiar with the code, of course) I'm a university student, I'll have some time in the summer. (though I don't promise this will be my first priority)
In case no constructive result comes out of this debate, obviously I can help with other parts of lazarus as well (if needed). :)

2010/5/31 Adem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100531/57c04df5/attachment.html>

Zitat von Adem :

Why should I fix all required source files to format one selection? IMHO a code formatter should not be as picky as a compiler and allow to format code even if it does not compile (e.g. because it is code for another target OS).

And I guess, even if the formatter uses the compiler parser, it will still lack behind. Every time the compiler parser is extended the formatter needs to be updated. I doubt that the compiler crew will maintain the formatter, so the formatter will be broken often (and probably often without noticing).
Using the fcl parser will at least give you helpful entries in the svn log solely about the parser.

My conclusion:
Using the compiler parser will limit the code formatter to compilable sources, increases maintenance cost, bites the goal of a fast compiler, and hardly allows to format fragments. Using the fcl parser will be easier and allows to create a more comfortable formatter. Or do it like the codetools and write several different parsers for the different goals.

Mattias