Hi all:

I have fond memories of Turbo Pascal and Delphi, and the latest FreePascal version seems promising enough. When I read that you can also write embedded firmware with it, I decided to have a look. I need a Windows and Linux GUI for my embedded firmware, and a development environment that can do both is pretty enticing.

The first thing I wondered is how come that not everyone is using FreePascal by now. After Visual Basic turned .NET and Delphi lost focus (the IDE became big, slow and expensive), FreePascal/Lazarus seems a perfect fit. What could be the reason? Only a few open-source projects are honest enough to have a "Caveats" or "Disadvantages" page, and FreePascal does not seem to be among them. The "Known Problem" page does not actually fit the bill.

The only thing I found when searching for "FreePascal disadvantages" was a general dislike of the Pascal language, which may be justified if there are many more gotchas like this:

"for loop variable value reliable after loop?"
http://lists.freepascal.org/pipermail/fpc-pascal/2015-October/045446.html

And there you go, I haven't quite started yet and I already discovered the first serious issue: there is no 'volatile' keyword. This matter has been brought up in the past, and it worries me that it has just been downplayed.

C's 'volatile' semantics are not perfect, but you need something along that line when writing multi-threaded programs, or when writing embedded software. If you take a look at the software libraries for STM or Atmel microcontrollers, you will see tons of 'volatile' (or ARM CMSIS' synonym '__IO') around. All of the peripheral registers are defined like that, and there are many of them in a modern microcontroller.

The lack of 'volatile' probably means constant paranoia and never-ending work-arounds when porting multithreading code to FreePascal, or when writing embedded software with it. From what I have seen, it also means that the compiler will never be able to optimise across units, preventing the kind of global optimisation that GCC's LTO does. If you haven't got the time to make sure that everything a loop needs is inside the same unit, LTO can make a huge difference. Or did I get the wrong impression from FreePascal in this area?

Maintaining your own independent compiler is hard, and I would have expected that FreePascal had turned to GCC or LLVM a long time ago. There is some effort in the LLVM front, but without 'volatile' in the main compiler, LLVM is probably not going to fix this issue, at least in the short or medium term. I had a quick look, and there does not seem to be any documentation about the reason why the LLVM port was started. I wonder what other deficiencies are lurking in the main compiler at the moment, because it looks like FreePascal users will have to live with the "traditional" compiler for a long time to come.

A few years ago and gave Lazarus a try. At the time I just wanted to write a small, portable GUI, and I had trouble debugging with the IDE. I think it uses GDB under the hood, which tends to be problematic if not written to perfection (I have made the same experience with Emacs, DDD and others). It may have to do with driving GDB with text, instead of using it as a library. FreePascal's text-mode IDE doesn't even have a webpage (or at least I haven't found one).

Things like this do not encourage trust either:

"Debugging smart linked code might not work correctly. This is due to the fact that no type information is emitted for smartlinked code."

Therefore, before I delve any deeper: are there any other serious drawbacks that I should know about before I invest more time looking at FreePascal?

Regards,
rdiez

I'm not sure what you mean by "more gotchas like this":
* language behaviour that is not in the documentation (i.e., documentation bugs)
* language behaviour that may be unexpected if you are used to e.g. C, but that is considered normal in Pascal (the above behaviour was already defined in the Extended Pascal standard in 1990/1991: http://www.pascal-central.com/docs/iso10206.pdf , Section 6.9.3.9.1 on the page numbered as 104, which is at page 116 of the PDF document).
* something else?

In the first two cases, the answer is definitely "yes".

The reason the need for volatile on embedded systems is far less in FPC, is the existence of the "absolute" keyword. In C, you have to declare all of those memory mapped registers as volatile pointers, because otherwise things will get optimised wrongly. In FPC, you can declare them as variables that are "absolute" at some address, and the compiler will never optimise their accesses. Only if you would take the address of one of these variables and store it in a pointer, you would need "volatile" in this context.

The lack of 'volatile' probably means constant paranoia and never-ending work-arounds when porting multithreading code to FreePascal,

In FPC, every statement is basically considered to be a compiler-level memory barrier for accesses to global variables. So within a single statement multiple accesses can be optimised, but that cannot happen not across statements. It's not perfect since it inhibits some optimisation, but on the flip side it's very simple and easy to reason about.

I think there will indeed be some issues related to volatile-like behaviour, moreover since calls to external routines effectively act as compiler-level memory barriers right now (e.g. a call to give up a time slice in a spinlocking loop -- although in that case there probably won't be any problems, since such a routine would be a black box even to GCC/LLVM).

Maintaining your own independent compiler is hard, and I would have expected that FreePascal had turned to GCC or LLVM a long time ago.

It's not so hard if you have a modular compiler design implemented in a language you're very familiar with (like Object Pascal :). Keeping up with the insane rate of change in LLVM, and/or being forced to write everything in C or C++, would be much harder.

There is some effort in the LLVM front, but without 'volatile' in the main compiler, LLVM is probably not going to fix this issue, at least in the short or medium term. I had a quick look, and there does not seem to be any documentation about the reason why the LLVM port was started.

It was started for the same reason that pretty much everything else in FPC was started: because it seemed interesting to someone who was willing to invest time in it. In this case, that was me, because I think it's interesting to see what its optimisers can do with our code.

Therefore, before I delve any deeper: are there any other serious drawbacks that I should know about before I invest more time looking at FreePascal?

From the top of my head:
* there is no official language standard we follow, there is only the Delphi "de facto" standard (which often has to be reverse-engineered through writing tests, and it's seldom completely documented) and occasionally our own extensions. For non-ISO modes, the official Pascal standards serve as broad guidelines, or to define a particular behaviour if we don't have a personal preference either.
* we are not backed by any big industry players (although that's also an advantage)
* we don't follow any hypes, nor do we launch any ourselves (may be a pro or a con, depending on the hype and your opinion about it)
* the compiler and RTL mainly get improved in the areas for which people write (good quality) patches. I.e., there is no grand strategy, other than the very general and subjective "build a good and usable Pascal compiler that supports multiple dialects and architectures"
* there are still a number of compiler switches left from the early days when pretty much everyone's personal "hey, this may be interesting one day and it seems to work for this one liner or this one platform, so it should be fine"-hack got committed, and which hence may not work very well other cases it's simply functionality that bit rotted over time. These get gradually cleaned up, but there are probably still some left
* similarly to the above, there are some areas of the compiler that are less well-tested than others because fewer people exercise them, and hence you may be more likely to encounter bugs in them (for example the Intel-style external assembler writers)

Jonas

R. Diez wrote

Hi all:

[...]

[...]

(semi-jokingly)

I know FreePascal is not Linux, the Team is not Torvalds etc ;) but have you seen this ?

https://lwn.net/Articles/233482/

[...]

(again, semi-jokingly)
It's Free and Open Software. You too can send a patch/suggestion ;)

Regards, rdiez

el es

Jonas Maebe wrote

R. Diez wrote:

Hello,

Please note that the original poster doesn't seem to be subscribed to the list (in Cc: now).

Tomas

In our previous episode, Jonas Maebe said:

(you also need it for variables you access from interrupt routines, as they might change while a procedure is running)

That's what I remember from earlier "volatile" discussions. But what about LLVM? If you want to squeeze performance out of it, won't that need volatile sooner or later?

But I agree that this is the usual smart shopping mentality. When shopping for a compiler, people drone on the most detail optimization opportunities, but once in production they put it on the lowest common denomitor they developed/tested with, and call that "validated".

marcov wrote on Fri, 18 Dec 2015:

In our previous episode, Jonas Maebe said: > The reason the need for volatile on embedded systems is far less in FPC, > is the existence of the "absolute" keyword. In C, you have to declare

[snip]

(you also need it for variables you access from interrupt routines, as they might change while a procedure is running)

If they're global variables, which they presumably are, that's no more necessary than in the multi-threading case.

It will help, but optimising accesses to global variables is not the only thing LLVM can do :) Right now, I'll indeed should mark all loads from and stores to global variables as volatile (which I don't do yet and therefore tests/test/tmt1.pp gets into an endless loop right now when compiled with LLVM).

Jonas

Thanks for your detailed answer.

The issue with the pointers would definitely impact me, as my microcontroller has many GPIOs, and I have written classes like "ClassSpi" that take GPIOs as pointers. This way, I can implement generic functionality which does not depend on the exact GPIOs being used.

Lukasz Sokol mentioned the approach taken by the Linux community (I didn't get his e-mail address). They have a fat kernel with lots of synchronisation primitives, which you do not normally have when programming for Cortex-Mx microcontrollers.

Issuing a generic memory barrier is workable, but it kills performance, as all variables will be reloaded again. Performance does matter when working on microcontrollers.

Basically, the Linux approach would then boil down to writing a wrapper function for every "special" hardware-related memory access. For the simple case of GPIOs, the wrapper function would then take a pointer to the GPIO and would access that pointer within a type-cast to 'volatile'. So basically you have to wrap everyhing, which is a pain for "simple" embedded Firmware, but is doable.

However, FreePascal does not have such a routine
that works with generic pointers, and not with fixed "absolute" global variables, I guess the only work-around would be to write some inline assembly, which would again kill performance, because FreePascal probably assumes that inline assembly can modify everything else, so all variables will be reloaded afterwards as in the case of a generic memory barrier. Is that right?

I see. Based on that information, and on the issue of function calls crossing units, I guess you could rephrase that in much less-friendly way. I'm playing evil advocate now: "FreePascal is so simple it can only optimise properly the kind of hand-written tight loops that you normally find in artificial benchmarks." }8-)

I find that kind of answer disappointing. In this particular case, the FreePascal developers could have just provided that guarantee. Is there at least a compilation warning if you access the variable after the loop?

If I have to learn a new programming language, I would avoid those with esoteric gotchas like that. The "it's in the spec" line is a lame excuse. The Java language, for example, is much cleaner and does not have such traps for the unwary (in my personal opinion).

At the risk of sounding too cynical, I'd like to say that I have been burnt many times. That's why I try to "poke around" a bit before investing too much time in the next cool thing. Besides, I hope this discussion helps other people. For example, I found the link to the Linux volatile discussion interesting.

Regards,
rdiez

In our previous episode, Jonas Maebe said:

True, but I write less low level hardware interfacing code there, and use a lot more globals on embedded :-)

Also the OS level synchronization primitives used in threaded "full" applications are usually not available on microcontrollers, volatile and the atomicity of (some) instructions is all you have to work with. (yes I do the microcontrollers in C, so the below condition doesn't apply)

Is this between statements caching rule really only globals, or anything non local (like a field of a class) ?

Hi,

R. Diez wrote

Maintaining your own independent compiler is hard, and I would have expected that FreePascal had turned to GCC or LLVM a long time ago.

Thank God that never happened (not as the first option anyway), considering the state of the GNU toolchain on some platforms we actively support. (And LLVM support is simply non-existing for many others.)

OK, these are usually very small platforms, but the large development communities usually consider these a "nuance", and actively left behind or pushed out of the main development trees, while with FPC we still somehow manage to happily co-exist with the big ones.

Charlie

R. Diez wrote

Issuing a generic memory barrier is workable, but it kills performance, as all variables will be reloaded again. Performance does matter when working on microcontrollers.

Could I ask for a reality check here. That sounds more like a cache flush, while a membar only ensures that any reordered reads and writes are disentangled: I'd not expect it to invalidate registers.

Actually, it is the other way round: FPC is not interested in optimziations taking a lot of time and mainly having a purpose in artifical benchmarks. E.g. spending a lot of time in optimizing globals variable access is for real world programs normally not really usefull as the number of global variables in good programs is really low.

Of course, if you are interested in highly optimized code squeezing out every processor cycle, you do not care about compilation times and you are only interested in embedded programming, FPC is not the right tool for you and this will not change.

Lukasz Sokol wrote

I know FreePascal is not Linux, the Team is not Torvalds etc;) but have you seen this ? https://lwn.net/Articles/233482/

Maybe a "volatile" intrinsic, just like we already have an "unaligned" intrinsic, would indeed be a better idea than adding the possibility to mark data as volatile. As an extra plus, it would map directly onto LLVM :) (which also only supports "volatile" as an attribute for load/store instructions).

Jonas

intrinsic, would indeed be a better idea than adding the possibility to mark data as volatile. As an extra plus, it would map directly onto LLVM :) (which also only supports "volatile" as an attribute for load/store instructions).

I've had the same thought when I read the article. :)

Regards,
Sven

On 12/18/2015 06:16 AM, R. Diez wrote:
[...]

not really... backwards compatibility and all that... borland's TP and BP do the same... at least back as far as v6... i can't get to my v5 or v3 to test but i daresay that they, too, do the same... looking forward from them, i would hazard a guess that delphi also does the same...

but then again, perhaps the devs could provide that guarantee in one of the available language modes... perhaps with an additional mode switch...

Is there at least a compilation warning if you access the variable after the loop?

i would expect not unless maybe with the above mentioned languages modes and mode switches... but they don't exist for this situation, AFAIK...

Hello,

I am interested in this problem too. My app is highly multithreaded, working mostly with TObject descendants such as TStringList and TCollections. I am ensuring thread safety thru CriticalSections. In fact I have made my own TThreadSafeStringList.

I have been distributing the app with optimizations OFF because it didn't work with optimizations on. I only try optimizations ON once every three to four years but quickly go back to optimizations OFF.

I optimize my code in such a way that compiler optimizations are not really needed, but obviously I wouldn't say no if they were reliable.

Could it be related to this discussion? Do optimizations make apps less thread safe in FPC?

Cheers,
Tobias

No. The fact that your code works without optimizations and not with either means that there's a bug in the optimizer, or that there's a bug in your code which happens to be triggered (more reliably) by an optimization.

E.g., if you don't initialise a variable somewhere, maybe it's value without optimizations (on the stack) happens to be 0 initially, but when it's moved to a register (with optimisations) it has another value.

Jonas

http://lists.freepascal.org/pipermail/fpc-pascal/2015-October/045446.html

do the same... at least back as far as v6... i can't get to my v5 or v3 to test but i daresay that they, too, do the same... looking forward from them, i would hazard a guess that delphi also does the same...

but then again, perhaps the devs could provide that guarantee in one of

the available language modes... perhaps with an additional mode switch...

Why should we? That will only prohibit potential optimizations of the loop...

Regards,
Sven

Sven Barth wrote

Am 19.12.2015 15:54 schrieb

[...]

that's why i said "perhaps" ;)

i've run into it years ago but didn't know why... the recent discussions about it educated me... but long ago, i changed my coding patterns so as to not run into the situation...