I had another idea yesterday while working with Swift and I was curious to hear what anyone thought. In Pascal we have “const”, “constref" “var”, “out” params for functions but no way to document or enforce optional params (which are pointers/classes).

It’s a very common pattern that a function takes a pointer as a parameter and we don’t know if we’re allowed to pass null unless we ready the comment or documentation. I thought there could be an “optional” or “opt” modifier that serves to 1) document the usage of the function and enforce nil checks on the pointer when dereferencing. It’s kind of like “const” for read-only parameters but it says “must-check” instead.

Basically, if a param (or result even?) was marked optional then you would get a compiler error if you tried to dereference the pointer, unless you were in an assigned block (or some other intrinsic since “assigned” is just a function).

What do you think of that? Sounds like an easy way to get some support for nil pointers deref’s and provides self documenting code.

====================

program test;
procedure FreeThis(optional var obj: TObject);
begin
  // must check assigned or we get a compiler error
  if assigned(obj) then
    begin
      obj.Free;
      obj := nil;
    end;
end;
procedure DoThis(optional obj: TObject);
begin
  //  must check assigned or we get a compiler error
  if assigned(obj) then
    obj.DoIt;
end;
// can return nil so return value must be checked
function MakeObject: TObject; optional;
begin
  result := TObject.Create;
end;
var
  obj: TObject;
begin
  // MakeObject result is optional so we need to check for assigned
  obj := MakeObject;
  // must check assigned or we get a compiler error when we read the class name
  if assigned(obj) then
    begin
      writeln(‘new object:’, obj.ClassName);
      DoThis(obj);
      FreeThis(obj);
      // "obj" is "optional var" so we get a compiler error unless we check for assigned
      writeln(obj.ClassName);
    end;
end.

Regards,
Ryan Joseph

Op 2019-04-12 om 17:23 schreef Ryan Joseph:

What do you think of that? Sounds like an easy way to get some support for nil pointers deref’s and provides self documenting code.

I think the same as when I read the suggestion for an inout variant of VAR. Move this out of the language syntax, and make it directives or attribute like syntax (like we will need to get anyway for const ref).

As to the functionality itself, IMHO it is a solution in search of a problem. Yes, it theoretically nails down the contract a bit more, but I rarely expect that this makes programming in pascal any easier.

It all is too much topdown, assuming that you can oversee all extensions and problems into the far future when definining a procedure.

I think the time can be better spent in making better debug tools and/or static analyser tools for what there is than bother the language with syntax that fixes only hypothetical corner cases.

On Apr 12, 2019, at 2:55 PM, Marco van de Voort wrote: I think the same as when I read the suggestion for an inout variant of VAR. Move this out of the language syntax, and make it directives or attribute like syntax (like we will need to get anyway for const ref).

Example please? I didn’t know there were alternatives to parameter modifiers.

As to the functionality itself, IMHO it is a solution in search of a problem. Yes, it theoretically nails down the contract a bit more, but I rarely expect that this makes programming in pascal any easier.

What do you do about the problem of not knowing when a parameter of a 3rd party function can accept null? That alone is an annoying little detail that arises often and requires looking at the code comments or manuals. Same situation with functions results, I often don’t know if the result could be nil and I perform a check just to make sure. Having the function itself document this would be helpful and a compiler error would be even more helpful.

Having said that I admit to not using “const” and almost regret its introduction because now programers use it everywhere, even when not needed. Constref on the other hand was a great addition and “out” and “var” are obviously needed.

Btw a little off topic, the reason I don’t use const is because it doesn’t do the one thing I want it to, i.e. ensure members are not modified in a function. In the example below why doesn’t const give me an error when I try to assign to the “x” member variable? That’s the one reason I want const and it doesn’t even do that one thing! Grrr. Maybe we need another modifier, “realconst". :)

type
  TMyRecord = record
    x: integer;
  end;
  PMyRecord = ^TMyRecord;
procedure DoThis(const p: PMyRecord);
begin
  p.x := 10;
end;

Regards,
Ryan Joseph

Ryan Joseph wrote

What do you do about the problem of not knowing when a parameter of a 3rd party function can accept null?

Normally, if it can't accept nil it should be var-parameter, and if it can then it should be a pointer.

This becomes less clear if a parameter type is already an implicit pointer type (like a class or interface), but there are fewer case of that kind.

Jonas

I actually suggest this solely for the use of pointer types in order to:

1) prevent nil pointer dereferencing (do I need to check for nil on this parameter/return value or is it safe to access?)
2) self documenting code (can this parameter accept nil?)

Using var and pointers to pointers are possible workarounds but they’re not really expressing what we mean so it seemed logical to add this to the list of parameter modifiers already existing.

Regards,
Ryan Joseph

Marco van de Voort schrieb am Fr., 12. Apr.
2019, 20:56:

Im definitely for new syntaxes (or the attribute one) than a directive. A directive can be anywhere and I'd need to go looking for it if I want to know whether it is set or not. Some syntax extension would be right at the declaration.

So, yeah, we'll probably compromise towards the attribute syntax...

Regards,
Sven

What's wrong with ...

procedure FreeThis; overload;
begin
end;
procedure FreeThis(var obj: TObject); overload;
begin
end;

There, now you have an optional argument with a var reference.

Maybe my example wasn’t very clear. Here’s a very common example. We have a function that takes params and has a return value all of which can be null. As the programmer we have to read function comments or check for nil just in case because the documents weren’t clear. Doing "if assigned(p) then” out of fear is not very fun and makes for messy code.

{
creates a new dude
@param name: name of the dude
@param items: list of items (optional)
@param attrs: list of attributes or nil for default attributes @return reference to new dude (may be nil)
}
function CreateNewDude(name: string; items: PItemsList; attrs: PAttrList): PDude;

The attribute in the same way const says the value can’t be changed, says “may be optional so you must check”. If you forget to check for nil the compiler gives you an error.

function CreateNewDude(name: string; optional items: PItemsList; optional attrs: PAttrList): PDude; optional;

It’s a minor detail in the same way const is but it adds a little extra information and gives a little extra safety (I don’t like testing for nil out of fear, I want to know forthright). Swift did a massive overblown implementation of this idea which is a nightmare but the basic concept is as sound as “const" is.

Am I wrong about this not being common? Marco seemed to think so but I encounter this daily in the real world. In fact just yesterday I parsed the IOKit framework (Mac) to Pascal and saw Apple went through great lengths to add null-defined macros that mark params as “nullable” or not. I.e.

typedef void (*IOHIDCallback)(
void * _Nullable context,
IOReturn result,

                                    void * _Nullable        sender);

So how is this not a common occurrence that deserves a solution?

Regards,
Ryan Joseph

Hi,

the parameter is already optional without any modifier, since you can always pass nil for it.

Perhaps there could be the opposite modifier, so the function cannot be called with nil. Like

procedure DoThis(nonnil var obj: TObject);

and then you need to check if it is nil, before calling DoThis .

Or you put it in the type. type TSomeObject = nonnil class(TObject)... and then it would never be allowed to be nil.
Or for pointers type PDude = nonnil ^TDude;
If you would want to use such a nonnil class as field in another class, you would need to initialize it in the constructor.

Bye,
Benito

Well nil is only with objects. For strings you could expect that there are not empty.
For integers you can have sub-ranges. But for float, you may have a condition that says > 0.

I think this goes towards https://en.wikipedia.org/wiki/Design_by_contract pre/post conditions.

Those can then be applied for anything. And the compiler can add compile/runtime checks, the same way it does range checking.

As stated on wikipaedia this also plays towards docs. If a function has a postcondition that either forbids nil, or explicitly allows it, then you do not need to look at its code. You know what to expect.

Sorry I must be doing a bad job explaining this and the naming of “optional” seems to be confusing since it conflicts with existing terms. The param/return is indeed optional but there’s no way to know that as the programmer.

Ok, here’s a historic real work example which is all too common:

  int pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg);

Can you pass null for attr? Well you need to read the man page and read through multiple paragraphs to find out "If attr is NULL, then the thread is created with default attributes.” See: http://man7.org/linux/man-pages/man3/pthread_create.3.html

Clearing up this situation with a tiny little syntax is all I propose. Make that fact known in code and force pthread_create to check if attr is nil before dereferencing it.

Perhaps there could be the opposite modifier, so the function cannot be called with nil. Like

Interesting idea but I’d have to think about it more to know if this is a real problem I’ve ever experienced.

Regards,
Ryan Joseph

Well that doesn't work. Unless you wrote pthread_create (or whatever other function).
Because the "opitonal" parameter (which really only serves as documentation hint) is itself optional.
So if the author of the code that you intend to use did not put "optional" in there, you are still where you are now. And you cant force the author to put it there. Same as you cant force the author to put it on top of the documentation.

Already today the author could add a comment (even pasdoc) to the declaration. And it serves the same effect.

Also the is a difference between:
- the parameter is nil-able (better term that optional, optional has a different meaning)
- the compiler needs to enforce an check for " nil"

If the parameter is only stored, for later usage, then the nil can be stored too. There is no check needed for that.
It will be needed later in some other function, but the compiler can not know where that may be.

You can even call methods on an object that is nil. So long as they are:
- not virtual, nor call virtual methods themself
- do not access "self" without prior check that it is not nil

Sure, if it’s not used then it’s not very helpful. :) Comments are the same effect? Then why not prefer comments for “const” params or even public/private sections. Surely this isn’t a good idea:

type
  TClassName = class (TObject)
    {private}
    m_var: integer;
    {public}
    procedure DoThis({const} p: TPoint);
  end;

They’re just simple hints that provide some compile time errors.

All true. I only propose this for cases where nil could be passed/returned and the programmer can be given more information and forced to check since this is the pattern they were supposed to do anyways. It’s not a magic catch-all, just a minor markup thing like “const” is.

Regards,
Ryan Joseph

Ryan Joseph wrote

> Already today the author could add a comment (even pasdoc) to the declaration. And it serves the same effect. Sure, if it’s not used then it’s not very helpful. :) Comments are the same effect? Then why not prefer comments for “const” params or even public/private sections. Surely this isn’t a good idea:

You misread me. I am sure inadvertently, but still. You advertised that this feature you desire was to fulfil two purposes. One of which you named to be documentation. Or as you described, the elimination of the need to look through copious amounts of implementation, or in other cases pages of documentation.

As for this part of your request, I made it known that it is my belief that comments can have the same effect.

As for the  second purpose that you had brought to our attention, I gave answer to it further down in my mail.

Well, as far as the "optional" part goes, the benefits of it will be fairly limited. Object variables occur in many places, and all of them can have the nil value, and the compiler has no idea if you did the check.

You pick a tiny subset of those cases, and attempt to add protection to it. Given the gain (or rather absence of it, according to the microscopic size of that subset) the effort (and chance of false positives) is not worth it.
I have already in todays code encountered the compiler telling me variables were not initialized, were clearly they were. But the compiler was not smart enough to follow the code through various conditional blocks. If the compiler were to force that check, when at the same time it is not smart enough to see, that I actually already have it, then clearly that would be a problem.

Besides there is a "loop"hole (pun intended / follows) in this feature. The feature creates a circular (loop) dependency.
If I want the compiler to force me to add the check, I need to add the keyword. But if I remember to add the keyword, well I may as well add the check instead. After all I am already aware of its need, or I would not add the keyword.

So if I am not aware of the need for the check, I am likely not to add the keyword, and I will never know (until my app crashes)

On Apr 13, 2019, at 3:58 PM, Martin Frb wrote: You misread me. I am sure inadvertently, but still. You advertised that this feature you desire was to fulfil two purposes. One of which you named to be documentation. Or as you described, the elimination of the need to look through copious amounts of implementation, or in other cases pages of documentation.

I really don’t disagree with you that much since my proposal is minor it has a minor benefit also. The system frameworks on Mac have used empty macros to the same effect so clearly comments work also. i.e.

IOHIDDeviceRef _Nullable IOHIDDeviceCreate(
CFAllocatorRef _Nullable allocator,
io_service_t service)

I appreciate Apple added those but we could do the same in Pascal with macros. The engineers at Apple clearly thought the problem was big enough to add it into their new language (Swift) and at least for pointers I agree it’s nice.

Well, as far as the "optional" part goes, the benefits of it will be fairly limited. Object variables occur in many places, and all of them can have the nil value, and the compiler has no idea if you did the check. You pick a tiny subset of those cases, and attempt to add protection to it. Given the gain (or rather absence of it, according to the microscopic size of that subset) the effort (and chance of false positives) is not worth it.

I’m not sure it’s possible to implement without having side effects. Just floating out the idea.

I can find many hundreds of examples like the one above in the Cocoa frameworks since Apple retroactively added them in. I don’t understand why you think this is a minor subset of cases. Btw, they added the macros into the Objective-C headers so the Swift parser could correctly interpret them as optional types (denoted with the ? symbol if you’re familiar with Swift).

Anyways, not a single person thought this was a good idea so I’ll leave it be. :)

Regards,
Ryan Joseph

Ryan Joseph wrote

I can find many hundreds of examples like the one above in the Cocoa frameworks since Apple retroactively added them in. I don’t understand why you think this is a minor subset of cases.

That "minor set" referred to compiler warnings you get, inside a function that has an optional argument, if you do not check that argument for nil:
Objects that are nil happen everywhere in the code.  The amount of object accesses with potential nil issues that will be in the scope of such an optional parameter is minor compared to the amount of overall such accesses.

The amount of method declaration that could benefit from additional clarity towards what values they can take in there arguments (and hopefully deductible from that their purpose) may indeed be noticeable. But for that there should be better solutions. Not at least because nil-ability is only one such concern.
Were it only the documentation of nil-ability then comments would do the job.
When it is about a more generic approach then see the wikipage about contracts. (I did mail a link)

On Apr 13, 2019, at 5:11 PM, Martin Frb wrote: Were it only the documentation of nil-ability then comments would do the job. When it is about a more generic approach then see the wikipage about contracts. (I did mail a link)

If there was any other way the compiler could make this contract more concrete and reliable then comments I’d be happy to hear it.

Regards,
Ryan Joseph

Ryan Joseph wrote

If there was any other way the compiler could make this contract more concrete and reliable then comments I’d be happy to hear it.

I don't know what others think of it, but
https://en.wikipedia.org/wiki/Design_by_contract
look at pre/post-conditions.

It basically is like assertions. But it is a bit more, because there are rules for inheritance. So it should be part of the declaration, not the implementation.

Once such conditions exists, the compiler could be enhanced in different ways:
- insert the correct assertions
- check at compile time, if possible (may give errors, warnings, hints, depending how certain the compiler is)

Of course it does a bit of the opposite of what you did. It allows to say, when a param is not allowed nil.

By implication any param that has no pre-condition is allowed nil, and the compiler should assume it may be nil.

----------
Back to your original issue.

To express that a param could be nil, to indicate it is omitted. If you can put it to the end of the list
  procedure Foo(Bar: TBar; Opt: TOpt = nil);
makes it perfectly clear, that nil is an intended accepted value

Well, there is Oxygene's concept of Class Contracts (see https://docs.elementscompiler.com/Concepts/ClassContracts/ ), so if anything in that direction would be done I'd be inclined towards their syntax (and I've played with the idea to implement this in FPC for quite some time already).

Though of course it wouldn't necessarily solve the point of knowing whether one may pass Nil or not, cause especially in compiled code you can't look at the source. So the compiler would need to store the conditions in the PPU as well, so that the IDE could present them as part of the tooltip even if no source is available...

Regards,
Sven

That’s an interesting idea. It’s certainly common that you test for these conditions at the top of the function and have bail out conditions so it makes sense to pull it out into syntax. But it has to rely on exceptions and all that overhead? That’s not so great if so.

Regards,
Ryan Joseph

Well it needs to be in the ppu anyway, because even without source, an object can be inherited from.

And the inherited method must fulfil some of the conditions
- It can't accept less than its base (requirements can be looser, but not tighter)
- It must fulfil the "ensure"s, or be even stricter. (as that would still fulfil them)

Also being in the ppu, would allow compile time errors, when passing violating constants.

There are a few things about the oxygen syntax.

1) require is outside the begin..end, but ensure is inside.

2) As the nature of those conditions can be inherited, they are more to be thought of as declaration, than implementation. That would call for them to be in the interface. Though of course that messes up the interface....

> Perhaps there could be the opposite modifier, so the function cannot be called with nil. Like Interesting idea but I’d have to think about it more to know if this is a real problem I’ve ever experienced.

for example, I store some interfaces in a list, and just replaced the default list with a faster list that assumes all interfaces in the list are non-nil. (When you know the interface is non-nil, you can replace the automatic reference counting with a direct call to ._addref, which is faster. )

But then it crashed because somewhere it was used with a nil interface. There it would have been useful to mark the new function as only non-nil accepting, so that it is clear it cannot be used like the old function, and the compiler have made sure there is no call with nil

You would need to have optional or non-nil markers at every variable, before it becomes really useful.

Benito

Ryan Joseph schrieb am So., 14. Apr. 2019,
00:17:

Your idea requires exceptions as well. The compiler is not capable checking whether valid entries are passed in at compile time except for simple, constant cases, so runtime checks are needed and the only way to abort something is by exceptions.

Regards,
Sven

On Apr 14, 2019, at 3:15 AM, Sven Barth via fpc-pascal wrote: Your idea requires exceptions as well. The compiler is not capable checking whether valid entries are passed in at compile time except for simple, constant cases, so runtime checks are needed and the only way to abort something is by exceptions.

My idea didn’t require the procedure to exit so I don’t see where an exception would be needed. It just says if a param was flagged you would need to check for nil before dereferencing other you get a compile time error.

I like the idea (because it feels Pascalish) to make a code section that checks requirements but the exit condition is up for debate. I personally wouldn’t use the feature if it meant I need to start the tedious process of wrapping everything in try blocks and taking on what performance costs there are to exceptions (we can’t opt into exceptions via modeswitch so I don’t know what those things are doing and where they are).

We have custom error systems in our code also so forcing everyone into exceptions would not be nice. I could imagine a bail condition that was jumped to but it would have to be in the code section. That’s usually how we do it now anyways so it shouldn’t be too strange.

function MakeHero (name: string; hp: integer; level: integer): THero;
require
  name  ‘’;
  hp > 0;
  (level > 0) and (level < 10);
begin
  …
bail
  MyCustomErrorHandler(‘MakeHero() params are invalid’);
  exit(THero.Invalid);
end;

Regards,
Ryan Joseph

Ryan Joseph wrote

> On Apr 14, 2019, at 3:15 AM, Sven Barth via fpc-pascal wrote: > > Your idea requires exceptions as well. The compiler is not capable checking whether valid entries are passed in at compile time except for simple, constant cases, so runtime checks are needed and the only way to abort something is by exceptions. My idea didn’t require the procedure to exit so I don’t see where an exception would be needed. It just says if a param was flagged you would need to check for nil before dereferencing other you get a compile time error.

First of all the max you should get is a note (warning, maybe). But never a compile time error.

I have code like
   If a > 7 then
      (* If b = nil then *) b:= somedefault;
   foo();
   if a = 8 then
      a := b;  // compiler claim b is not initialized     (* a := b.Foo *)

So if your request for a compile time error was implemented, and above was a pointer deref on the last line, the code would not compile. Yet it is safe.

Also see my previous comment.
What is the point, if the compiler checks for potential issues, but only if I insert a request for the check on top of the potential unsave code.

If I have to write "optional" so the compiler checks for "if assigned" then what good is it? I can write "if assigned " immediately and save myself the optional.

And please do not counter this with: "But it documents for the user that he can use nil". The documentation is a separate distinct issue, and it can be solved in other ways.
You proposed one keyword to archive 2 features, be prepared to discuss each of them separately.

I like the idea (because it feels Pascalish) to make a code section that checks requirements but the exit condition is up for debate. I personally wouldn’t use the feature if it meant I need to start the tedious process of wrapping everything in try blocks and taking on what performance costs there are to exceptions (we can’t opt into exceptions via modeswitch so I don’t know what those things are doing and where they are).

There would be no need for try blocks.

It is the same as "assert". An assert raises an exception. But you do not write try except blocks for it.
An assert should never happen, therefore you do not handle it. An assert is a controlled crash of the app. Asserts are compiled into your app, if you test it. But you leave them out if you release. Once you have tested (automatic test case), you no longer need the assert, because the case can never happen. (Or the third party using your code have to test, they follow the API. Or you allow nil for them, and you tested that it does work)

I understand that this is not the same you propose. It is for cases where nil is NOT allowed, rather than for cases were nil is allowed. (as above, if nil is allowed, the code must be tested with nil. The compiler can never detect all cases, therefore the compiler can never enforce that you do "if assigned"... You may have done, without the compiler having noted)

You do not need the exit. Because the purpose of the assert is to abort the entire application (controlled crash).

If you get a call with invalid param, you have to assume this is because something else went wrong before. So in this case you have to assume worst case, something like the entire memory of the application has been corrupted.
There no longer is a way to rescue the app. All you can do is make sure that it will not do further damage. So you exit the app.

The IDE for example uses Application.OnException.
It tells the user that it is now unstable. Save your work and exit.

As for your feature of getting a note from the compiler, if you deref a param (or other variable) that according to the compilers awareness could be nil at the time.

There is on need to say "optional".
Technically any nil-able var could be nil. So the compiler note should be for any such variable. Exactly like currently the "variable not initialized".

The downside: That would flood current code with compiler-notes. Because any param could be nil, even if the programmer has documented somewhere that this is not allowed.

But if pre-conditions where available, this could be combined. The compiler could assume that if you wrote pre-conditions for a method, and that if you did non add "foo nil" to those conditions, then you expect that foo can be nil.
And in that case the compiler could add appropriate notes, if it finds you do not check for nil.

Existing code has no pre-conditions and is not affected. New code then must contain this.

Or the warning could apply to all code, but by default is not generated. You can enable it for each unit via commandline switch or directive. Again no "optional" needed.

Ryan Joseph wrote

> On Apr 14, 2019, at 3:15 AM, Sven Barth via fpc-pascal wrote: > > Your idea requires exceptions as well. The compiler is not capable checking whether valid entries are passed in at compile time except for simple, constant cases, so runtime checks are needed and the only way to abort something is by exceptions. My idea didn’t require the procedure to exit so I don’t see where an exception would be needed. It just says if a param was flagged you would need to check for nil before dereferencing other you get a compile time error.

As already said by Martin: the compiler *can not* determine all cases whether the parameter is Nil or not, so it *must* be done at runtime to ensure this. Otherwise the feature is just as useful as this:

=== code begin ===

{$macro on}
{$define optional:=}
procedure Blubb(aArg: TObject optional);
begin
end;

=== code end ===

I like the idea (because it feels Pascalish) to make a code section that checks requirements but the exit condition is up for debate. I personally wouldn’t use the feature if it meant I need to start the tedious process of wrapping everything in try blocks and taking on what performance costs there are to exceptions (we can’t opt into exceptions via modeswitch so I don’t know what those things are doing and where they are).

As Martin said: this is an error condition like an Assert that should trigger during development with the default result being a termination of the application.

Invalid contracts are an error that should trigger during development, but not necessarily when released. So coupling them with the default exception mechanism is a valid solution.

Regards,
Sven

On Apr 14, 2019, at 1:38 PM, Sven Barth via fpc-pascal wrote: As already said by Martin: the compiler *can not* determine all cases whether the parameter is Nil or not, so it *must* be done at runtime to ensure this. Otherwise the feature is just as useful as this:

I’ve read over what Martin said and honestly I’m confused now. :) I’m not sure if I don’t understand you guys or you don’t understand me.

Lets use the example of the optional (i.e. "could be nil") return value because it’s most easy to understand. Why can’t the compiler know that the result of GetThing is an optional and therefore you *must* always check any code that dereferences it? Isn’t this a compile time issue?

In the example below I would always check for not nil if the documentation said it may be nil so why can’t the compiler just make you do that anyways and provide the information right there in the declaration? I don’t understand where the runtime element of this is.

function GetThing: TThing; optional;
var
  thing: TThing;
begin
  thing := GetThing;
  if assigned(thing) then
    writeln(thing.name);

Invalid contracts are an error that should trigger during development, but not necessarily when released. So coupling them with the default exception mechanism is a valid solution.

Could you give a custom message for the exception or would it just say “Function X failed conditions”? Usually we give more detailed messages so we know went wrong. Since this is the compiler maybe it could say which condition failed, like “condition x < 0 failed”.

Regards,
Ryan Joseph

Well the answer has several parts.

One thing is what checks can be done, to help the programmer not to accidentality  de-ref nil.
The other thing is how.

1) The "optional" is actually the default. If anything towards a deref-warn feature was done, then *every* parameter,  *every* return-value (and even every other var) of a nil-able type should be seen as optional.
If a type is not optional, then it should be declared as "not nil-able" => function foo(notnil a: TObject): notnil TOBject;

That is in the same way as range checks. There is on "type x=array[1..50] with range checks of integer;"

Using your "Optional" is just specifying what already applies. In other words: redundant.

Now specifying it for a result, is different than specifying it for a parameter.
- For a parameter, the problem was:
   that if I want the compiler to check that I test for "assigned",    => then I must remember that I want the compiler to do that,    => because if I do not remember, then I would not add the "optional"   But:
  If I already remember that I want to have a check for "assigned", then I can do it right away.
  I do not need the "optional", so that I get a warning for forgetting something that I actually just though of.

As something applied to the result, the reminder is for some other person. The person that will write the code.
Now that idea is perfectly fine.
- But the "optional" still is not. The user, the compiler, everyone already knows that the result can be nil. Every TObject can be nil. This is the default.
- So to improve this we would need a way to say: It will never be nil. (I.e., it is different from the default)
If we had that, then there would be nothing wrong with the compiler giving a hint, in case the user does not check.

So that is about the how.
- Reverse the spec, so it acknowledges the current default.
- Then issue warnings for every parameter or return value, that has not been marked as non-nil.

2) If the compiler does check, we have to consider that all current code defaults to nil-able.
To avoid floods of warnings the check would need to be off by default. It could be enabled by
- directive
- command line to the compiler
- fpc.cfg

3) If the compiler does check, it can issue (at compile time) hints, notes, or in very very few cases warnings.
You can use -we to tread them as error, but they can never be error by default.

Code can be very complex. The compiler can never understand all possible ways of implementation. Therefore the compiler will always have false positives.
If those were errors by default, then perfectly valid code could not be compiled.
So hints, notes, and occasional warnings.

4) runtime checks:
Are entirely independent of the above.

However they can be helpful. They would work exactly like range-check errors work today.
If you declare a function can never return nil, then the compiler can (as a separate feature, with its own option) in addition to the hints/notes at compile time, add do not write try except for range checks (at least that is not what range checks are meant for), and you would not do that for nil checks.

-------------------
In conclusion:

The nil-deref-protection would be very much like range checks. (except that the nature of tracing the nil value is more complex, meaning you get notes instead of errors)

range checks can occur at compile time SomeString[-1] should give a compile time error (in that case actually an error) range checks also   does the same.
It applies to *ALL* nil-able types by default.

In order to be useful it is necessary that the programmer has a way to tell the complier that variable/param/returnval do not fall into the default, and that they are never nil.

> Invalid contracts are an error that should trigger during development, but not necessarily when released. So coupling them with the default exception mechanism is a valid solution. Could you give a custom message for the exception or would it just say “Function X failed conditions”? Usually we give more detailed messages so we know went wrong. Since this is the compiler maybe it could say which condition failed, like “condition x < 0 failed”.

See the oxygen page. Yes there are custom messages. But they should be for the developper. Not for the end user. Though of course nothing stops you from (ab-)using them for the enduser.

I read through but I think we’re stuck on this first misunderstanding. This is not the default, I would never suggest this since as you mentioned it leads to massive overreach. Never would I want this added to *every” param and return. Just to get that out of the way. Swift has done this and it SUCKS that we’re forced to unwrap optionals all the time. It’s so much friction and I really hate it.

I’m also not suggesting this means “not nil” either. I mean simply *may* be nil so you need to check. I know that all pointer types COULD be nil.

Per my example, currently we know that GetThing() COULD return nil so unless we know what the function does by reading the documentation or looking at the implementation we need to test for nil. Right? I suggest the modifier simply to enforce that check and let it be known that’s the desired usage.

So, in other words:

	function FindThing: TThing; optional;

means you MUST check for nil. If the value is nil that’s not an error. “optional” is probably a bad name for this I know.

	function MakeThing: TThing;

this means maybe check for nil or maybe not check for nil. We don’t know what the programmer intended because no extra information was given (at least in the declaration). Because of the name we probably assume this is going to return a new object which is safe so checking for nil would be a waste. But we don’t know that except for what the name implies or if we read the code.

Is that more clear? If there’s a better way to express this then please let me know.

For parameters I like what Sven suggested to have a “requires” section in functions to check this but wouldn’t it make sense if the function parameter gave some hint also? “can be nil” or “can’t be nil” would both be helpful in terms of documentation and compile time errors instead of runtime errors.

Regards,
Ryan Joseph

Ryan Joseph wrote

Per my example, currently we know that GetThing() COULD return nil so unless we know what the function does by reading the documentation or looking at the implementation we need to test for nil. Right? I suggest the modifier simply to enforce that check and let it be known that’s the desired usage.

I do get what you want. But not why (the personal why I get, the objective one not).
Or not what sense is it makes... Where again I get your description, but it seems arbitrary/randomly picked.
Sure perfectly fine on the few examples you gave. But then by that measure others will bring their examples (and believe me they will, and it will be examples neither of us would be able to derive today). And then we need further rules/modifiers/dedicated-checks for whatever examples they bring up...

As you wrote: "we need to test for nil"
Yes, and we need to do that for all of them. Even if they are not marked as "expect-me-to-be-nil". Because even if they are not marked, they still can be.

That means if we start warning in selected cases (the ones that are marked), then we give the user **false security** over all other cases. (I did not get a warning, so why does it crash on a nil-deref?)

The only place where we do not need to check for nil, is if we know it can not be nil.
We can gain that knowledge from 1 of 2 places
- some sort of modifier to the type declaration - or some sort of class-contract
- the docs

We can NOT get it from code review. Code may change, and then all the depended code becomes wrong. (It may be that the code should return nil, but due to a bug the current implementation does not)

Don't worry about the name.

How does the compiler know that I have checked. Code can be arbitrarily complex. There will always be code that does check, but the compiler can not tell (heck I may have a checkAssigned procedure in assembler). So at best the compiler can warn during compilation, but never give an error.

function MakeThing: TThing; this means maybe check for nil or maybe not check for nil. We don’t know what the programmer intended because no extra information was given (at least in the declaration). Because of the name we probably assume this is going to return a new object which is safe so checking for nil would be a waste.

What if it run out of memory? And catches the exception. Checking for nil is never a waste. Unless you have proof (in the mathematical sense) that nil can not happen.
Or if your rules allow you to trust the docs, and the docs say you can relay on it.

But we don’t know that except for what the name implies or if we read the code.

Neither name, nor implementation are a reliable source for that info. See above.

Is that more clear? If there’s a better way to express this then please let me know.

Its been perfectly clear from (almost) the start.

I admit the contract/assert stuff may have made it seem different. But that is because this approach does not hold up (IMHO). And therefore the closest alternative was presented.

In the very end it is to avoid crashing on nil de-ref. And the contract/assert can help there. Though in a completely different way.

The pre-condition (requires) would have the same amount of compile time checks (and the remainder runtime) as a keyword like notnil or maybenil. In the same way range checks and others go part compile, part runtime.

As for "documentation". I disagree with the way it is done in oxygen. But I am not sure I have any good alternative.
For me a class contract (require/ensure) is part of the interface.

So it would have to be like (and similar for plain procedures, no class)
// can be all on one line....
type
   TFoo= class
      function DoTheFooThing(Foo1, Foo2: TFoo): Tbar;
         requires
            assigned(Foo1) or assigned(Foo2): 'Need at least 1 foo, for
the connection';
         guarantees // better than ensure
            assigned(result);
            result.KnowsMyFoo = true: 'Connection exists';  // the =true
is redundant
     procedure DoTheBar;
   end;

So reading this declaration, you immediately know what is valid. And if custom messages are give, they may tell you way

It is to be noted, that requires and guarantees contain conditions (expressions), not pascal code. So technically it is ok, that they are in the interface and not implementation.
If I want something in the implementation, I can use normal "assert" (which by the way, have often great documenting value)

The contract actually can lead to the thing you want. Though you need to complete it for *all* your code, rather than just one or two functions

If/Once you have your entire code base with contracts, you know that any method/function that does not have a "assigned" contract, can return nil (i.e. nil is a documented expected return value / or input )

So at that time, you can enable a compiler warning of the kind you wanted. It would act on all the params/results that per contract can be nil (are not forbidden to be nil).

-----
The only thing you do not have, is the undefined bit. The it can return nil, but I do not want to check it.
If you really want that, there will be a directive like {$PUSH}{$R-} .... {$POP}.

I would actually argue that it is infinitely more important to check for nil after MakeThing. (Unless I know for sure it is not needed: proof, docs)

Assuming (taking the danger of doing so) both do what there name indicates, then:
- If I forget to check after FindThing, it is likely to cause an error very soon, and most probably while I am testing myself.
- If I forget it after MakeThing, it may be very rare to cause an error. It will likely pass my tests, and cause a customer of mine some stress.

I understand all your concerns over not wanting to use this so I would suggest to avoid it (we all feel that way about certain features). Since no-one thinks this is a good idea I’ll forget about because it’s not that important anyways. Waiting for nullable types is maybe a better option anyways and those are in the pipeline via default properties.

What Sven brought up is more immediately useful anyways and I can think of many times I have the mandatory top part of the function that checks to make sure parameters are valid. Formalizing this process would be a smart thing I think. If you don’t agree, then don’t use it. ;)

That’s what I read from the link posted. As for the “ensure” block now that I don’t see as a pattern that I can recognize. Any real world examples of this?

Here’s what I imagine that would look like:

procedure CreateHero (name: string; hp: integer);
requires
  name  '';
  (hp > 0) and (hp < 100);
begin
  // compilers parses the conditions and inserts as if statements:
  // if name  ‘’ then
  //   assert(‘condition "name  ‘'" for CreateHero failed’);
  // if (hp > 0) and (hp < 100) then
  //   assert(‘condition "(hp > 0) and (hp < 100)" for CreateHero failed);
end;

Pretty simple right?

Regards,
Ryan Joseph

Almost. assert takes the condition itself

assert(condition_that_must_be_true, 'error message')

so it would be
   assert(name'', 'requires "name  ''''" failed for CreateHero');

You can already insert such asserts yourself. assert exists.

If you compile with -Sa then they are compiled in the code, otherwise not.

From the compilers perspective isn’t it faster to test the condition first in an if statement so the assert function doesn’t get called? Assert tests for false also so it needs to invert the test to “name = ‘’” which is not how the “requires” statement is worded.

I’ve always made a “fatal” function which works in the opposite direction as assert, i.e if the condition is *true* then crash. Don’t know why but that’s easier for me to understand. I like the requires section because it works in the same logic.

Regards,
Ryan Joseph

You had

requires
   name  '';

Or in other words: bail if name = ''

  assert(name'', ...)
does that.
It bails if name = ''. (then the condition "name '' " would be false => bail)

requires, like assert want the condition(s) to be true

Assert is for testing, so speed doesn't matter.
I don't know how it is internally implemented.

An Assert() might look like a mere procedure call, but it's in fact an intrinsic. If assertions are disabled ({$ASSERTIONS OFF}) then it completely does not exist and otherwise it's the check followed by a call to 'fpc_assert'.

Also an assertion "fires" if it's condition is False, so Assert() would do the same check as "requires" or "ensure" need:

requires
  Assigned(MyArg)

becomes

Assert(Assigned(MyArg), ...);

Regards,
Sven

The problem is that both "require" and "ensure" can check for fields as well (think of a setter for example or something that changes the state). Thus you could use a field name that the compiler does not yet know. So you would need to order your methods in a order dictated by the compiler due to the method's implementation instead of an order of your own choosing.

And yes, that would also be a problem with "class invariants", though there it's more clear if we say that only identifiers can be used that have been declared before as it's the same with properties and their setters/getters.

Regards,
Sven

Op 2019-04-12 om 21:11 schreef Ryan Joseph:

What do you do about the problem of not knowing when a parameter of a 3rd party function can accept null?

If it is native Pascal it is probably VAR, as Jonas said. If it is an header for an external package/API, consult their documentation.

Op 2019-04-13 om 09:47 schreef Sven Barth via fpc-pascal:

Such "inventions" are usually for specific APIs, and then the directive can be set global. Make it a new language syntax, and it turns up everywhere, also where it was not intended for.

So, yeah, we'll probably compromise towards the attribute syntax...

That's a second. Note that the stuff where this is borrowed from (like Windows headers [size_is are also attributes.

Also to be ask if properties can have contract conditions. Especially thinking of indexed properties, where the index is a constant. Several properties using the same getter/setter (with diff index) may have different conditions for the contract.

Yes accessing fields is valid. Though easily leads to confusion what should and what should not be in the contract. (Yet that is not to be enforced...)
I saw an example, where the "ensure" of a setter checked that the field was set. That may be valid, but it also may have been something for a normal assert. Depends on the intend.
The ensure (again I prefer guarantees) is a contract. (between the class/method and someone else).
So if the check for the field is an internal detail, it should be a normal assert.  (the getter result, could be part of the contract then / that is different)

Only a normal assert is not quite the same. Contracts are checked too, if an inherited overwritten method is called. Assert are not (not if there is no call to inherited).
So a contract post condition could be used to make sure that any overwritten setter method will have changed the field. In that case the partner in the contract is the inherited class. And the post condition is less of a guarantee, but more a hidden requirement to the contract partner.

Anyway about the field order.... (and there could be circular requires/guarantees)
It would be an option to specify the contract details deferred

type
   TFoo= class
      function DoTheFooThing(Foo1, Foo2: TFoo): Tbar;
      procedure DoTheBar;
// need full declaration, could be overloaded // may skip param names?,
just types should be ok
      requirements for DoTheFooThing(Foo1, Foo2: TFoo): Tbar;
            assigned(Foo1) or assigned(Foo2): 'Need at least 1 foo, for
the connection';
      guarantees for DoTheFooThing(TFoo, TFoo): Tbar;
            assigned(result);
            result.KnowsMyFoo = true: 'Connection exists';  // the =true
is redundant
   end;

Bit awkward because you need the full signature.... The compiler could allow both the "requires" right after the method, and the deferred "requirement for"

Alternative (but I do like that even less) named forward requirements
type
   TFoo= class
      function DoTheFooThing(Foo1, Foo2: TFoo): Tbar; requires
'WhatTheFooWants';
      procedure DoTheBar;
      requirements for WhatTheFooWants;
            assigned(Foo1) or assigned(Foo2): 'Need at least 1 foo, for
the connection';
   end;

That way, 2 methods could refer to the same requirements block. Then it could even be possible to refer to several blocks....      function DoTheFooThing(Foo1, Foo2: TFoo): Tbar; requires 'WhatTheFooWants', 'TheFooSpecials';

But this will make it hard to read. Especially in the reverse, if you are in a named requirement block, it is not easy to find all methods that are affected by it.

Anyway just throwing in some ideas and possibilities.

Sven Barth wrote

Am 14.04.2019 um 23:48 schrieb Martin Frb: > As for "documentation". I disagree with the way it is done in oxygen. > But I am not sure I have any good alternative. > For me a class contract (require/ensure) is part of the interface.

Putting them in the implementation as oxygen does may also cause a technical issue.
As pointed out, for a class the contract also affects inherited methods.

Unit A
interface
   TFoo= class
     procedure DoFoo(a:TSome); virtual;
  end;
implementation
uses B;
   procedure DoFoo(a:TSome);
   requires
     whatever
Unit B;
interface
uses A; // AFAIK can only read the interface, the implementation has a
circular "uses B"
   TFooSpecial= class(TFoo)
     procedure DoFoo(a:TSome); override; // needs to know implementation
details from unit A
  end;