Hi,

Consider the following program (compiled using revision 26807 on OSX):

=======
Program QWordToExtended;
Begin
	WriteLn(Sizeof(Extended));
	WriteLn(Sizeof(QWord));
	WriteLn(Sizeof(Longword));
	WriteLn(Extended(QWord($FFFFFFFFFFFFFFFF)));	//1
	WriteLn(Extended($FFFFFFFFFFFFFFFF));			//2
	WriteLn(Extended(Longword($FFFFFFFF)));			//3
	WriteLn(Extended($FFFFFFFF));					//4
End.
=======

Here I would expect four rather large values on standard output. The thing is that the second test gives me -1. I suspect this is due to the compiler interpreting the constant as an Int64; this is confirmed by the first test. Since I was curious as to the intended-ness of this behaviour I wrote the second set of tests which show me that this is not true for longwords: both 3 & 4 give the same result. This lead me to the conclusion that this is a bug. Shall I file it?

Some important note, I think: the first sizeof's of the example return 10, 8, 4 on this machine.

http://bugs.freepascal.org/view.php?id=21284

Since I was curious as to the intended-ness of this behaviour I wrote the second set of tests which show me that this is not true for longwords: both 3 & 4 give the same result. This lead me to the conclusion that this is a bug. Shall I file it?

All hexadecimal constants are (conceptually) parsed as int64, so this is by design. int64($00000000ffffffff) is not -1.

Jonas

So all numeric constants that are not floats are parsed as Int64's?

Isn't that view about numeric constants a bit limited (why an Int64 for example, you could've picked a virtual Int256 just as well)? Especially if you have a data type that can contain the number in it's original intention? Delphi compatibility I read in the bug report you mentioned, and I understand that in mode delphi (see below though for a bit of `issues`), but the example program is in mode fpc (or how is it called?). Can that at least be called a bug (in the documentation at the very least)?

By the way, what do you do when you want to port fpc to a one's comlement machine (if they still exist)? Is $FFFF FFFF FFFF FFFF equal to 0 then? And when you have a CPU that has a native integer size of 128 bit, how do you do the transformation then? Just truncate the constant to a 64 bit wide integer? Admitted, it are rare cases...

They are parsed as the smallest integer type that can represent them. Because of the (unPascalish) decision to have an unsigned version of the largest supported integer type, there are indeed some cases that require decisions to define the behaviour and regardless of what decision you make, some people won't like it.

Isn't that view about numeric constants a bit limited (why an Int64 for example, you could've picked a virtual Int256 just as well)?

Supporting larger constant values in the compiler than what we support in the language itself would be very counter-intuitive.

Especially if you have a data type that can contain the number in it's original intention?

Hexadecimal numbers have no "intention" as far as signedness is concerned. How they are interpreted is up to the language definition.

Delphi compatibility I read in the bug report you mentioned, and I understand that in mode delphi (see below though for a bit of `issues`), but the example program is in mode fpc (or how is it called?). Can that at least be called a bug (in the documentation at the very least)?

No, unless the documentation states that the behaviour is different for mode fpc. The behaviour, in both FPC and Delphi modes, is by design.

By the way, what do you do when you want to port fpc to a one's comlement machine (if they still exist)? Is $FFFF FFFF FFFF FFFF equal to 0 then?

The internal representation of a machine is unrelated to how values in the source code are interpreted. Just like 'a' in an ASCII source file will always mean the character 'a', even if that source file is compiled for a machine that uses EBDIC. Numbers in FPC source files will always be interpreted as two's complement.

And when you have a CPU that has a native integer size of 128 bit, how do you do the transformation then? Just truncate the constant to a 64 bit wide integer? Admitted, it are rare cases...

If FPC is ever ported to such a machine and if a built-in integer type of 128 bit would then be added for constants (the second would not automatically follow from the first), then the behaviour /might/ change. Just like it did from Turbo Pascal to Delphi ($fffffffff is interpreted as -1 in TP, but as high(cardinal) in Delphi). Such a change would probably be done based on a modeswitch, rather than by default in all syntax modes.

Jonas

Ha, but an Int64 cannot represent this number. The maximum postive range of an Int64 is 2**63 - 1. The constant I used was 2**64 - 1. It doesn't fit. So this brings us to another issue: the constant cannot be represented. Yet no error or even a warning is raised.

Because of the (unPascalish) decision to have an unsigned version of the largest supported integer type, there are indeed some cases that require decisions to define the behaviour

That is perfectly true. But shouldn't the most basic behaviour of a language be at the very least intuitive? There is an unsigned type that can hold this constant, yet it is not used except if you use a typecast.

and regardless of what decision you make, some people won't like it.

I don't believe that anyone ever expected the $FXXX XXXX XXXX XXXX to become less than zero (X denotes anything from 0 to F).

Its not about `liking` a decision. When we're talking about multiple inheritance, interfaces, strings, generics, we are talking about liking IMHO: everyone has his/her own coding style/habits/preferences. I have yet to meet someone who likes $FF.....FFFF to be -1.

> Isn't that view about numeric constants a bit limited (why an Int64 for example, you could've picked a virtual Int256 just as well)? Supporting larger constant values in the compiler than what we support in the language itself would be very counter-intuitive.

In a way yes, in another way it allows the compiler to be able to just interpret the number and see where it gets. The thing is that you wouldn't get this weird behaviour at all since this only happens when the constant is about as large as the maximum internal compiler representable constant. Increase this size and there would be no issue. Take, for example the number -1. This could be represented with: $1 FFFF FFFF FFFF FFFFF (take a word length of 65 bit)

Then take the other constant $FFFF FFFF FFFF FFFF, this could be represented as $0 FFFF FFFF FFFF FFFF

No data is lost in this process.

This mimics the exact behaviour of 32 constants. The exact datatype size can afterwards be resolved by removing either all 0's or all 1's from the left hand side of this representation. If you removed zero's, pick an unsigned datatype. Extend to the right size (1,2,4,8 for time being) with 0's. If you removed one's, it was signed. You thus need one extra bit. Pick the right signed datatype and fill with 1's on the left hand side.

The only numbers that cannot be represented are -$FXXX XXXX XXXX XXXX. This could be fixed by going to a 66 bit internal datatype, but is would be pointless since there is no datatype that can represent this constant.

Really, all I'm asking is one extra bit ;-)

> Especially if you have a data type that can contain the number in it's original intention? Hexadecimal numbers have no "intention" as far as signedness is concerned.

That's debatable (but, as you mention, classified as part of the language definition), but a constant has a certain magnitude. It is IMO this magnitude that is the point. I don't expect a number with a lot of digits (2**64 - 1) to become rather small in magnitude (1). That is what I mean by `intention`. With this I haven't said anything about signedness: I don't care if 762 is interpreted as a signed 32 bit integer or unsigned 16 bit integer, as long as the meaning remains.

If I don't want this `intention`, when performing `evil bit level hacks` (algorithms like the famous 0x5f3759df algorithm), I use a typecast to instruct the compiler how to store it.

I was looking at the wrong page (I was looking as `Numbers` instead of `Ordinal Types`). My mistake.

> By the way, what do you do when you want to port fpc to a one's comlement machine (if they still exist)? Is $FFFF FFFF FFFF FFFF equal to 0 then? The internal representation of a machine is unrelated to how values in the source code are interpreted. Just like 'a' in an ASCII source file will always mean the character 'a', even if that source file is compiled for a machine that uses EBDIC. Numbers in FPC source files will always be interpreted as two's complement.

Good point.

> And when you have a CPU that has a native integer size of 128 bit, how do you do the transformation then? Just truncate the constant to a 64 bit wide integer? Admitted, it are rare cases... If FPC is ever ported to such a machine and if a built-in integer type of 128 bit would then be added for constants (the second would not automatically follow from the first), then the behaviour /might/ change. Just like it did from Turbo Pascal to Delphi ($fffffffff is interpreted as -1 in TP, but as high(cardinal) in Delphi). Such a change would probably be done based on a modeswitch, rather than by default in all syntax modes.

Here you have it: it might change. The thing I say/argue is that if the compiler would interpret the constant as it is written in code:
- A minus (`-`) before it means negative, nothing before it means positive
- A lot of digits = A large number in magnitude
- The larger the most significant digit of an equally long constant (#digits), the larger it is in magnitude

..., then the compiler doesn't need a change, it simply needs to be notified of the new datatype.

The subject of this thread violates all three intuitive interpretations [intuitive to my judgement -- this is how I learned numbers in basic school].

If the compiler doesn't find a fitting datatype for the constant, it should stop compiling and give an error IMHO - this is what it does, as you know, when you enter `$1 FFFF FFFF FFFF FFFF` as a constant. Actually, until today, I always thought it did just that.

Numerical constants, where the sign matters, should only be encoded in decimal. The other formats (hex,oct,bin...) are intended for use with binary values, where the bit pattern is important. Then the code compiles correctly on any kind of machine.

Assumptions about type sizes and encodings can make *application* code unportable. E.g. the Extended type doesn't have a guaranteed size and binary representation, IIRC it's equivalent to Double on x64.

DoDi

Hans-Peter Diettrich wrote

Numerical constants, where the sign matters, should only be encoded in decimal. The other formats (hex,oct,bin...) are intended for use with binary values, where the bit pattern is important. Then the code compiles correctly on any kind of machine.

Yeah, the only problem is that there are few calculators lying around here that can actually calculate the decimal value of $FFFF FFFF FFFF FFFF.

But you are right. If I use 18446744073709551615 instead of $FFFF FFFF FFFF FFFF, it appears to be right. This is again what I meant with unintuitive behaviour: 18446744073709551615 should be equal to $FFFF FFFF FFFF FFFF IMHO. Anyway, I've got this feeling that I won't be able to convince any compiler developer to fix this, so I will leave 'll fix my precompiler instead to handle this scenario for me.

Assumptions about type sizes and encodings can make *application* code unportable. E.g. the Extended type doesn't have a guaranteed size and binary representation,

That is true. But I don't make any assumptions about sizes of types. I use the name QWord because it happens to be largest possible unsigned integer in fpc, not because it is 8 bytes long. That is also why I started my example program with the 3 sizeof statements: to remove assumptions about datasizes.

IIRC it's equivalent to Double on x64.

Nowp, Double is 8 bytes long whereas Extended is 10 bytes long on this machine. Then again it doesn't matter here. Replace `Extended` in the entire thread with `Double` and you still get the same bahaviour.

It should be well-defined and consistent. One of Pascal's basic principles is that the native signed integer type is used for pretty much everything. While we (and Borland/CodeGear/Embarcadero) have extended the language beyond that (e.g. we support int64 on 32 bit platforms, and there's also the qword type), this principle is still followed as much as possible. In this case: constants are always preferably interpreted as signed numbers.

> Hexadecimal numbers have no "intention" as far as signedness is concerned. That's debatable (but, as you mention, classified as part of the language definition), but a constant has a certain magnitude. It is IMO this magnitude that is the point. I don't expect a number with a lot of digits (2**64 - 1) to become rather small in magnitude (1). That is what I mean by `intention`. With this I haven't said anything about signedness: I don't care if 762 is interpreted as a signed 32 bit integer or unsigned 16 bit integer, as long as the meaning remains.

You do care about the signedness, because the only way to represent int64(-1) in hexadecimal is as $ffffffffffffffff. Numbers in two's complement do no consist of a single sign bit followed by a magnitude. Those top 63 '1' bits together form the "-" sign in this number.

> If FPC is ever ported to such a machine and if a built-in integer type of 128 bit would then be added for constants (the second would not automatically follow from the first), then the behaviour /might/ change. Just like it did from Turbo Pascal to Delphi ($fffffffff is interpreted as -1 in TP, but as high(cardinal) in Delphi). Such a change would probably be done based on a modeswitch, rather than by default in all syntax modes. Here you have it: it might change.

Yes, indeed. I agree that's not ideal, but both the historical precedent (TP -> Delphi), current Delphi compatibility (it interprets hex numbers in the same way as FPC) and the general Pascal tradition (preferring signed over unsigned types) offset any hypothetical future inconvenience as far as I'm concerned.

If the compiler doesn't find a fitting datatype for the constant, it should stop compiling and give an error IMHO

As mentioned before, for the compiler $ffffffffffffffff fits perfectly into an int64 because it parses it using val(str,int64var,value).

Jonas

Talking about principles:
If hexadecimal is actually used to represent bit patterns (as Hans-Peter Diettrich wrote), then the decision to use a signed type here seems to violate this (represent bitpatterns) principle, since the highest bit in a signed number has a different meaning than the other bits, where in a bitpattern all bits have equal meaning.

It seems like sticking to one principle (signed integer as much as possible) actually breaks another principle (bitpattern).

You do care about the signedness, because the only way to represent int64(-1) in hexadecimal is as $ffffffffffffffff.

And what about -$1? Or is that too far fetched?

Numbers in two's complement do no consist of a single sign bit followed by a magnitude. Those top 63 '1' bits together form the "-" sign in this number.

Yes, but this can all be solved by parsing the string and storing it with one extra MSBit (if there is a `-` in front of the constant it must be negative, otherwise it should be positive). This highest bit then reflects the sign. `-1` would then be $1 FFFF FFFF FFFF FFFF, whereas $FFFF FFFF FFFF FFFF would be $0 FFFF FFFF FFFF FFFF. It really is quite easy to store it like that and `fix` things [picking a fitting datatype] afterwards. Anyway, then you have got backwards compatibility to take care of, since there will be someone out there who's code actually depends on this behaviour.

As mentioned before, for the compiler $ffffffffffffffff fits perfectly into an int64 because it parses it using val(str,int64var,value).

That is circular reasoning.

Hans-Peter Diettrich is not a particular authority on FPC. Additionally, at least in Pascal hexadecimal numbers are simply another way to write numbers, next to decimal, binary and octal. In all cases, eventually bit pattern has to be interpreted, and this is what gives a meaning to the bits. There is no "raw bits without meaning" type in Pascal in general or in FPC in particular.

In the end, it all still boils down to the fact that you do not consider interpreting hexadecimal constants as int64 is appropriate/logical/good, while I think the opposite. While in theory a modeswitch could be added, requiring people to first inspect all possible mode switches that are in effect before they can know what a particular code fragment means is bad (or similarly: not being able to know what a code fragment presented in isolation means is very bad). Modeswitches that change the meaning of code as opposed to enabling/disabling certain features are very bad and should be limited to the absolute minimum.

And yes, this means that when a bad decision has been made in the past in a language's design, it is extremely hard to change it later on. That's why discussions about and implementations of new features often take so long, because once they're "out there", certain aspects are more or less cast in stone.

Jonas

That's correct.

It seems like sticking to one principle (signed integer as much as possible) actually breaks another principle (bitpattern).

Wirth and his Pascal language are well designed with signed types above all, and unsigned types being subranges. In so far one could consider hex constants with the sign bit set as syntactical errors.

> You do care about the signedness, because the only way to represent > int64(-1) in hexadecimal is as $ffffffffffffffff.

Negative numbers never should be expressed in hex.

And what about -$1? Or is that too far fetched?

That's correct, because -$1 is -1 is a valid integral expression, without signedness problems.

This is why Wirth considers all types being signed, without such problems.

This highest bit then reflects the sign.

The sign representation is machine specific, as you know. On 1's complement machines there exist two representation of zero, as +0 and -0, and you cannot express both as hexadecimal constants in an portable way. That's why high level languages, like Pascal, forbid hex representations of (possibly) negative numerical values.

`-1` would then be $1 FFFF FFFF FFFF FFFF, whereas $FFFF FFFF FFFF FFFF would be $0 FFFF FFFF FFFF FFFF. It really is quite easy to store it like that and `fix` things [picking a fitting datatype] afterwards.

The datatype has to be constructed/determined first, and *if* there exists a type with more than 64 bits, then it will be a signed type, with a 65 bit (unsigned) subrange matching your needs. But if no such type exists, you are lost.

Anyway, then you have got backwards compatibility to take care of, since there will be someone out there who's code actually depends on this behaviour.

When we agree that a bitpattern of $FFFF FFFF FFFF FFFF can be interperted differently on different 32 bit machines, as -1 or -MaxInt, then it's obvious that such a textual representation should cause an compilation error "not portable...". We know that such an error message has not yet been implemented, but if you insist on writing unportable code... :-]

DoDi

Well, a warning or something like that (note, hint?) would be welcome indeed. Something like `Your constant will probably not be interpreted like you expect it due to [....]`.

>> You do care about the signedness, because the only way to represent >> int64(-1) in hexadecimal is as $ffffffffffffffff. Negative numbers never should be expressed in hex.

Agreed, if one wants a negative number, one should put a `-` in front of it.

> And what about -$1? Or is that too far fetched? That's correct, because -$1 is -1 is a valid integral expression, without signedness problems.

Yeah, that is what I thought as well...

> This highest bit then reflects the sign. The sign representation is machine specific, as you know. On 1's complement machines there exist two representation of zero, as +0 and -0, and you cannot express both as hexadecimal constants in an portable way.

Yes, but Jonas wrote that:

`The internal representation of a machine is unrelated to how values in the source code are interpreted. Just like 'a' in an ASCII source file will always mean the character 'a', even if that source file is compiled for a machine that uses EBDIC. Numbers in FPC source files will always be interpreted as two's complement.`

This means that the sign representation must be portable because it is always 2's complement. I suppose that the compiler will convert the representation of the sign to the appropriate notation for the machine/architecture it is compiling to.

So, hence my 2's complement notation for a _potential_ type.

That's why high level languages, like Pascal, forbid hex representations of (possibly) negative numerical values.

That indeed would be a solution, but freepascal doesn't forbid it. Actually I am a bit opposed to forbidding stuff, so a simple warning of some sort would do just aswell for me.

Yes, that is true, but there always is a 64 signed/unsigned type (perhaps not native). On machines where, for example, only 32 bit wide datatypes are allowed, the virtual subrange should be 33 instead of 65 bytes.

Anyway, that is the way how I parse constants. The important rule here is that you don't need the full 65 bit in the final representation. The signedness of the type can fix this loss of the one bit.

Why `on 32 bit machines`? I'm fairly confident that this particular constant on this particular compiler version will generate the same outcome on every possible architecture out there (just change the `extended` to `single` in the original example, because extended tends to vary).

then it's obvious that such a textual representation should cause an compilation error "not portable...". We know that such an error message has not yet been implemented, but if you insist on writing unportable code... :-]

I insist on using a constant that is:
- 64 bit wide
- Only contains 1's
- Is interpreted as an unsigned number wherever mathematical operations are performed.

Those demands are quite portable, no?

My original problem was easily solved with a typecast QWord(<gargantuan constant goes here>), so that was no longer an issue. What baffled me though was the fact that this (mis-: in my opinion) mis-parsing of certain constants is by design.

Subranges are expressed in low..hi notation, not in bits, meaning that the hi value must be expressable in a valid signed positive number.

Anyway, that is the way how I parse constants. The important rule here is that you don't need the full 65 bit in the final representation. The signedness of the type can fix this loss of the one bit.

How (which data type) does *your* parser store untyped numerical constants?

IMO your problem arises from the fact that a bitpattern, with the highest of 64 bits set, cannot be stored in a larger (signed) type, as required. All such untyped constants will cause problems when assigned to typed variables. Test yourself what happens when you convert such an QWORD value into Extended.

You're right, my guess of the number of bits was wrong.

That's my expectation, too.

Then you have to choose a different language. What will C++, C# or Java do in these cases?

Those demands are quite portable, no?

No.

My original problem was easily solved with a typecast QWord(<gargantuan constant goes here>), so that was no longer an issue. What baffled me though was the fact that this (mis-: in my opinion) mis-parsing of certain constants is by design.

What you observed was related to the argument passed to WriteLn. When WriteLn includes code to output a QWord, then the output should reflect the bitpattern (unsigned number). The output of an Extended value reflects the value converted from integral to floating point, and that conversion assumes signed values. IIRC the x87 FPU doesn't have an instruction to load unsigned integral values, so that no compiler has a chance to make it load an unsigned value.

DoDi

That is why is use the word `virtual`. It is of course a more complex datatype than simply a subrange. See below.

Roughly like this (syntax may be a bit awry, but you get the point):

TIntegerNumber = Record
Case SignedNess: TSignedNess of
	snPositive:	UValue: QWord;
	snNegative:  SValue: Int64;
End;

The parser detects wether there is a `-` in front of the constant and stores the right sign in the SignedNess field.

IMO your problem arises from the fact that a bitpattern, with the highest of 64 bits set, cannot be stored in a larger (signed) type, as required.

Yes, spot on :-)

All such untyped constants will cause problems when assigned to typed variables. Test yourself what happens when you convert such an QWORD value into Extended.

That was the first example of this thread. The effect is the QWord becomes the mantissa of the extended, the exponent is set right and the sign bit is set to zero. But I believe the compiler must do this for me since (as you mention at the bottom of this quote) there is no instruction to load an unsigned integer into the 8087.

Java I'll ignore, my C# is way too rusty and C/C++ won't parse the constant as is, because it overflows the normal 32 bit range. Here you have to type something like 0xffffffffffffffffLLU. I haven't tried casting the bugger into a float though.

> Those demands are quite portable, no? No.

Alright, let me rephrase my demands:
- I want to store the value 18446744073709551615 in any kind of variable without loss of precision.

Both set of demands are fine by me. Is that a more portable demand? ;-)

Alright, that I did not know. Thanks for the info. I always assumed there was an instruction FULD, but it isn't there apparently.

Nevertheless, Hans-Peter's statement is wrong. FPC can perfectly convert any qword value into an extended value. It's not because something cannot be done using a single instruction, that it cannot be

A parser doesn't work like that - too many possible cases with unary minus. If you need an datatype for integers with more bits than provided by the compiler, you must roll your own datatype.

Alright, let me rephrase my demands: - I want to store the value 18446744073709551615 in any kind of variable without loss of precision.

See above.

DoDi

cannot

be done using a single instruction, that it cannot be done at all.

Yes, that is also why I wrote `I guess freepascal must have done this for me`. Nonetheless it is an interesting revelation :-)

Perhaps yours doesn't, I didn't say what kind of syntax `my parser` parses... Quite frankly I don't care how my parser, your parser or fpc's parser work in detail, as long as something gets parsed. Hey, that's the purpose of a parser after all ;-)

If you need an datatype for integers with more bits than provided by the compiler, you must roll your own datatype.

Of coure. But nobody stipulated that I can't use native pascal types (like qword, int64, byte, ...) for storing this.