Hello

you wrote

> By debugging the compiler and checking the value that the token > variable gets after reading the token.

a> You're very lucky. I tried to debug it with IDE from 1.0.4 and it a> didn't work (see http://www.freepascal.org/bugs/showrec.php3?ID=1515), a> and then I took the IDE from 1.0.5 and I overwrote it in my current a> 1.0.4 installation and I didn't work neither.

About debugging... I live under Linux 2.2.38 and use RHGDB to debug FPC (RHIDE have to many bugs and can not be used in real life). RHGDB do not work very nice�too, but it is useable (I use patch for gdbpas to rebuild RHGDB).

Regards,
Sergey Korshunoff

PS: It is possible to find a remote programming work? Currently I work in Moscow as street cleaner for $10 per week. It is possible to find here a work as system administrator or programmer for 100$ per week, but it is not easy to find a work in Linux or TurboVision proramming.

And it would be very nice if I can found some work to port DOS TV C++ programs to Linux (or Windows, or OS/2). Or to write a new Linux text mode programs in TV style.

This work can help me to release more useable RHGDB. May be I write to SuSE or RedHat and try ask about this work. Or try to get a work in ASP Linux team.

PPS: About HEAP manager in FPC...
I think I found a solution how to speed up my version of heap.inc This version do merging of free blocks only in GetMem and have speed problems because it merge all blocks (and small blocks too) before asking new block from OS. Now I try to remove merging small blocks and would do this only if a used by FPC memory is more
than a value of some FPC variable (which would be in Linux by default equal to physical memory, this value can be taken as size of /proc/kcore or be readed from /proc/meminfo).

Well, as most of you perhaps know I am trying to translate freepascal to spanish. Well, I have many problems with tokens.pas file.

And the boards, aren't as much as I wanted, so I came to this emaillist.

The one that is in pp\source\compiler

What I try is to recompile the compiler, with a changed tokens.pas file

I've noticed that in tokens.pas there's a procedure that orders the normal words, which are treated by this procedure because they are not special (boolean special is false). But it needs the token str to be ordered as they are... you know first by length and when the same length, you order it alphabetacally.

I've seen many files that treat tokens defined in tokens.pas, that means that PERHAPS they have to be ordered in tokens.pas BUT I'M NOT SURE.

And I'm also not sure of these files being the unique to access tokens.pas and need something ordered.

htypechk.pas THIS ONE I'M QUITE SURE IT DOESN'T GO TO TOKENS.PAS AND BOTHER ABOUT ORDER BUT... PERHAPS IT IS

tcadd.pas THIS ONE I'M QUITE SURE IT DOESN'T GO TO TOKENS.PAS AND BOTHER ABOUT ORDER BUT... PERHAPS IT IS

In pexpr.pas I found this:

  const
     { Warning these stay be ordered !! }
     operator_levels:array[Toperator_precedence] of set of Ttoken=
        ([_LT,_LTE,_GT,_GTE,_EQUAL,_UNEQUAL,_OP_IN,_OP_IS],
         [_PLUS,_MINUS,_OP_OR,_OP_XOR],
         [_CARET,_SYMDIF,_STARSTAR,_STAR,_SLASH,
          _OP_AS,_OP_AND,_OP_DIV,_OP_MOD,_OP_SHL,_OP_SHR],
         [_STARSTAR] );

in line 2027 if you want to check.

WHAT DOES MEAN "Warning these stay be ordered "?
They have to be ordered as they are in tokens.pas
like the original pexpr.pas or what?

psub.pas
in line 425 I have this:

if not(token in [_PLUS..last_overloaded]) then
  Message(parser_e_overload_operator_failed);
optoken:=token;
consume(Token);

What means that goes from _PLUS (I think it is because it is supposed to be always the first token )

and to last_overloaded, which is defined somewhere else:

in symtable.pas that has in line 404:

 first_overloaded = _PLUS;
 last_overloaded  = _ASSIGNMENT;

this is because these are supposed to be first and the last operators.

Well,... this is not very good programming, i think first_overloaded and last_overloaded should be defined in tokens.pas with an integer const (myconst) which says how many operators that can be overloaded do we have.

Something like first token is first_overloaded and myconst-th token is last_overloaded.

In this way we will able to change the tokens order easier, but now that I think, normal programmers don't do it usually.

Well...

I know that tokens.pas NORMAL WORDS have to be ordered but...

In fact the question is what about operators? How must them be ordered? Like they are in the original version? With the same order pattern than Normal words? Or what?

And another thing about the procedure that orders (well ok, in fact it creates an index) I don't know if it works ok.

With this tokens.pas (www.geocities.com/beatlesnap/tokens.pas) "Repite" token (I change repit token and i put repite) is not detected as a known identifier.

Is it that I don't order it ok, or it's a bug?

Thank you for all, "repite" not known as a token is driving me mad.

Adrian15.

P.S.: If you are curious and want to know why I'm not using Y,O, instead of YY and OO, the answer is that my intuition tells me that you cannot have tokens of one character in
freepascal.

---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

The order in should be by length and then by alphabet. The only thing you need to take care off is to change the order accordingly in the enum declaration. And also all operators should be after eachother so that they can be checked by a range.

The predence order. The order of the tokens itself doesn't matter here as it's a set.

This is a thing that has been forgotten to update, the constants can easily be moved to tokens.pas and fixed in the psub unit

And another thing about the procedure that orders (well ok, in fact it creates an index) I don't know if it works ok.

It only determines at which placed in the array a word with length X and starting with letter Y can be found so it reduces the place where it needs to search for the word.

Here it's working correct. Did you delete the .ppu files first and then recompiled the compiler?

Here it's working correct. Did you delete the .ppu files first and then

recompiled the

compiler?

Yes, I always do that not to have problems. I think... repite error I haven't it any more but... there some parts of the code (My new rtl) that doesn't understand AND token, and I don't know why, does it matter the operators order in token const declared in tokens.pas?

If it does, Why ASSIGNMENT is the last token if it has only length 2.

Can I declare a TOKEN of length 1 such as "y" or "o"?

I've seen that the assambler words are defined in other file something as att386.inc (I dont remember very well the name). Is this the only place where to change the "assambler tokens".

Is there any place (I think I've seen it but I'm not sure) where there are defined some assembler tokens in order to assemble the file generated by this program with an external assembler? I want to know it not to change it.

===================
I don't know... how have you managed to know if it detected "repite" that's to say.

The first thing I do is change tokens.pas and recompile the compiler, and then I try to compile a very easy file (writing hello in screen) but it doesn't compile it automatically, it needs system unit, and many others in inc and i386. And so... these files have to have the new tokens, that's to say,

I have to put unidad instead of unit. Programa instead of program and this with a lot of files.

You seem to have made it easier. How have done it? Because myself I had to made a program (www.geocities.com/beatlesnap/pashis.zip) in order to translate the basic rtl into my new tokens rtl.

If you have a better way of compiling the compiler without losing interpretation of my new tokens, PLEASE TELL ME. Perhaps I've been wasting my time.

Take your time. I dont know your self, but my June exams are very near.

Thanks for all,
adrian15.

---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

Can I declare a TOKEN of length 1 such as "y" or "o"?

Currently, the scanner knows that there are no single chars that are tokens :

See this text extracted from
tscannerfile.readtoken (line 1227 of current CVS state)

          { keyword or any other known token,
            pattern is always uppercased }
            if (pattern[1]'_') and (length(pattern) in [2..tokenidlen]) then
             begin
               low:=ord(tokenidx^[length(pattern),pattern[1]].first);
               high:=ord(tokenidx^[length(pattern),pattern[1]].last);

If you have one char tokens, you need to change the [2..tokenidlen] into a [1..tokenidlen].

The following lines also explain why you need to have alphabetically ordered tokens.

The order of the operators is not used for searching so the length and alphabetic ordering is not necessary. I don't know if any other order is required, i guess not, i don't have the time to check this out.

Can I declare a TOKEN of length 1 such as "y" or "o"?

It should be possible if you change the length check in scanner in the function readtoken.

By debugging the compiler and checking the value that the token variable gets after reading the token.

Rename the innr.inc to compinnr.inc and fix the compiler accordingly to include that file instead of innr.inc. This was already done in the 1.0.5 development compiler.

<< Can you tell me in which file you have found it? I'm browsing CVS in "http://www.freepascal.org/cgi-bin/cvsweb/fpc/compiler/" I've searched tscannerfile, or readtoken or state (there weren't such files) and in tokens.pas, in the last version, I didn't find (only in tokens.pas) that the procedure/function you have told me about exists.

<< Another question in tokens.pas just before const arraytokeninfo:

ttokenidx=array[2..tokenidlen,'A'..'Z'] of tokenidxrec;

Do I have to change 2 and put a 1, to admit tokens of only 1 char? And what is ttokenidx because it's like a matrix of tokenidxrec which is a record with first and last which are tokens.

It's like a chart or table, in which square there are two tokens, quite strange, could you please explain it to me?

<<Why tokenideln value is: 14? Any special reason?

<< Well, I've noticed tokens.pas declares tokens, but the only important procedure is create_tokenidx, Which only process NORMAL WORDS cause their special boolean value is false. And nothing more.

So could you tell me please where the other tokens(operators mainly) are proccesed (in which other files) to make an index or whatever?

---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

> Can I declare a TOKEN of length 1 such as "y" or "o"? It should be possible if you change the length check in scanner in the

function readtoken.
<< Sorry, for the previous message I didn't look for scanner file in CVS enough. Now, I'm seeing this file exists.
And I've studied it I have to change 2 to 1 in:

tscannerfile.readtoken procedure
and in
is_keyword function

And I've found InitScanner procedure which I supposed is the one that proccess such things
as "defined parametres", i.e. {$ifdef TP}.
Am I right? Is this the last thing about InitScanner correct?

how have you managed to know if it detected "repite" that's to

> say. By debugging the compiler and checking the value that the token variable gets

after

reading the token.

<< You're very lucky. I tried to debug it with IDE from 1.0.4 and it didn't work (see http://www.freepascal.org/bugs/showrec.php3?ID=1515 ), and then I took the IDE from 1.0.5 and I overwrote it in my current 1.0.4 installation and I didn't work neither.

> > The first thing I do is change tokens.pas and recompile the compiler, and

then

with

> a lot of files. > > You seem to have made it easier. How have done it? Because myself I had to > made a program (www.geocities.com/beatlesnap/pashis.zip) in order to

translate

> the basic rtl into my new tokens rtl. Rename the innr.inc to compinnr.inc and fix the compiler accordingly to

include that file

instead of innr.inc. This was already done in the 1.0.5 development compiler.

<< I think you haven't understood me, I don't find anything in this file (http://www.freepascal.org/cgi-bin/cvsweb/fpc/rtl/inc/innr.inc?rev=1.3&content- type=text/x-cvsweb-markup), to make the compiler understand an rtl with old tokens as they were new ones.

And I've been searching compinnr.inc in (http://www.freepascal.org/cgi- bin/cvsweb/projects/lazarus/) and I haven't found it, although I must confess I haven't searched in all the folders. (Perhaps should I find it at http://lazarus.freepascal.org snapshot?)

Thanks for all
again.

---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/

In scanner.pas at line 1241.

Yes, you also need to adjust this table

It's like a chart or table, in which square there are two tokens, quite strange, could you please explain it to me?

It's to limit the binary search. When a token is read from the source the length of the token and the first character of the token are used to get the lowest and highest value from this table. These values are then used to limit the binary search in the token list.

<<Why tokenideln value is: 14? Any special reason?

Just to limit, the largest english keyword is 14 characters, if longer keywords are available it'll be increased.

They are assigned in scanner.pas when parsing things like a bracket or plus character. And for tokens like 'in' they are assigned using the op field from the token array.

That is from the 1.1 branch. The 1.0.x branch doesn't have this function. But the function adds the directives like {$I} and {$ifdef}, not the symbols, those are mostly added from options.pas (search for def_symbol).

I'm at work and used plain gdbpas.exe to debug. But using the latest go32v2 1.0.5 IDEs should also be possible under Win9X. Under WinNT/2K there can be problems as those have strange Dos support.

Sorry, it was innrcomp.inc

Maybe downloading fpc10.zip from the pub/fpc/source/ dir is better suited for you. Then you've all the latest files.

Btw. Translating the assembler tokens is dangerous as they are also used for writing external assembler files. If you translate them you can only use the internal assembler.