Scanner/Tokenizer: what changed
Revision 0 to 1. Removed lines have a minus sign; added lines have a plus sign.
Title: Scanner/Tokenizer
| Old | New | Line | |
|---|---|---|---|
| 1 | + | back to contents [[FPC internals]] | |
| 2 | + | ||
| 3 | + | # Scanner/Tokenizer | |
| 4 | + | ||
| 5 | + | The scanner and tokenizer is used to construct an input stream of tokens which will be fed to the parser. It is in this stage that the preprocessing is done, that all read compiler directives change the internal state variables of the compiler, and that all illegal characters found in the input stream cause an error. | |
| 6 | + | ||
| 7 | + | Info about how macros work: [[Macro internals]] | |
| 8 | + | ||
| 9 | + | ## Architecture | |
| 10 | + | ||
| 11 | + | The general architecture of the scanner is shown in the following figure: | |
| 12 | + | ||
| 13 | + | <http://www.pjh2.de/fpc/CompilerInternalsFigure02.png> | |
| 14 | + | ||
| 15 | + | Several types can be read from the input stream, a string, handled by `readstring`, a numeric value, handled by `readnumeric`, comments , compiler and preprocessor directives. | |
| 16 | + | ||
| 17 | + | ### Input stream | |
| 18 | + | ||
| 19 | + | *(last updated for fpc version 1.0.x)* | |
| 20 | + | ||
| 21 | + | The input data is handled via the standard way of handling all the I/O in the compiler. That is to say, that it is a hook which can be overriden in **comphook.pas (do_openinputfile)**, in case where another I/O method wants to be used. | |
| 22 | + | ||
| 23 | + | The default hook uses a non-buffered dos stream contained in **files.pas** | |
| 24 | + | ||
| 25 | + | ### Preprocessor | |
| 26 | + | ||
| 27 | + | *(last updated for fpc version 1.0.x)* | |
| 28 | + | ||
| 29 | + | The scanner resolves all preprocessor directives and only gives to the parser the visible parts of the code (such as those which are included in conditional compilation). Compiler switches and directives are also saved in global variables while in the preprocessor, therefore this is part is completely independent of the parser. | |
| 30 | + | ||
| 31 | + | #### Conditional compilation (scandir.inc, scanner.pas) | |
| 32 | + | ||
| 33 | + | *(last updated for fpc version 1.0.x)* | |
| 34 | + | ||
| 35 | + | The conditional compilation is handled via a preprocessor stack, where each directive is pushed on a stack, and popped when it is resolved. The actual implementation of the stack is a linked list of preprocessor directive items. | |
| 36 | + | ||
| 37 | + | #### Compiler switches (scandir.inc, switches.pas) | |
| 38 | + | ||
| 39 | + | *(last updated for fpc version 1.0.x)* | |
| 40 | + | ||
| 41 | + | The compiler switches are handled via a lookup table which is linearly searched. Then another lookup table takes care of setting the appropriate bit flags and variables in the switches for this compilation process. | |
| 42 | + | ||
| 43 | + | ## Scanner interface | |
| 44 | + | ||
| 45 | + | *(last updated for fpc version 1.0.x)* | |
| 46 | + | ||
| 47 | + | The parser only receives tokens as its input, where a token is a enumeration which indicates the type of the token, either a reserved word, a special character, an operator, a numeric constant, string, or an identifier. | |
| 48 | + | ||
| 49 | + | Resolution of the string into a token is done via lookup which searches the string table to find the equivalent token. This search is done using a binary search algorithm through the string table. | |
| 50 | + | ||
| 51 | + | In the case of identifiers, constants (including numeric values), the value is returned in the **pattern** string variable , with the appropriate return value of the token (numeric values are also returned as non-converted strings, with any special prefix included). In the case of operators, and reserved words, only the token itself must be assumed to be preserved. The read input string is assmued to be lost. | |
| 52 | + | ||
| 53 | + | Therefore the interface with the parser is with the **readtoken()** routine and the **pattern** variable. | |
| 54 | + | ||
| 55 | + | ### Routines | |
| 56 | + | ||
| 57 | + | #### ReadToken | |
| 58 | + | ||
| 59 | + | | | | | |
| 60 | + | |----|----| | |
| 61 | + | | Declaration: | <b>`procedure`</b>` ReadToken;` | | |
| 62 | + | | Description: | Sets the global variable token to the current token read, and sets the pattern variable appropriately (if required). | | |
| 63 | + | ||
| 64 | + | ### Variables | |
| 65 | + | ||
| 66 | + | #### Token | |
| 67 | + | ||
| 68 | + | | | | | |
| 69 | + | |----|----| | |
| 70 | + | | Declaration: | <b>`var`</b>` Token: TToken;` | | |
| 71 | + | | Description: | Contains the contain token which was last read by a call to ReadToken | | |
| 72 | + | | See also: | ReadToken | | |
| 73 | + | ||
| 74 | + | #### Pattern | |
| 75 | + | ||
| 76 | + | | | | | |
| 77 | + | |----|----| | |
| 78 | + | | Declaration: | <b>`var`</b>` Pattern: String;` | | |
| 79 | + | | Description: | Contains the string of the last pattern read by a call to ReadToken | | |
| 80 | + | | See also: | ReadToken | | |
| 81 | + | ||
| 82 | + | ## Assembler parser interface | |
| 83 | + | ||
| 84 | + | *(last updated for fpc version 1.0.x)* | |
| 85 | + | ||
| 86 | + | The inline assembler parser is completely separate from the pascal parser, therefore its scanning process is also completely independent. The scanner only takes care of the preprocessor part and comments, all the rest is passed character per character to the assembler parser via the **AsmGetChar()** scanner routine. | |
| 87 | + | ||
| 88 | + | ### Routines | |
| 89 | + | ||
| 90 | + | #### AsmGetChar | |
| 91 | + | ||
| 92 | + | | | | | |
| 93 | + | |--------------|-------------------------------------------------| | |
| 94 | + | | Declaration: | <b>`function`</b>` AsmGetChar: Char;` | | |
| 95 | + | | Description: | Returns the next character in the input stream. | | |
| 96 | + | ||
| 97 | + | Next chapter: [[The parse tree]] | |
| 98 | + | ||
| 99 | + | --- | |
| 100 | + | *From the Free Pascal wiki, page "Scanner/Tokenizer", as it stood on 2020-05-24 after a change by E-ric, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).* | |
| 101 | + | Category: FPC internals |