SynEdit Highlighter: what changed

Revision 1 to 2. Removed lines have a minus sign; added lines have a plus sign.

Old	New		Line
1	1		For more info on SynEdit go to: [[SynEdit]]\
2	2		Also see [[SynEdit Markup]]
3	3
4	4		------------------------------------------------------------------------
5	5
6	6		# Understanding the SynEdit Highlighter
7	7
8	8		## SynEdit - Highlighter relationship
9	9
10	10		SynEdit to Highlighter have an N to 1 relationship.
11	11
12	12		- One instance of a Highlighter can serve N (many) instances of SynEdits
13	13		- Each SynEdit only has one Highlighter
14	14		- But: one text (text-buffer) can have many highlighters, if shared by several SynEdit (each SynEdit will have one HL, but all HL will work on the same document)
15	15
16	16		As a result of this:
17	17
18	18		- no Highlighter Instance has a (fixed) reference to the SynEdit.
19	19
20	20
21	21		(Highlighters however keep a list of SynEditTextBuffers to which they are attached)
22	22
23	23		- All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").
24	24
25	25		However, before each call to the Highlighter, SynEdit ensures that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed. The Format of the data storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines).
26	26
27	27		## Scanning and Returning Highlight attributes
28	28
29	29		The Highlighter is expected to work on a per Line base.
30	30
31	31		If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.
32	32
33	33		If Highlight attributes are required, SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false.
34	34
35	35		Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is: whenever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).
36	36
37	37		Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.
38	38
39	39		## Folding
40	40
41	41		SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighters that implement folding are to be based on TSynCustomFoldHighlighter.
42	42
43	43		The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):
44	44
45	45		- FoldLevel at the end of line
46	46		- Minimum FoldLevel encountered anywhere on the line
47	47
48	48		The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends:
49	49
50	50		` EndLvl MinLvl`\
51	51		` Procedure a; 1 - 0`\
52	52		` Begin 2 -- 1 -`\
53	53		` b:= 1; 2 -- 2 --`\
54	54		` if c > b then begin 3 --- 2 --`\
55	55		` c:=b; 3 --- 3 ---`\
56	56		` end else begin 3 --- 2 --`\
57	57		` b:=c; 3 --- 3 ---`\
58	58		` end; 2 -- 2 --`\
59	59		` end; 0 0 // The end closes both: begin and procedure fold`
60	60
61	61		In the line
62	62
63	63		` Procedure a; 1 - 0`
64	64
65	65		the MinLvl is 0, because the line started with a Level of 0 (and it never went down / no folds closed). Similar in all lines where there is only an opening fold keyword ("begin").
66	66
67	67		But the line
68	68
69	69		` end else begin 3 --- 2 --`
70	70
71	71		starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.
72	72
73	73		Without the MinLvl it would not be possible to tell that a fold ends in this line.
74	74
75	75		There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.
76	76
77	77		` if a then begin b:=1; c:=2; end; // no fold on that line`
78	78
79	79		# Creating a SynEdit Highlighter
80	80
81	81		Since 0.9.31 Revision 35115 the fold-highlighter has changed. Implementing basic folding is now easier.
82	82
83	83		All Sources can be found in the Lazarus installation directory under:
84	84
85	85		:* Lazarus 2.3 and up: examples\Components\SynEdit\NewHighlighterTutorial
86	86
87	87		:* Lazarus before 2.2.*: examples\SynEdit\NewHighlighterTutorial\\
88	88
89	89		The project HighlighterTutorial contains 3 difference example highlighters:
90	90
91	91		:* SimpleHl: used in Step 1 below
92	92
93	93		:* ContextHl: used in Step 2 below
94	94
95	95		:* FoldHl: used in Step 3 below
96	96
97	97		SimpleHl and ContextHl will work with 0.9.30 too
98	98
99	99		The folding highlighter in action:
100	100
101		-	~~<figure>~~
102		-	~~<img src="SynEditFoldingHighlighterDemo.png" title="SynEditFoldingHighlighterDemo.png" />~~
103		-	~~<figcaption>SynEditFoldingHighlighterDemo.png</figcaption>~~
104		-	~~</figure>~~
	101	+	![](/wiki-images/a/a2/SynEditFoldingHighlighterDemo.png)
105	102
106	103		## The Basics: Returning Tokens and Attributes
107	104
108	105		As indicated, the SimpleHl unit demonstrates this process.
109	106
110	107		### What it does
111	108
112	109		- It splits each line into words and spaces (or tabs)
113	110		- The spaces are part of the text, and must be highlighted too
114	111		- This example allows specifying different colors for
115	112
116	113
117	114
118	115		\- text (defaults to not-highlighted)
119	116
120	117		\- spaces (defaults to silver frame)
121	118
122	119		\- words, separated by spaces, that start with a,e,i,o,u (defaults to bold)
123	120
124	121		\- the word "not" (defaults to red background)
125	122
126	123		### How it works
127	124
128	125		- Creation
129	126
130	127		The Highlighter creates Attributes that it can return the Words and Spaces.
131	128
132	129		- SetLine
133	130
134	131		Is called by SynEdit before a line gets painted (or before highlight info is needed)
135	132
136	133		- GetTokenEx, GetTokenAttribute, Next, GetEol
137	134
138	135		Are used by SynEdit to iterate over the Line. Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.
139	136
140	137		[[Important:]] The tokens returned for each line must represent the original line-text, and be returned in the correct order.
141	138
142	139		- GetToken, GetTokenPos, GetTokenKind
143	140
144	141		SynEdit uses them e.g for finding matching brackets. If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g. if there was a string attribute, brackets outside a string would not match brackets inside a string).
145	142
146	143		### Other notes
147	144
148	145		For readability, the highlighter has no optimization, so it may be very slow on larger texts. Many of the supplied highlighters use hash functions, to find what word (or any group of chars) is.
149	146
150	147		## Step 2: Using Ranges
151	148
152	149		As indicated, the ContextHl unit demonstrates this process
153	150
154	151		The next example allows content of a line that influences other lines that follow. An example: a "(" in Pascal makes all following lines a comment until a ")" is found.
155	152
156	153		This example extends the SimpleHl show above: The tokens -- and ++ (must be surrounded by space or line-begin/end to be a token of their own) will toggle words that start with a,e,i,o,u
157	154
158	155		Multiple ++ and -- can be nested. Then for each -- a ++ must be given, before the words highlight again.
159	156
160	157		Then we extend the scanner. The pre-scan to store the information calls the same functions as the highlighter. It is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)
161	158
162	159		The current amount of "--" is counted in
163	160
164	161		``` pascal
165	162		FCurRange: Integer;
166	163		```
167	164
168	165		The amount is decreased by "++"
169	166
170	167		To store the info we use:
171	168
172	169		GetRange
173	170		Called after a line is completely scanned, to get the value at the end of the line. The value will be stored.
174	171
175	172		SetRange
176	173		Called before a line gets scanned. Sets the value stored from the end of the previous line.
177	174
178	175		ResetRange
179	176		Called before the 1st line is scanned (as there is no previous line).
180	177
181	178		> Note: A scan is triggered by every change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had. See: <http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139420.html#msg139420>
182	179
183	180		### Important note on Ranges
184	181
185	182		The purpose of the range is to allow the HL to start scanning in any line. The HL will not never need to look at a previous line. Any information needed to scan the current line can be derived from the ranges value.
186	183
187	184		Example:
188	185
189	186		``` pascal
190	187		writeln; (*
191	188		readln;
192	189		*)
193	190		```
194	191
195	192		when scanning the "readln" the HL knows from the range that it is in a comment, it does not need to look back at previous lines.
196	193
197	194		Therefore scanning can start at any line.
198	195
199	196		This also explains the note in the previous chapter. "until a line returns the same range that it already had". For even if the text of a line was not changed, if the value of the range at the lines start changed, then the scan result will change too.
200	197
201	198		## Step 3: Add Folding
202	199
203	200		As indicated, the FoldHl unit demonstrates this process
204	201
205	202		For the example, the highlighter should fold everything between free-standing "-(-", "-)-".
206	203
207	204		Change inheritance:
208	205
209	206		``` pascal
210	207		uses SynEditHighlighterFoldBase;
211	208		...
212	209		TSynDemoHl = class(TSynCustomFoldHighlighter)
213	210		```
214	211
215	212		Change the way range info is stored, since the base class uses it for fold-info:
216	213
217	214		``` pascal
218	215		procedure TSynDemoHl.SetRange(Value: Pointer);
219	216		begin
220	217		inherited;
221	218		FCurRange := PtrInt(CodeFoldRange.RangeType);
222	219		end;
223	220
224	221		procedure TSynDemoHl.ResetRange;
225	222		begin
226	223		inherited;
227	224		FCurRange := 0;
228	225		end;
229	226
230	227		function TSynDemoHl.GetRange: Pointer;
231	228		begin
232	229		CodeFoldRange.RangeType := Pointer(PtrInt(FCurRange));
233	230		inherited;
234	231		end;
235	232		```
236	233
237	234		Now add code to the scanner which tells the highlighter about opening and closing folds:
238	235
239	236		``` pascal
240	237		procedure TSynDemoHl.FindTokenEnd;
241	238		begin
242	239		...
243	240
244	241		if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '[') then
245	242		StartCodeFoldBlock(nil);
246	243		if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ']') then
247	244		EndCodeFoldBlock();
248	245		end;
249	246		```
250	247
251	248		- For 0.9.30
252	249
253	250		Please see the history of this page, if you are using a 0.9.30 version [[http://wiki.lazarus.freepascal.org/index.php?title=SynEdit Highlighter&oldid=76020]]
254	251
255	252		#### More info on StartCodeFoldBlock / EndCodeFoldBlock
256	253
257	254		``` pascal
258	255		function StartCodeFoldBlock(ABlockType: Pointer; IncreaseLevel: Boolean = true): TSynCustomCodeFoldBlock; virtual;
259	256		```
260	257
261	258		ABlockType
262	259		Can be used to specify an ID for the block.\
263	260
264	261		The field is not normally used as a pointer (though this is permitted). Normally IDs are a numeric enumeration.\
265	262		If you have different types of block (e.g. in Pascal: begin/end; repeat/until, ...), and you do not want a block being closed by the wrong keyword ("end" should not close "repeat"), then you can give them IDs:\
266		-	~~<b>StartCodeFoldBlock(PtrUInt(1))</b> // or other numbers\~~
	263	+	StartCodeFoldBlock(PtrUInt(1)) // or other numbers\
267	264
268	265		IncreaseLevel
269	266		If set to False, then a block will be inserted that can not be folded.\
270	267		Blocks like that can be used for internal tracking.
271	268
272	269		NOTE: All folds must be nested, they can not overlap. That is, the last opened fold must be closed first.\
273	270		This refers to the "EndLvl" as shown in "Folding"(1.3) above\
274	271		Overlaps (like IFDEF and begin in the IDE) can not be done this way.
275	272
276	273		``` pascal
277	274		procedure EndCodeFoldBlock(DecreaseLevel: Boolean = True); virtual;
278	275		```
279	276
280	277		DecreaseLevel
281	278		This must match IncreaseLevel, as it was given on the StartCodeFoldBlock\
282	279
283	280		True means a fold ends; False means an internal block is ended. If mismatched then folds will either continue, or end early.\
284	281		TopCodeFoldBlockType can be used to indicate the ID of the innermost open block. One can use different IDs for internal blocks, and use this to set the value.
285	282
286	283		``` pascal
287	284		function TopCodeFoldBlockType(DownIndex: Integer = 0): Pointer;
288	285		```
289	286
290	287		Returns the ID of the innermost block.
291	288
292	289		DownIndex
293	290		can be used to get ID for the other block.
294	291
295	292		DownIndex=1 means the block that is surrounding the innermost.
296	293
297	294		# Configurable Highlighters (incl. 3rd party)
298	295
299	296		## SynAnySyn
300	297
301	298		A real simple Highlighter. More a reference implementation, than a real life useable Highlighter.
302	299
303	300		## SynUniHighlighter
304	301
305	302		Uses XML-based configurations files. [old official link](http://web.archive.org/web/20090610040212/http://www.delphist.com:80/UniHighlighter.html)
306	303
307	304		## SynFacilSyn
308	305
309	306		Flexible fully configurable Highlighter. <https://github.com/t-edson/SynFacilSyn>
310	307
311	308		# References
312	309
313	310		Threads on the forum:
314	311
315	312		- "Making highligher for C syntaxis" [[http://www.lazarus.freepascal.org/index.php/topic,10260.0.html topic,10260]]
316	313		- "Creating a Syntax Highlighter" [[http://www.lazarus.freepascal.org/index.php/topic,10959.msg54714 topic,10959.msg54714]]
317	314		- obtaining highlight for printing [[http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html\|http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html#msg57160 11384.msg57160]]
318	315		- Some details on ranges, and using objects to store more info [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139354.html\|http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139354.html#msg139354 topic,21727.msg139354]] and [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139419.html\|http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139419.html#msg139419 topic,21727.msg139419]]
319	316		- More on ranges, working copy vs immutable, foldblocks... [[http://forum.lazarus.freepascal.org/index.php/topic,30685.msg195464.html\|http://forum.lazarus.freepascal.org/index.php/topic,30685.msg195464.html#msg195464 http://forum.lazarus.freepascal.org/index.php/topic,30685]]
320	317		- Markup (Highlight) all occurrences of a token (or regex) [[http://forum.lazarus.freepascal.org/index.php?topic=26833 topic=26833]]
321	318		- Markup and foldinfo: [[http://forum.lazarus.freepascal.org/index.php?topic=30122]]
322	319		- How to do invalidation in TSynEditMarkup [[http://forum.lazarus.freepascal.org/index.php/topic,30122.msg207931.html\|http://forum.lazarus.freepascal.org/index.php/topic,30122.msg207931.html#msg207931]]
323	320
324	321		- Folding
325	322		- "SynEdit - improved highlighters to handle Code Folding?" [[http://www.lazarus.freepascal.org/index.php/topic,7879.0.html topic,7879]]
326	323		- "SynEdit - Add support code folding for Java" [[http://www.lazarus.freepascal.org/index.php/topic,7338.0.html topic,7338]]
327	324		- "CodeFolding" Config [[http://www.lazarus.freepascal.org/index.php/topic,11064 topic,11064]]
328	325		- Fold blocks "end" keyword (end on the line before the next keyword) [[http://forum.lazarus.freepascal.org/index.php/topic,23411.msg139621.html\|http://forum.lazarus.freepascal.org/index.php/topic,23411.msg139621.html#msg139621 topic,23411.msg139621]]
329	326		- Folding selected text from code (user/application code): [[http://forum.lazarus.freepascal.org/index.php/topic,24473.msg147312.html\|http://forum.lazarus.freepascal.org/index.php/topic,24473.msg147312.html#msg147312 24473.msg147312]]
330	327		- Obtaining state of folding (save fold state with session) [[http://forum.lazarus.freepascal.org/index.php?topic=26748 topic=26748]]
331	328
332	329		---
333	330		From the Free Pascal wiki, page "SynEdit Highlighter", as it stood on 2022-05-04 after a change by Martin, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).
334	331		Category: SynEdit

Back to history