SynEdit Highlighter: what changed

Revision 2 to 3. Removed lines have a minus sign; added lines have a plus sign.

Old	New		Line
1	1		For more info on SynEdit go to: [[SynEdit]]\
2	2		Also see [[SynEdit Markup]]
3	3
4	4		------------------------------------------------------------------------
5	5
6	6		# Understanding the SynEdit Highlighter
7	7
8	8		## SynEdit - Highlighter relationship
9	9
10	10		SynEdit to Highlighter have an N to 1 relationship.
11	11
12	12		- One instance of a Highlighter can serve N (many) instances of SynEdits
13	13		- Each SynEdit only has one Highlighter
14	14		- But: one text (text-buffer) can have many highlighters, if shared by several SynEdit (each SynEdit will have one HL, but all HL will work on the same document)
15	15
16	16		As a result of this:
17	17
18	18		- no Highlighter Instance has a (fixed) reference to the SynEdit.
19	19
20	20
21	21		(Highlighters however keep a list of SynEditTextBuffers to which they are attached)
22	22
23	23		- All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").
24	24
25	25		However, before each call to the Highlighter, SynEdit ensures that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed. The Format of the data storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines).
26	26
27	27		## Scanning and Returning Highlight attributes
28	28
29	29		The Highlighter is expected to work on a per Line base.
30	30
31	31		If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.
32	32
33	33		If Highlight attributes are required, SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false.
34	34
35	35		Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is: whenever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).
36	36
37	37		Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.
38	38
39	39		## Folding
40	40
41	41		SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighters that implement folding are to be based on TSynCustomFoldHighlighter.
42	42
43	43		The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):
44	44
45	45		- FoldLevel at the end of line
46	46		- Minimum FoldLevel encountered anywhere on the line
47	47
48	48		The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends:
49	49
50		-	~~` EndLvl MinLvl`\~~
51		-	~~` Procedure a; 1 - 0`\~~
52		-	~~` Begin 2 -- 1 -`\~~
53		-	~~` b:= 1; 2 -- 2 --`\~~
54		-	~~` if c > b then begin 3 --- 2 --`\~~
55		-	~~` c:=b; 3 --- 3 ---`\~~
56		-	~~` end else begin 3 --- 2 --`\~~
57		-	~~` b:=c; 3 --- 3 ---`\~~
58		-	~~` end; 2 -- 2 --`\~~
59		-	~~` end; 0 0 // The end closes both: begin and procedure fold`~~
	50	+	``` pascal
	51	+	EndLvl MinLvl
	52	+	Procedure a; 1 - 0
	53	+	Begin 2 -- 1 -
	54	+	b:= 1; 2 -- 2 --
	55	+	if c > b then begin 3 --- 2 --
	56	+	c:=b; 3 --- 3 ---
	57	+	end else begin 3 --- 2 --
	58	+	b:=c; 3 --- 3 ---
	59	+	end; 2 -- 2 --
	60	+	end; 0 0 // The end closes both: begin and procedure fold
	61	+	```
60	62
61	63		In the line
62	64
63		-	~~` Procedure a; 1 - 0`~~
	65	+	``` pascal
	66	+	Procedure a; 1 - 0
	67	+	```
64	68
65	69		the MinLvl is 0, because the line started with a Level of 0 (and it never went down / no folds closed). Similar in all lines where there is only an opening fold keyword ("begin").
66	70
67	71		But the line
68	72
69		-	~~` end else begin 3 --- 2 --`~~
	73	+	``` pascal
	74	+	end else begin 3 --- 2 --
	75	+	```
70	76
71	77		starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.
72	78
73	79		Without the MinLvl it would not be possible to tell that a fold ends in this line.
74	80
75	81		There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.
76	82
77		-	~~` if a then begin b:=1; c:=2; end; // no fold on that line`~~
	83	+	``` pascal
	84	+	if a then begin b:=1; c:=2; end; // no fold on that line
	85	+	```
78	86
79	87		# Creating a SynEdit Highlighter
80	88
81	89		Since 0.9.31 Revision 35115 the fold-highlighter has changed. Implementing basic folding is now easier.
82	90
83	91		All Sources can be found in the Lazarus installation directory under:
84	92
85	93		:* Lazarus 2.3 and up: examples\Components\SynEdit\NewHighlighterTutorial
86	94
87	95		:* Lazarus before 2.2.*: examples\SynEdit\NewHighlighterTutorial\\
88	96
89	97		The project HighlighterTutorial contains 3 difference example highlighters:
90	98
91	99		:* SimpleHl: used in Step 1 below
92	100
93	101		:* ContextHl: used in Step 2 below
94	102
95	103		:* FoldHl: used in Step 3 below
96	104
97	105		SimpleHl and ContextHl will work with 0.9.30 too
98	106
99	107		The folding highlighter in action:
100	108
101	109		![](/wiki-images/a/a2/SynEditFoldingHighlighterDemo.png)
102	110
103	111		## The Basics: Returning Tokens and Attributes
104	112
105	113		As indicated, the SimpleHl unit demonstrates this process.
106	114
107	115		### What it does
108	116
109	117		- It splits each line into words and spaces (or tabs)
110	118		- The spaces are part of the text, and must be highlighted too
111	119		- This example allows specifying different colors for
112	120
113	121
114	122
115	123		\- text (defaults to not-highlighted)
116	124
117	125		\- spaces (defaults to silver frame)
118	126
119	127		\- words, separated by spaces, that start with a,e,i,o,u (defaults to bold)
120	128
121	129		\- the word "not" (defaults to red background)
122	130
123	131		### How it works
124	132
125	133		- Creation
126	134
127	135		The Highlighter creates Attributes that it can return the Words and Spaces.
128	136
129	137		- SetLine
130	138
131	139		Is called by SynEdit before a line gets painted (or before highlight info is needed)
132	140
133	141		- GetTokenEx, GetTokenAttribute, Next, GetEol
134	142
135	143		Are used by SynEdit to iterate over the Line. Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.
136	144
137	145		[[Important:]] The tokens returned for each line must represent the original line-text, and be returned in the correct order.
138	146
139	147		- GetToken, GetTokenPos, GetTokenKind
140	148
141	149		SynEdit uses them e.g for finding matching brackets. If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g. if there was a string attribute, brackets outside a string would not match brackets inside a string).
142	150
143	151		### Other notes
144	152
145	153		For readability, the highlighter has no optimization, so it may be very slow on larger texts. Many of the supplied highlighters use hash functions, to find what word (or any group of chars) is.
146	154
147	155		## Step 2: Using Ranges
148	156
149	157		As indicated, the ContextHl unit demonstrates this process
150	158
151	159		The next example allows content of a line that influences other lines that follow. An example: a "(" in Pascal makes all following lines a comment until a ")" is found.
152	160
153	161		This example extends the SimpleHl show above: The tokens -- and ++ (must be surrounded by space or line-begin/end to be a token of their own) will toggle words that start with a,e,i,o,u
154	162
155	163		Multiple ++ and -- can be nested. Then for each -- a ++ must be given, before the words highlight again.
156	164
157	165		Then we extend the scanner. The pre-scan to store the information calls the same functions as the highlighter. It is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)
158	166
159	167		The current amount of "--" is counted in
160	168
161	169		``` pascal
162	170		FCurRange: Integer;
163	171		```
164	172
165	173		The amount is decreased by "++"
166	174
167	175		To store the info we use:
168	176
169	177		GetRange
170	178		Called after a line is completely scanned, to get the value at the end of the line. The value will be stored.
171	179
172	180		SetRange
173	181		Called before a line gets scanned. Sets the value stored from the end of the previous line.
174	182
175	183		ResetRange
176	184		Called before the 1st line is scanned (as there is no previous line).
177	185
178	186		> Note: A scan is triggered by every change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had. See: <http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139420.html#msg139420>
179	187
180	188		### Important note on Ranges
181	189
182	190		The purpose of the range is to allow the HL to start scanning in any line. The HL will not never need to look at a previous line. Any information needed to scan the current line can be derived from the ranges value.
183	191
184	192		Example:
185	193
186	194		``` pascal
187	195		writeln; (*
188	196		readln;
189	197		*)
190	198		```
191	199
192	200		when scanning the "readln" the HL knows from the range that it is in a comment, it does not need to look back at previous lines.
193	201
194	202		Therefore scanning can start at any line.
195	203
196	204		This also explains the note in the previous chapter. "until a line returns the same range that it already had". For even if the text of a line was not changed, if the value of the range at the lines start changed, then the scan result will change too.
197	205
198	206		## Step 3: Add Folding
199	207
200	208		As indicated, the FoldHl unit demonstrates this process
201	209
202	210		For the example, the highlighter should fold everything between free-standing "-(-", "-)-".
203	211
204	212		Change inheritance:
205	213
206	214		``` pascal
207	215		uses SynEditHighlighterFoldBase;
208	216		...
209	217		TSynDemoHl = class(TSynCustomFoldHighlighter)
210	218		```
211	219
212	220		Change the way range info is stored, since the base class uses it for fold-info:
213	221
214	222		``` pascal
215	223		procedure TSynDemoHl.SetRange(Value: Pointer);
216	224		begin
217	225		inherited;
218	226		FCurRange := PtrInt(CodeFoldRange.RangeType);
219	227		end;
220	228
221	229		procedure TSynDemoHl.ResetRange;
222	230		begin
223	231		inherited;
224	232		FCurRange := 0;
225	233		end;
226	234
227	235		function TSynDemoHl.GetRange: Pointer;
228	236		begin
229	237		CodeFoldRange.RangeType := Pointer(PtrInt(FCurRange));
230	238		inherited;
231	239		end;
232	240		```
233	241
234	242		Now add code to the scanner which tells the highlighter about opening and closing folds:
235	243
236	244		``` pascal
237	245		procedure TSynDemoHl.FindTokenEnd;
238	246		begin
239	247		...
240	248
241	249		if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '[') then
242	250		StartCodeFoldBlock(nil);
243	251		if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ']') then
244	252		EndCodeFoldBlock();
245	253		end;
246	254		```
247	255
248	256		- For 0.9.30
249	257
250	258		Please see the history of this page, if you are using a 0.9.30 version [[http://wiki.lazarus.freepascal.org/index.php?title=SynEdit Highlighter&oldid=76020]]
251	259
252	260		#### More info on StartCodeFoldBlock / EndCodeFoldBlock
253	261
254	262		``` pascal
255	263		function StartCodeFoldBlock(ABlockType: Pointer; IncreaseLevel: Boolean = true): TSynCustomCodeFoldBlock; virtual;
256	264		```
257	265
258	266		ABlockType
259	267		Can be used to specify an ID for the block.\
260	268
261	269		The field is not normally used as a pointer (though this is permitted). Normally IDs are a numeric enumeration.\
262	270		If you have different types of block (e.g. in Pascal: begin/end; repeat/until, ...), and you do not want a block being closed by the wrong keyword ("end" should not close "repeat"), then you can give them IDs:\
263	271		StartCodeFoldBlock(PtrUInt(1)) // or other numbers\
264	272
265	273		IncreaseLevel
266	274		If set to False, then a block will be inserted that can not be folded.\
267	275		Blocks like that can be used for internal tracking.
268	276
269	277		NOTE: All folds must be nested, they can not overlap. That is, the last opened fold must be closed first.\
270	278		This refers to the "EndLvl" as shown in "Folding"(1.3) above\
271	279		Overlaps (like IFDEF and begin in the IDE) can not be done this way.
272	280
273	281		``` pascal
274	282		procedure EndCodeFoldBlock(DecreaseLevel: Boolean = True); virtual;
275	283		```
276	284
277	285		DecreaseLevel
278	286		This must match IncreaseLevel, as it was given on the StartCodeFoldBlock\
279	287
280	288		True means a fold ends; False means an internal block is ended. If mismatched then folds will either continue, or end early.\
281	289		TopCodeFoldBlockType can be used to indicate the ID of the innermost open block. One can use different IDs for internal blocks, and use this to set the value.
282	290
283	291		``` pascal
284	292		function TopCodeFoldBlockType(DownIndex: Integer = 0): Pointer;
285	293		```
286	294
287	295		Returns the ID of the innermost block.
288	296
289	297		DownIndex
290	298		can be used to get ID for the other block.
291	299
292	300		DownIndex=1 means the block that is surrounding the innermost.
293	301
294	302		# Configurable Highlighters (incl. 3rd party)
295	303
296	304		## SynAnySyn
297	305
298	306		A real simple Highlighter. More a reference implementation, than a real life useable Highlighter.
299	307
300	308		## SynUniHighlighter
301	309
302	310		Uses XML-based configurations files. [old official link](http://web.archive.org/web/20090610040212/http://www.delphist.com:80/UniHighlighter.html)
303	311
304	312		## SynFacilSyn
305	313
306	314		Flexible fully configurable Highlighter. <https://github.com/t-edson/SynFacilSyn>
307	315
308	316		# References
309	317
310	318		Threads on the forum:
311	319
312	320		- "Making highligher for C syntaxis" [[http://www.lazarus.freepascal.org/index.php/topic,10260.0.html topic,10260]]
313	321		- "Creating a Syntax Highlighter" [[http://www.lazarus.freepascal.org/index.php/topic,10959.msg54714 topic,10959.msg54714]]
314	322		- obtaining highlight for printing [[http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html\|http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html#msg57160 11384.msg57160]]
315	323		- Some details on ranges, and using objects to store more info [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139354.html\|http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139354.html#msg139354 topic,21727.msg139354]] and [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139419.html\|http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139419.html#msg139419 topic,21727.msg139419]]
316	324		- More on ranges, working copy vs immutable, foldblocks... [[http://forum.lazarus.freepascal.org/index.php/topic,30685.msg195464.html\|http://forum.lazarus.freepascal.org/index.php/topic,30685.msg195464.html#msg195464 http://forum.lazarus.freepascal.org/index.php/topic,30685]]
317	325		- Markup (Highlight) all occurrences of a token (or regex) [[http://forum.lazarus.freepascal.org/index.php?topic=26833 topic=26833]]
318	326		- Markup and foldinfo: [[http://forum.lazarus.freepascal.org/index.php?topic=30122]]
319	327		- How to do invalidation in TSynEditMarkup [[http://forum.lazarus.freepascal.org/index.php/topic,30122.msg207931.html\|http://forum.lazarus.freepascal.org/index.php/topic,30122.msg207931.html#msg207931]]
320	328
321	329		- Folding
322	330		- "SynEdit - improved highlighters to handle Code Folding?" [[http://www.lazarus.freepascal.org/index.php/topic,7879.0.html topic,7879]]
323	331		- "SynEdit - Add support code folding for Java" [[http://www.lazarus.freepascal.org/index.php/topic,7338.0.html topic,7338]]
324	332		- "CodeFolding" Config [[http://www.lazarus.freepascal.org/index.php/topic,11064 topic,11064]]
325	333		- Fold blocks "end" keyword (end on the line before the next keyword) [[http://forum.lazarus.freepascal.org/index.php/topic,23411.msg139621.html\|http://forum.lazarus.freepascal.org/index.php/topic,23411.msg139621.html#msg139621 topic,23411.msg139621]]
326	334		- Folding selected text from code (user/application code): [[http://forum.lazarus.freepascal.org/index.php/topic,24473.msg147312.html\|http://forum.lazarus.freepascal.org/index.php/topic,24473.msg147312.html#msg147312 24473.msg147312]]
327	335		- Obtaining state of folding (save fold state with session) [[http://forum.lazarus.freepascal.org/index.php?topic=26748 topic=26748]]
328	336
329	337		---
330	338		From the Free Pascal wiki, page "SynEdit Highlighter", as it stood on 2022-05-04 after a change by Martin, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).
331	339		Category: SynEdit

Back to history