Textmode IDE development: what changed

Revision 1 to 2. Removed lines have a minus sign; added lines have a plus sign.

Old	New		Line
1	1		See also [[Textmode IDE]] for more user-oriented hints, and [[go32v2 development]] for hints about the particular dos problems of the IDE.
2	2
3	3		## General problems
4	4
5	5		While the obvious task for the textmode IDE is of course fixing several or some of the 80 bugs, some other stuff comes to mind as future projects. Nearly all have to do with the fact that the IDE effectively stopped evolving significantly since 1.0 times. A lot of problems and ugliness lies in the TP/DOS legacy from this period:
6	6
7	7		- the fact that it uses Dos, rather than Sysutils for its filename operations.
8	8		- now that the compiler uses sysutils mostly, the DOS versions are the less tested ones.
9	9		- the fact that shortstrings (or parts of them) are used for filenames.
10	10		- under windows there is no support for unc-paths
11	11
12	12		- Even the shortstring filename handling is far from perfect, since the IDE clearly hasn't had the end-user testing for *nix targets that dos/win32 had. Grepping for fixfilename() might yield such problems (like e.g. bug \#5614), where filenames are pretty printing, however effectively killing case sensitive filesystems. Bringing this up to ansistring/sysutils level might also help to unify these routines between textmode IDE and Lazarus.
13	13
14	14		- Some functionality is deprecated or not active or documented. Examples are browser and code folding (see comments of bug \#6696 Some of the functionality mentioned in \#11583 might also be "newer"). Note that this kind of functionality is often not finished, e.g. afaik there is no way to save folds. (make them persistent some how). Also the editor components seem to have incremental syntaxhighlighting under ifdef.
15	15
16	16		- Another major problem is that the docs have significantly grown. On a PII-233 (128MB and a more recent 20GB disc) it took 20-40 minutes to install the HTML files from the help. Nowadays we have chm support in the FPC tree, maybe it is time to kill all the old helpfile formats and standarize on CHM for a while. (removing the need to have separate files HTML is I think more important than killing e.g. .tph) Initial CHM support added to trunk See the separate section about this topic.
17	17
18	18		- On *nix, terminals are swiftly adopting UTF-8 as output type. Supporting this, even very minimally (say ISO8859-1+linedrawing) will be a quite complex task. Required FV rewrite is [[Free Vision\|done]] as of February, 2022.
19	19
20	20		## Compatibility
21	21
22	22		In the past the Textmode IDE was compatible with lots of compilers and platforms that are dead. This compatibility has not been maintained/tested for close to a decade, so it is doubtful it works at all. So IMHO it can be removed, and dependencies on things like sysutils are no longer forbidden.
23	23
24	24		## ansistringification
25	25
26	26		Since FV has reduced in meaning, maybe it can be modernised using the delphi model a bit (streaming and constructors/destructors). I know this is heresy, but a tpobj version could continue to exist. It is mainly for the IDE. I also don't know if it is worthwhile at all (most notably if FV exploits the static nature of objects...).
27	27
28	28		Maybe a good first approximation would be to try to introduce ansistring everywhere. (Florian has said he has tried this in the past, but that he got stuck. I did an attempt and also got stuck.)
29	29
30	30		I did a real minor attempt to kill some redundant string routines, but are stuck with the fact that one can't import sysutils into a major TV or IDE because that will cause incompatibilities due to the pstring type. (which is defined both by TV as by sysutils). This is probably the reason why wutils wraps so many sysutils routines, together with the fact that cross-unit overloading seems a bit ambiguous if the relevant parameter types are convertible to eachother (e.g. shortstring and ansistring, or char) It would be major FV re-engineering, and a significant loss of backwards compatibility to fix this. )
31	31
32	32		It could be easily remedied by reworking TV/IDE to use a "pshortstring" everywhere, which is still compatible with pstring for backwards compatibility as long as sysutils isn't added.
33	33
34	34		A second stage would replace all path related strings by ansistrings.
35	35
36	36		For this it is also interesting to look at the C++ version, since there some features only more recently introduced to pascal are added (operator overloading, generics). However a new codebase doesn't have to strictly follow C++, since FPC has certain features that are useful (open arrays, arrays of const, dyn arrays) too.
37	37
38	38		note: it seems that (static) TP objects are not initialized/finalized. Using ansistrings with them is thus hazardous. Heap TP objects seem to be initialized/finalized. Fixed by Sergei r16632,16635
39	39
40	40		## Chm Support
41	41
42	42		The [[chm]] support was mainly meant to remedy the defects of the loose html based help; mostly install time, HD size (fragmentation, uncompressed) the fact that indexing must be done on the fly, and the constant little problems with filenames in generated HTML files.
43	43
44	44		Extract the loose HTML help on a Athlon XP2000 takes 8 minutes, and creating the index another 5. CHM reduces this to effectively zero, and adds the full text search capability. On the P-II 233MHz that I occasionally use for DOS testing, times can be over an hour on plain dos.
45	45
46	46		The CHM support has been in the IDE since 2.2.4, and most seems to be working. Sometimes the html generation from the latex documents has problems, but strictly speaking that is not a chm problem
47	47
48	48		The LaTeX docs used to be a problem, but recently there has been progress in this department, see [[chm backend for fpdoc\|Chm backend for fpdoc, latex part.]] Other improvements to be done:
49	49
50	50		- Links over CHM file boundaries still have to be fixed, this is needed for an effective main page. I already have created a simple main page that works in Windows. (and can generate it) (done an initial implementation)
51	51		- installation into the IDE by the installer. (done but not properly installed yet)
52	52		- Index majeur?
53	53		- index for latex pages. (done for ref archive from ref.kwd)
54	54		- preparing the main index takes several seconds on a XP 2000+, specially with LCL.chm installed. Might be annoying on slow computers. Cache? It might also be that the index collection, suffers from the inserted sorting dimensioning problem. (tcollections and tlist are both one big array of pointers. (Ordered) Inserting in such an large array requires a lot of fields to move to make room )
55	55		- there are problems with labels. E.g. from the main TOCs. For now they are stripped, but maybe first labels should be tried, and only then revert to main page.
56	56		- The ref/user/prog chms are in UTF-8 which is not nice for the textmode IDE. (fixed in +/- 2.4.0, there might be detail bugs left though)
57	57
58	58		### CHM boundaries problem
59	59
60	60		This problem is partial to a deeper problem in the textmode IDE. The way a help page/topic is identified in the textmode IDE is by two (16-bit?) integers. One is the fileid, the other is the topicid in the file.
61	61
62	62		HTML and CHM's however work string based. IOW a string for file, and a string for the topic (combination of path/file#label). This is solved by allocating a fileid for each chm dynamically onload, and similarly for the topic files. The html subsystem maintains a links-in-current topic cache, and the indexes into that cache act as topicids. (in other wpords a bit dynamic, not all URLs in the chm have one).
63	63
64	64		However since helpfiles are loaded on a need-to basis, and one might encounter a link to a.chm/path/file in another chm before a.chm is loaded, making it hard to resolve the IDs. This is currently solved by having a registry of CHM files as a global. However that totally bypasses the central helpsystem.
65	65
66	66		The plain HTML help didn't suffer from this because they cheated: all HTML must be installed in one directory, and links are fixed in the HTML itself. (refer to hardcoded paths ../prog/) So the entire HTML became one big helpfile in practice. Which is fine as a hack, but not really nice or reproducible for other formats.
67	67
68	68		A minor rewrite that changes the identification to something else (a record with both string and topic id's?) is a possibility, but for that I must first dig up examples of all helpfile types supported and test how they work.
69	69
70	70		## Files of the IDE
71	71
72	72		Here a short list of the units in the IDE, and their contents. A rough size in kb is added to indicate the relative size of the unit.
73	73
74	74		\| Unit \| Size \| Remark \|
75	75		\|----\|----\|----\|
76	76		\| fp \| 14k \| The main program for the IDE. Parameter processing and a lot (400 lines) of initialization. \|
77	77		\| fpcalc \| 14k \| Calculator object for the IDE \|
78	78		\| fpcatch \| 6k \| IDE exception handling. \|
79	79		\| fpcodcmp \| 20k \| Code Complete routines \|
80	80		\| fpcodtmp \| 18k \| Code Template routines \|
81	81		\| fpcompil \| 38k \| Compiler call routines for the IDE \|
82	82		\| fpconst \| 17k \| Constants used by the IDE \|
83	83		\| fpcygwin \| 2k \| Cygwin helpers (versioning, dll name) \|
84	84		\| fpdebug \| 103k \| Debugger call routines for the IDE \|
85	85		\| fpdesk \| 29k \| Desktop loading/saving routines \|
86	86		\| fpdpansi \| 3k \| Unit to export current screen buffer to an ansi file (screendump) \|
87	87		\| fpevalw \| 2k \| Debug expression evaluator dialog ? \|
88	88		\| fphelp \| 24k \| Help routines for the IDE \|
89	89		\| fpide \| 72k \| Main IDEApp object \|
90	90		\| fpini \| 24k \| Write/Read Options to INI File \|
91	91		\| fpintf \| 8k \| Misc routines for the IDE. compiling/running related mostly. \|
92	92		\| fpkeys \| 8k \| Learn keys routines for the IDE \|
93	93		\| fpredir \| 19k \| Unit to redirect output and error to files \|
94	94		\| fpregs \| 50k \| (cpu)Register debug routines for the IDE \|
95	95		\| fpswitch \| 49k \| Compiler switches routines for the IDE \|
96	96		\| fpsymbol \| 58k \| Symbol browse support routines for the IDE \|
97	97		\| fptemplt \| 7k \| Template support routines for the IDE \|
98	98		\| fptools \| 47k \| Tool (menu) support for the IDE \|
99	99		\| fpusrscr \| 41k \| User screen support routines \|
100	100		\| fputils \| 11k \| Utilility routines used by the IDE. Path/file/string related mostly. \|
101	101		\| fpvars \| 5k \| Global variables for the IDE \|
102	102		\| fpviews \| 131k \| Views and view-related functions for the IDE \|
103	103		\| pmode \| 6k \| Support routines for DPMI programs (dos support) \|
104	104		\| vesa \| 20k \| VESA (graphmode switching) support routines \|
105	105		\| wansi \| 28k \| ANSI support \|
106	106		\| wcedit \| 58k \| Code editor template objects \|
107	107		\| wchmhwrap \| 5k \| Chm wrapper. CHM reader parts that deal with classes, to obtain a clean TV - Delphi model separation \|
108	108		\| wconsole \| 2k \| Contains some routines that save/restore console state when executing built programs. \|
109	109		\| wconsts \| 0k \| Strings for common utilities. A handful of numeric consts, and a big includefile with textual constants (localisation?) \|
110	110		\| weditor \| 208k \| Code editor template objects \|
111	111		\| whelp \| 27k \| Help support & Borland OA .HLP reader objects and routines (the latter are moved to woahelp?) \|
112	112		\| whlpview \| 41k \| Help display objects. The base widgets for the help system (?) \|
113	113		\| whtml \| 30k \| base object definitions for html help? \|
114	114		\| whtmlhlp \| 43k \| Main html parser. (for separate html files, and the base of .chm support) \|
115	115		\| whtmlscn \| 28k \| HTML scanner objects \|
116	116		\| winclip \| 5k \| Functions that wrap the windows clipboard for Dos and Windows \|
117	117		\| windebug \| 4k \| Helper functions for use of GDB under Windows. cygdrive + update of running program's windowtitle. \|
118	118		\| wini \| 12k \| Ini file using TP objects \|
119	119		\| wnghelp \| 13k \| Help support for Norton Guide (.NG) files \|
120	120		\| woahelp \| 15k \| Borland OA .HLP reader objects and routines \|
121	121		\| wos2help \| 19k \| Help support for OS/2 (.INF) help files \|
122	122		\| wresourc \| 22k \| (TP?) Resource File support objects and routines \|
123	123		\| wtphwrit \| 7k \| Routines to create .tph files \|
124	124		\| [[fv:wutils\|wutils]] \| 32k \| Utils, mostly collections, string routines and file(name/path) routines. \|
125	125		\| wviews \| 69k \| Enhanced View s(widgets) (special for IDE?) \|
126	126		\| wvphelp \| 4k \| Help support for (.VPH) help files \|
127	127		\| wwinhelp \| 49k \| Help support for Windows Help (.HLP) files \|
128	128		\| compunit \| 0k \| (compiler/ dir) build unit for the compiler. \|
129	129		\| gdbcon \| 2k \| (fakegdb/) fake gdb controller unit to be used when no gdb support is included \|
130	130		\| gdbint \| 5k \| (fakegdb/) fake gdb controller unit to be used when no gdb support is included \|
131	131		\| \| \| \|
132	132
133	133		Some functionality seems to be split between a w* unit and a fp* unit. (wini,fpini, wviews, fpview). Possibly this is an artifact of an older division between reusable and non reusable units. The fact that there are multiple utility units is also a bit of an indicator for this.
134	134
135	135		If you know/find out what the criteria for it is, please add it here. If you find out something about a certain unit, make a link out of it prefixed with fv: (see the wutils example).
136	136
137	137		Besides units, there are also a bunch of include files, most of which seem to be part of fpide, and group routines according to menu category (file, edit, compile etc)
138	138
139	139		\| Unit \| Size \| Unit \| Remark \|
140	140		\|--------------\|------\|---------\|-------------------------------------------------\|
141	141		\| empty.inc \| 0k \| test \| Part of IDE testproject, not IDE itself. \|
142	142		\| fpmansi.inc \| 4k \| fpide \| Ansi dump capability \|
143	143		\| fpmcomp.inc \| 4k \| fpide \| Compiler menu entries \|
144	144		\| fpmdebug.inc \| 8k \| fpide \| Debug menu entries \|
145	145		\| fpmedit.inc \| 4k \| fpide \| Edit menu entries (one proc only) \|
146	146		\| fpmfile.inc \| 8k \| fpide \| File menu entries \|
147	147		\| fpmhelp.inc \| 12k \| fpide \| Help menu entries \|
148	148		\| fpmopts.inc \| 48k \| fpide \| Options menu entries \|
149	149		\| fpmrun.inc \| 16k \| fpide \| Run menu entries \|
150	150		\| fpmsrch.inc \| 8k \| fpide \| Search menu entries \|
151	151		\| fpmtools.inc \| 12k \| fpide \| Tools menu entries \|
152	152		\| fpmwnd.inc \| 8k \| fpide \| Window menu entries \|
153	153		\| globdir.inc \| 4k \| MANY \| IDE project wide defines, included in 19 files. \|
154	154		\| wconstse.inc \| 8k \| wconsts \| English constants. \|
155	155		\| wconstsh.inc \| 4k \| wconsts \| Hungarian constants. \|
156	156		\| \| \| \| \|
157	157
158	158		### Classchart of objects + FV + IDE
159	159
160	160		An improvised classchart (objectchart?) of all code containing TP style objects ( unit objects, FV and the IDE source) is available here: [Object Chart (Postscript view with GV)](http://www.stack.nl/~marcov/objectchart.ps) or [pdf version, less often updated](http://www.stack.nl/~marcov/objectchart.pdf)
161	161
162	162		## Debugging the IDE
163	163
164	164		Note that the methods describe here are mostly for *nix.
165	165
166	166		### With GDB tty
167	167
168	168		Normally one would debug the IDE as follows (Unix)
169	169
170	170		Terminal Window 1:
171	171
172		-	~~`tty`~~
	172	+	```text
	173	+	tty
	174	+	```
173	175
174	176		Example output:
175	177
176		-	~~`/dev/pts/5`\~~
177		-	~~` `~~
	178	+	```text
	179	+	/dev/pts/5
	180	+	```
178	181
179	182		Terminal Window 2:
180	183
181	184		``` bash
182	185		gdb ./fp
183	186		tty /dev/pts/5
184	187		r
185	188		```
186	189
187	190		The tty command should then reroute all I/O to tty /dev/pts/5 (Terminal window 1), leaving Terminal Window 2 free to communicate with GDB.
188	191
189	192		Unfortunately, due to the special keyboard support of the textmode IDE (to capture all CTRL-ALT variants) input is not properly rerouted. This means you can debug IDE startup (and more if you save state smartly), but not the normal working. This method is mainly useful when debugging the start of the IDE.
190	193
191	194		### With GDB attach
192	195
193	196		This method works better if you want to debug after the IDE has start up.
194	197
195	198		Terminal Window 1:
196	199
197		-	~~`Run the IDE from the command line, and wait till it has start up. `~~
	200	+	```text
	201	+	Run the IDE from the command line, and wait till it has start up.
	202	+	```
198	203
199	204		Terminal Window 2:
200	205
201		-	~~`Find the PID of the running IDE In window 1 using ps aux \|grep fp`\~~
202		-	~~`Start gdb with the correct "fp" file (the one that is running!)`\~~
203		-	~~`Type attach <pid> on the GDB cmdline`\~~
204		-	~~`Note that gdb will immediately break upon attach. If you want to carry on, type "c" (continue)`~~
	206	+	``` pascal
	207	+	Find the PID of the running IDE In window 1 using ps aux \|grep fp
	208	+	Start gdb with the correct "fp" file (the one that is running!)
	209	+	Type attach <pid> on the GDB cmdline
	210	+	Note that gdb will immediately break upon attach. If you want to carry on, type "c" (continue)
	211	+	```
205	212
206	213		### Writeln like
207	214
208	215		While developing the CHM package I needed to debug browsing also. For this, at first I simply wrote to stderr, and pipe the output to a different terminal (using bash as shell)
209	216
210	217		Terminal Window 1
211	218
212	219		``` bash
213	220		tty
214	221		```
215	222
216	223		Example output:
217	224
218		-	~~`/dev/pts/5`~~
	225	+	```text
	226	+	/dev/pts/5
	227	+	```
219	228
220	229		Terminal Window 2
221	230
222	231		``` bash
223	232		./fp 2> /dev/pts/5
224	233		```
225	234
226	235		However, while this would work, it seems sometimes buffering is a problem. Adding a flush would help, but then you have to go after all those writelns again. So it is better to extract it to a procedure. Such a procedure already existed in wutils, and I enhanced it a bit. There are two versions now. "DebugMessage" is the old one, and DebugMessageS is the new one.
227	236
228	237		To avoid any garbage on the cmdline, all debug messages are always placed under IFDEF. Historically the define "DEBUG" was used for it, but apparantly this is also used in the compiler, and a broken debug feature in the compiler breaks debugging the IDE this way. So I use WDEBUG (mostly in whtmlhlp). The typical debug code fragment then becomes:
229	238
230	239		``` pascal
231	240		{$IFDEF WDEBUG}
232	241		DebugMessageS({$i %file%',' Adding Name "'+Name+'"',%line%,'1');
233	242		{$endif WDEBUG}
234	243		```
235	244
236	245		Note that we use a compiler feature to fill in filename + linenumber for us.
237	246
238	247		On Windows:
239	248
240	249		```dos
241	250		fp.exe 2> logme.txt
242	251		```
243	252
244	253		### Exceptions to stderr
245	254
246	255		When debugging the IDE, I often change the RTL to log errors to stderr instead of stdout, since stderr is logged using the above 2\> redirects, I use the following patch:
247	256
248		-	~~` ===================================================================`\~~
249		-	~~` --- sysutils/sysutils.inc (revision 12440)`\~~
250		-	~~` +++ sysutils/sysutils.inc (working copy)`\~~
251		-	~~` @@ -234,7 +234,7 @@`\~~
252		-	~~` i : longint;`\~~
253		-	~~` hstdout : ^text;`\~~
254		-	~~` begin`\~~
255		-	~~` - hstdout:=@stdout;`\~~
256		-	~~` + hstdout:=@stderr;`\~~
257		-	~~` Writeln(hstdout^,'An unhandled exception occurred at $',HexStr(PtrUInt(Addr),sizeof(PtrUInt)*2),' :');`~~
	257	+	``` pascal
	258	+	===================================================================
	259	+	--- sysutils/sysutils.inc (revision 12440)
	260	+	+++ sysutils/sysutils.inc (working copy)
	261	+	@@ -234,7 +234,7 @@
	262	+	i : longint;
	263	+	hstdout : ^text;
	264	+	begin
	265	+	- hstdout:=@stdout;
	266	+	+ hstdout:=@stderr;
	267	+	Writeln(hstdout^,'An unhandled exception occurred at $',HexStr(PtrUInt(Addr),sizeof(PtrUInt)*2),' :');
	268	+	```
258	269
259	270		---
260	271		From the Free Pascal wiki, page "Textmode IDE development", as it stood on 2022-11-08 after a change by Unxed, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).
261	272		Category: FPC development

Back to history