Hello,Free Pascal development team!

Sorry, I speak English very bad.

In file fpcsrc/rtl/unix/video.pp nessesary take away lines:

         If fpGetEnv('TERM')='xterm' then
           NoExtendedFrame := true;  {use of acs for xterm is ok}

because it prevents using cyrillic symbols (codepage CP1251).

Kotov Matvey

2 November 2006

Before we apply this patch: What is the side effect for graphic characters ?

Michael.

Afaik this kills linedrawing.

This is something to be addressed in video.pp - we need to detect the character set supported by console at startup (optionally with a possibility to change it later on). This is something we partly touched in discussion with Daniel in Eindhoven (but I haven't managed to do anything regarding video.pp since then, unfortunately :-((( ). Ability to use linedrawing characters directly depends on that, obviously.

Does anybody here have an idea how to address that issue in Unix/Linux environment (like an environment variable similar to TERM, etc.)? I can imagine we might need to define our own solution if no standard solution exists (e.g. we might require users to setup some environment variable explicitly on systems not providing such information in an automated way, plus have some reasonable default).

Tomas

Tomas Hajny wrote

Does anybody here have an idea how to address that issue in Unix/Linux environment (like an environment variable similar to TERM, etc.)? I can

The part after the dot in $LANG. It's possible though that there is no dot, no idea what to do, maybe assume iso-8859-1 ?

Micha

As far as I understand, line drawing comes instead of font drawing. I mean if I have font file loaded, it is responsible for line drawing. The line drawing ability is to allow others to draw their own fonts if no font file was loaded ...

So if I got it right, what is the problem ?!

Tomas

Ido

No, that will be really bad idea to assume iso-8859-1 . It really depends on the country and language code. For example :

LANG=he_IL.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="he_IL.UTF-8"
LC_NUMERIC="he_IL.UTF-8"
LC_TIME="he_IL.UTF-8"
LC_COLLATE="he_IL.UTF-8"
LC_MONETARY="he_IL.UTF-8"
LC_MESSAGES="he_IL.UTF-8"
LC_PAPER="he_IL.UTF-8"
LC_NAME="he_IL.UTF-8"
LC_ADDRESS="he_IL.UTF-8"
LC_TELEPHONE="he_IL.UTF-8"
LC_MEASUREMENT="he_IL.UTF-8"
LC_IDENTIFICATION="he_IL.UTF-8"
LC_ALL=

If I'll remove the UTF-8 it will be he_IL regardless... but the encoding will be iso8859-8 ... If you will place iso8859-1, it will stop working .. I mean you could not use the language your system is set to.

Any language out there that does not use iso8859-1 will suffer from that ...

Micha

Ido

It doesn't depend on the language, but on the font. OTOH, it's probably a good assumption people have configured a proper font for their language.

Micha

Op Fri, 3 Nov 2006, schreef Tomas Hajny:

No, this is a different issue than we talked about in Eindhoven. In Eindhoven we talked about how FV should access line drawing characters. This is about how the video unit draws them.

This sucks big time. There is no way to detect which encoding the terminal uses. The user is expected to set his language in a LC_* variable, and the libc assumes the encoding of the terminal matches the encoding that belongs to the language.

Xterm supports two modes: ISO-8859-1 and UTF-8. There are two ways to use a different code page:
* Put the xterm in ISO-8859-1 mode and just load a font with different characters in #128..#255.
* Use proper fonts, but use an emulator that switches the terminal into UTF-8 mode and emulates a terminal with the desired encoding. I forgot the name of the emulator.

The line in question enables the use of the vt100 alternate character set. You access the normal font by switching the terminal to the G0 charactert set, and the ACS by switching it into the G1 character set. The ACS should be available when code page 1251 is in use.

However, the video unit emulates a code page 850 on a iso-8859-1, using the ACS for the line-drawing characters. This doesn't work for Russian. As a short term solution a conversion table should be added from a Cyrillic Dos code page to 1251+ACS.

Daniel

It is also possible there is no LANG, like on all four Stack systems I quickchecked. (freebsd,Solaris and debian mixed)

p.s. no other of the env vars or anything looking like it either.

But this isn't our fault then? People have to set their LANG then?

No language => assume POSIX => assume iso-8859-1 ?

Running 'locale' gives POSIX for me if I didn't export a LANG.

Micha

Op Fri, 3 Nov 2006, schreef Micha Nelissen:

No: no language -> C locale. In the C locale you don't know which character set is used, and sorting is done according to ASCII value and not alphabetically.

Daniel

While ok in general, some older systems might still be in whatever the vgafont specifies (iow dos CP437) on the console.

Running 'locale' gives POSIX for me if I didn't export a LANG.

[Snail] </home/marcov> locale
LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_TIME="C"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

No, why? You are trying to derive terminal settings via some random locale setting which didn't use to be related, and only set mandatorily on relatively new system. (a lot of older distro's do have it, but not in base)

Setting lang might totally mess up an older system, since other programs might start to try to configure themselves while locale packages are not installed.

Moreover, I think the IDE should be usable without a lot of configuring and safe fallback on older systems, So if you want to set a sane default if no LANG var is present (or even a special "old systems mode", that is ok with me, but don't require setting lang.

Op Fri, 3 Nov 2006, schreef Marco van de Voort:

While ok in general, some older systems might still be in whatever the vgafont specifies (iow dos CP437) on the console.

The Linux console always boots up in Latin-1 mode, with a code page 437 loaded into the VGA. The console is another matter, which complicates the situation even more.

What is the FreeBSD situation on the console?

XFree86 3 systems have ISO-8859-1 only and need
font tricks to use different character sets.

Daniel

One can choose a font during setup. The tool has a very slackware feel. Afaik this font setting is totally separate from any other lang settings.

Op Fri, 3 Nov 2006, schreef Marco van de Voort:

Does the FreeBSD console go straight to the font or does it do translation like Linux?

Daniel

Of course, that's what I meant, if there is no LANG setting people have to live with that what the IDE does.

Well, it's at least partly related to that discussion, IMHO - see below.

.
.

UTF-8 mode should work, shouldn't it (as long as video unit can draw the line drawing characters using their UTF-8 codes, of course)?

I don't think this is a reasonable solution. Why not to use translation capabilities provided by the WideString manager? Otherwise you'd need to implement translation tables for all the different pairs...

Tomas

Micha Nelissen napsal(a):

That assumption is correct, but not directly useable, unless you want to maintain information about useable character sets for all possible locale combinations (I don't ;-) ).

Tomas

.
.

We were discussing the default behaviour used if no information about encoding is available. This only means that those who cannot use the default (ISO 8859-1) have to make sure to provide the real information, otherwise it just isn't guaranteed to work correctly. I don't think this should be a problem, but I'm open to your arguments if you think otherwise.

BTW, note that one can always set environment variable even just for a particular session (e.g. in a shell script starting the particular binary), so even if it isn't possible to change the default for some reason (e.g. due to other programs running on the same machine under the same user), it should be still useable (responding to Marco's comment about old machines here).

Tomas

The problem is that we do not know if there is some sort of font loaded or not. For example, in my Ubuntu, because in the installation I set up "Hebrew", it configured by default a font for the VT called iso8. But when I'm using X, this fonts does not work for terminals, so I can use terminals such as konsole, mlterm or xterm-unicode (I think some others as well) to display Hebrew characters (don't talk about bi-directional support... that's just too much :)).

On my Konsole (kde's terminal), it's the font and a checkbox (inside the program) that tells it to use Hebrew, rather then the locale. That's for example, the way I'm able to translate the Hebrew version of FPC to cp1255 (Windows Hebrew) while I am at UTF-8 charset. I think it is up to the user to set the locale, and if there is no locale, then ASCII is our locale, rather then latin1.

Tomas

Ido

Op Fri, 3 Nov 2006, schreef Tomas Hajny:

You must assume a common base, which is the VT100, the only standard that every terminal emulator implements. This is the level the video unit currently supports.

Then, you can detect wether the terminal supports UTF-8. This is not possible, but you can write a character and detect wether the cursor moves one or two positions.

If the terminal supports UTF-8, there are still a few pitfalls: The terminal emulator might not recognize line drawing characters. In this case the font might still provide them, but the terminal emulator cannot draw continuous lines in this case. The situation gets more difficult if the line drawing characters are not in the font.

To use UTF-8, you must first determine wether a UTF-8 mode is available. To make it worse, you cannot, although there exists a hack to write a character and test if the cursor advances one or two positions.

The widestring manager is part of a generic solution, but we aren't quite there yet. There are two code pages to worry about:
* The filesystem and terminal encoding
* The VGA font encoding

Normally the file system encoding is one of the iso-8859-* standard, or UTF8, the VGA font is normally one of the Dos code pages. The video unit has to care about both code pages, and therefore needs to keep track of

Otherwise you'd need to implement translation tables for all the different pairs...

Between the Dos code page and its matching iso code page, yes. This has by the way an advantage, and that is that a fallback character can be used in many situations. For example an up arrow is replaced by a ^. This is not possible with a widestring manager at this time, but necessary to be able to display scroll bars.

Keep in mind that we are just starting: Before 2.0.4 just the line drawing characters were converted and except those only #32..#127 could be drawn in a reliable way.

Daniël

One can load additional translation maps by hand. I don't know if this is done by the default selection.

Well, either ASCII or latin1, the end result is
the same for you (i.e. Hebrew and other
characters not working there), isn't it?

Tomas

First of all it easier to detect ASCII only then latin1/2 etc ... Because there are only 127 possible chars (well less because you can see from 32-127), while in latin1, you will see higher values, but it will take time to understand why you see the *wrong* chars rather then the correct ones. At least for :)

Tomas

Ido