Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2008-04

Unicode support in source editor

10 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Mattias Gaertner

Hi all,

The IDE now (svn 14904) tries to find out the encoding of a source file and converts it to UTF-8 on loading and back on saving.

The converter currently supports the following encodings: ISO-8859-1
windows code pages
1250 central europe
1251 cyrillic
1252 latin 1
1253 greek
1254 turkish
1255 hebrew
1256 arabic
1257 baltic
1258 vietnam
874 thai

The IDE uses a simple heuristic to find out the current encoding. (unit LConvEncoding, function GuessEncoding). It tests for BOM, {%encoding eee}, ASCII and UTF-8. If none fits then the system encoding is used. If the system encoding is UTF-8 then ISO-8859-1 is chosen.

See here:
http://wiki.lazarus.freepascal.org/IDE_Development#UTF-8_Sources

Mattias

Reply by email

ik
to Mattias Gaertner

Mattias.

I believe that it is a wrong way to do it. For example the 12xx are Windows language codes. Sometimes you wish to use them as-is, so this will actually create a bug imho.

You should create a tool that asks on each string (or yes to all) to convert it into UTF-8 instead.

Ido

Mattias Gaertner wrote 34 lines
Hi all, The IDE now (svn 14904) tries to find out the encoding of a source file and converts it to UTF-8 on loading and back on saving. The converter currently supports the following encodings: ISO-8859-1 windows code pages 1250 central europe 1251 cyrillic 1252 latin 1 1253 greek 1254 turkish 1255 hebrew 1256 arabic 1257 baltic 1258 vietnam 874 thai The IDE uses a simple heuristic to find out the current encoding. (unit LConvEncoding, function GuessEncoding). It tests for BOM, {%encoding eee}, ASCII and UTF-8. If none fits then the system encoding is used. If the system encoding is UTF-8 then ISO-8859-1 is chosen. See here: http://wiki.lazarus.freepascal.org/IDE_Development#UTF-8_Sources Mattias _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Felipe Monteiro de Carvalho
to ik
I believe that it is a wrong way to do it. For example the 12xx are Windows language codes. Sometimes you wish to use them as-is, so this will actually create a bug imho.

That's trivial to solve. Don't add a BOM to your file and use:

{%encoding utf-8}

(By the way, is this supported? I don't see this option in the wiki article)

This way both the IDE and the Compiler will ignore your encoding, and no conversion takes place. On the bad side, you also won't be able to edit non-utf-8 strings in the editor, but it's always impossible to edit more then one encoding at the same time.

About asking for each string, I think this would be a inviable solution.

thanks,

Felipe Monteiro de Carvalho

Reply by email

Mattias Gaertner
to ik: Mattias. I believe that it is a wrong way to do it. For example the 12xx are Windows…
Show quoted text 8 lines
Mattias. I believe that it is a wrong way to do it. For example the 12xx are Windows language codes. Sometimes you wish to use them as-is, so this will actually create a bug imho. You should create a tool that asks on each string (or yes to all) to convert it into UTF-8 instead.

I'm not sure, what you mean.

The LCL expects UTF-8 strings, so does synedit and all other IDE controls.
When the IDE opens a text file (e.g. unit1.pas) it checks if it is UTF-8. Many Delphi sources are encoded in a windows code page. If it is not UTF-8 then it must be converted. And when the IDE saves a file to disk, it must convert it back to the original encoding.

> The IDE now (svn 14904) tries to find out the encoding of a source > file and converts it to UTF-8 on loading and back on saving.

Mattias

Reply by email

Mattias Gaertner
to Felipe Monteiro de Carvalho: That's trivial to solve. Don't add a BOM to your file and use: {%encoding utf-8} (By the…
Show quoted text 10 lines
> I believe that it is a wrong way to do it. For example the 12xx are > Windows language codes. Sometimes you wish to use them as-is, so > this will actually create a bug imho. That's trivial to solve. Don't add a BOM to your file and use: {%encoding utf-8} (By the way, is this supported? I don't see this option in the wiki article)

Yes. The example in the wiki is {%encoding cp1250}.

http://wiki.lazarus.freepascal.org/IDE_Development#UTF-8_Sources

Show quoted text 7 lines
This way both the IDE and the Compiler will ignore your encoding, and no conversion takes place. On the bad side, you also won't be able to edit non-utf-8 strings in the editor, but it's always impossible to edit more then one encoding at the same time. About asking for each string, I think this would be a inviable solution.

I will add a more complete explanation to the wiki:

Formerly the IDE only supported one encoding. Under windows this was the windows code page. Under gtk2 it was UTF-8. Now the IDE can edit files with various encodings. (one encoding per file). For instance, the user opens a unit1.pas file created on a german windows. Then the german umlaute will be encoded with the windows code page cp1250. The IDE now "sees", that this is not UTF-8, assumes it is the system encoding and converts it to UTF-8 (only in memory, not on disk), before passing it to SynEdit. Now SynEdit can edit it. When the file is saved, the IDE converts it back to cp1250. This way the file on disk can have the windows system encoding, but all IDE functions only need to support UTF-8.

Mattias

Reply by email

Felipe Monteiro de Carvalho
to Mattias Gaertner: On Sun, 20 Apr 2008 13:34:23 +0300 I'm not sure, what you mean. The LCL expects UTF-8…

Opening one of the files of the magnifier it showed the file as a blank line. Editing the file in that state would probably destroy it (irrelevant if your file is under versioning system). Adding {%encoding utf-8} with an external editor solved it.

But this was probably a impossible file to detect as utf-8, because it has both utf-8 and iso, ifdefed for fpc and delphi and no bom =)

Should I post a bug report?

Felipe Monteiro de Carvalho

Reply by email

Mattias Gaertner
to Felipe Monteiro de Carvalho
Show quoted text 9 lines
Opening one of the files of the magnifier it showed the file as a blank line. Editing the file in that state would probably destroy it (irrelevant if your file is under versioning system). Adding {%encoding utf-8} with an external editor solved it. But this was probably a impossible file to detect as utf-8, because it has both utf-8 and iso, ifdefed for fpc and delphi and no bom =) Should I post a bug report?

Yes, please.

Mattias

Reply by email