Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2012-04

Unicode text difference between source editor and other Edit controls on Windows

9 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Juha Manninen

Partly related to issue
21817: "FileUtil.CopyFile" would failed to work with Unicode Characters filename.
http://bugs.freepascal.org/view.php?id=21817

My tests failed when I copied the file name directly to source editor. They succeeded when I used a FileNameEdit to select the file name. Why does it happen?

I remember I participated in a forum discussion about a similar issue. I can dig it out if needed.
Then it happened on Linux, too. I piece of unicode text looked different on source editor compared to other places, including Kate editor (part of KDE).

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120425/4ce2d417/attachment.html>

Reply by email

ik
to Juha Manninen
Juha Manninen wrote 8 lines
Partly related to issue ? 21817: "FileUtil.CopyFile" would failed to work with Unicode Characters filename. http://bugs.freepascal.org/view.php?id=21817 My tests failed when I copied the file name directly to source editor. They succeeded when I used a FileNameEdit to select the file name. Why does it happen?

NTFS and FAT works in a very wrong way. They have code pages for the filename. so if you are using an Arabic filename for example, you must first know if you are using code page for Arabic (and there is more then one for Arabic afaik), or does it use UTF-8 or UTF-16 afaik.

Show quoted text 5 lines
I remember I participated in a forum discussion about a similar issue. I can dig it out if needed. Then it happened on Linux, too. I piece of unicode text looked different on source editor compared to other places, including Kate editor (part of KDE).

In KDE you have also Okteka that is a hex editor, what are the differences that you see there ?

Show quoted text 10 lines
Juha -- _______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Sven Barth
to ik: NTFS and FAT works in a very wrong way. They have code pages for the filename. so if you…
ik wrote 14 lines
On Wed, Apr 25, 2012 at 01:26, Juha Manninen wrote: > Partly related to issue > 21817: "FileUtil.CopyFile" would failed to work with Unicode Characters > filename. > http://bugs.freepascal.org/view.php?id=21817 > > My tests failed when I copied the file name directly to source editor. > They succeeded when I used a FileNameEdit to select the file name. > Why does it happen? NTFS and FAT works in a very wrong way. They have code pages for the filename. so if you are using an Arabic filename for example, you must first know if you are using code page for Arabic (and there is more then one for Arabic afaik), or does it use UTF-8 or UTF-16 afaik.

I don't know about FAT, but NTFS stores filenames as UTF-16 (or UCS-2 at least) and thus does not need any code pages.

Regards,
Sven

Reply by email

Mattias Gaertner
to Sven Barth
Show quoted text 10 lines
Am 25.04.2012 00:43, schrieb ik: > On Wed, Apr 25, 2012 at 01:26, Juha Manninen wrote: >> Partly related to issue >> 21817: "FileUtil.CopyFile" would failed to work with Unicode Characters >> filename. >> http://bugs.freepascal.org/view.php?id=21817 >> >> My tests failed when I copied the file name directly to source editor. >> They succeeded when I used a FileNameEdit to select the file name. >> Why does it happen?

From where are you pasting?
Via drag and drop or via clipboard?
SynEdit uses a different font than TFilenameEdit and treats some characters special.

Show quoted text 7 lines
> NTFS and FAT works in a very wrong way. They have code pages for the > filename. so if you are using an Arabic filename for example, you must > first know if you are using code page for Arabic (and there is more > then one for Arabic afaik), or does it use UTF-8 or UTF-16 afaik. I don't know about FAT, but NTFS stores filenames as UTF-16 (or UCS-2 at least) and thus does not need any code pages.

You can not paste from a file system directly to SynEdit. The bug is somewhere else.

Mattias

Reply by email

Juha Manninen
to Mattias Gaertner: On Wed, 25 Apr 2012 10:04:54 +0200 From where are you pasting? Via drag and drop or via…

On Wed, Apr 25, 2012 at 11:23 AM, Mattias Gaertner <

Show quoted text 7 lines
wrote: From where are you pasting? Via drag and drop or via clipboard? SynEdit uses a different font than TFilenameEdit and treats some characters special.

Hi ik, cobines, Sven, Mattias and DoDi

I copied it through clipboard. The original cyrillic string came from the related issue:

19483: Lazarus can't save to a directory which has unicode characters in its name
http://bugs.freepascal.org/view.php?id=19483

I copied the cyrillic part and created a file: C:\SW\?????1.txt I gave the hardcoded name to CopyFile function call and it failed. Then I created a test app with FileNameEdit, run it, selected the same file name and it worked.
No, there is no {$codepage ..} directive.

Could someone please test if it can be reproduced? It may be somehow caused by my locale, who knows.

The case happens on Windows and the text looks correct in editor, although CopyFile fails.
There was another case on Linux where the exact same text, read from the same file, looked very different in Lazarus source editor than it looked anywhere else. I found the forum thread:

http://lazarus.freepascal.org/index.php/topic,15137.msg81342.html#msg81342

It was also cyrillic text and Lazarus showed it wrong. It was confirmed by one person, ludob.
Could someone explain what happened there? It can't be my system's default encoding because it showed correctly in other editors.

The char encoding stuff is so messy I still don't understand it well. I will not work on it more now because I am busy with both a day-job and school.
Besides, I have a course about Haskell language. I think it should be renamed to Brainfuck language.
Unfortunately there already is a language called Brainfuck. It must be renamed to something else.

http://en.wikipedia.org/wiki/Haskell_%28programming_language%29 http://en.wikipedia.org/wiki/Brainfuck

Regards,
Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120426/dc57f61a/attachment.html>

Reply by email

Hans-Peter Diettrich
to Juha Manninen
Show quoted text 12 lines
On Wed, Apr 25, 2012 at 11:23 AM, Mattias Gaertner < <mailto:>> wrote: From where are you pasting? Via drag and drop or via clipboard? SynEdit uses a different font than TFilenameEdit and treats some characters special. Hi ik, cobines, Sven, Mattias and DoDi I copied it through clipboard.

The Windows clipboard can have content in multiple formats. An application or component should ask for *Unicode* text (CF_UNICODETEXT), in order to get the true filename. Retrieving only CF_TEXT IMO will result in an Ansi string (CP_ACP), with loss of all characters outside the system codepage.

DoDi

Reply by email

cobines
to Juha Manninen: On Wed, Apr 25, 2012 at 11:23 AM, Mattias Gaertner < Hi ik, cobines, Sven, Mattias and…

2012/4/25 Juha Manninen :

Show quoted text 9 lines
I copied it through clipboard. The original cyrillic string came from the related issue: 19483: Lazarus can't save to a directory which has unicode characters in its name ? http://bugs.freepascal.org/view.php?id=19483 I copied the cyrillic part and created a file: C:\SW\?????1.txt I gave the hardcoded name to CopyFile function call and it failed.

I did the same thing as you. I hardcoded a cyrillic and a japanese filename in the source file. I added neither UTF8 BOM nor {$codepage} and CopyFile succeeded.

cobines

Reply by email