Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2022-03

converting to UTF8

5 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Unknown author

Lazarus 2.012
FPC: 3.2.0
SVN: 64642
OS: Windows 10 Pro/win64

I'm reading from a file with character set win1252, I want to convert it to utf8, but I can't.

procedure TestString;
var
   Original:string;
   Converted:string;
begin
   original:='ESPA'#209'A'; //ESPAÑA WIN1252
   Converted:=ansiToUtf8(original);  // converts to 'ESPA'#239#191#189'A'
  // converted Should be 'ESPA'#195#145'A'
end;

I've tried playing with strings types, string, rawstring,ansistring, utf8string. No way. Any hint?

------------
Saludos
Santiago A.

LacaK
to Unknown author
Show quoted text 13 lines
procedure TestString; var   Original:string;   Converted:string; begin   original:='ESPA'#209'A'; //ESPAÑA WIN1252   Converted:=ansiToUtf8(original);  // converts to 'ESPA'#239#191#189'A'  // converted Should be 'ESPA'#195#145'A' end; I've tried playing with strings types, string, rawstring,ansistring, utf8string. No way. Any hint?

To explicityly convert between run-time code pages you can use procedure: SetCodePage(var s: RawByteString; CodePage: TSystemCodePage; Convert: Boolean = True)

SetCodePage(original, 1252, False);
SetCodePage(original, CP_UTF8, True);

-Laco.

Mattias Gaertner
to Unknown author: Lazarus 2.012 FPC: 3.2.0 SVN: 64642 OS: Windows 10 Pro/win64 I'm reading from a file with…
Show quoted text 14 lines
Lazarus 2.012 FPC: 3.2.0 SVN: 64642 OS: Windows 10 Pro/win64 I'm reading from a file with character set win1252, I want to convert it to utf8, but I can't. procedure TestString; var Original:string; Converted:string; begin original:='ESPA'#209'A'; //ESPAÑA WIN1252

FPC does not yet understand comments, so maybe it does not know this literal is cp1252. Add {$codepage cp1252} somewhere at the start of the unit.

If this is part of a Lazarus application, then String is by default UTF-8, so your "original" is already converted to UTF-8.

Converted:=ansiToUtf8(original); // converts to 'ESPA'#239#191#189'A' // converted Should be 'ESPA'#195#145'A'

If you want to load a string encoded in CP1252, then you can use from unit lconvencoding:

s:=CP1252ToUTF8(StringFromFile);
end; I've tried playing with strings types, string, rawstring,ansistring, utf8string. No way. Any hint?

Mattias

Santi
to Mattias Gaertner
Mattias Gaertner wrote
FPC does not yet understand comments,

What do you mean with "understand comments"?

Show quoted text 6 lines
so maybe it does not know this literal is cp1252. Add {$codepage cp1252} somewhere at the start of the unit. If this is part of a Lazarus application, then String is by default UTF-8, so your "original" is already converted to UTF-8.

So, as far as Ive understood,  without {$codepage cp1252}, when I read the string from a file, it supposes it is utf8, even when it is not a valid utf8 string.

And ansiToUtf8 doesn't suppose that the input string ansi, but utf8? I can't understand what is ansiToUtf8  for.

If you want to load a string encoded in CP1252, then you can use from unit lconvencoding: s:=CP1252ToUTF8(StringFromFile);

OK. that is the solution I've chosen.
Thanks

Mattias _______________________________________________ fpc-pascal maillist - https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Saludos Santi
Tomas Hajny
to Santi
Santi wrote
El 23/03/2022 a las 10:51, Mattias Gaertner via fpc-pascal escribió: > > FPC does not yet understand comments, What do you mean with "understand comments"?

Mattias meant that comments were the only place where information about use of CP1252 was included.

Show quoted text 9 lines
> so maybe it does not know this > literal is cp1252. Add {$codepage cp1252} somewhere at the start of > the unit. > > If this is part of a Lazarus application, then String is by > default UTF-8, so your "original" is already converted to UTF-8. So, as far as Ive understood,  without {$codepage cp1252}, when I read the string from a file, it supposes it is utf8, even when it is not a valid utf8 string.

First, there are many cases when a file may be a valid UTF-8 file, yet it is still a valid file in ANSI encoding. There must be some default. Lazarus uses UTF-8 as the default.

And ansiToUtf8 doesn't suppose that the input string ansi, but utf8? I can't understand what is ansiToUtf8  for.

A particular encoding must be already defined when the string is assigned to the string. The string contents may not change depending on which function is called on it (moreover, implicit conversions migt be used regardless from the original encoding.

Tomas