Hello folks,

I need to do some conversion from one character set encoding, given by its name (e.g. value of encoding attribute in XML header), to another (used internally, based on locale), and back.
I know of iconv on Linux/Unix OSes, but I would like to do it in a portable manner, if possible.
Do you know an existing solution ?
I saw there is a 'charset' unit in the RTL of FPC, but it is not documented, ant it is not yet very clear to me how to use it or even if it does what I need...

Dominique.

Op Thu, 7 Dec 2006, schreef Dominique Leducq:

For the website generation, I use a tweaked cwstring widestring manager, see http://svn.freepascal.org/svn/html/adp2html.pp. For a more structural and portable approach, the functionality to specify encodings needs to be integrated into the widestringmanager.

Daniël

Thanks Daniël,

The changes you made to cwstring is merely to export the iconv API from it. As I stated, I already know how to use iconv on Linux/Unix. And I don't necessarily have to deal with widestrings (for example to convert latin-1 to UTF-8).
On MS Windows, I found WideCharToMultiByte and MultiByteToWideChar, which take a numerical CodePage as parameter. But I don't know how to map a charset name ('UTF-8', 'Latin-1', 'ISO-8859-15'...) to a CodePage (any hint ?).

So apparently there doesn't exist a portable API on top of this.

Dominique

P.S.: About the cwstring unit, I experimented that nl_langinfo(CODESET) always gives ANSI_X3.4-1968 (US-ASCII), unless you call setlocale(LC_CTYPE, ...) before (... here being an explicit locale name, or empty string to use the locale environment variables setting). So the widestring manager don't work as expected...

Op Thu, 7 Dec 2006, schreef Dominique Leducq:

The changes you made to cwstring is merely to export the iconv API from it. As I stated, I already know how to use iconv on Linux/Unix.

Yes, I only gave it as example how I worked around it.

And I don't necessarily have to deal with widestrings (for example to convert latin-1 to UTF-8).

I think it will be best to convert to widestring on input and to the desired encoding on output; FPC's support is being built around the widestring manager, which is, as its name says, mostly designed for widestrings.

On MS Windows, I found WideCharToMultiByte and MultiByteToWideChar, which take a numerical CodePage as parameter. But I don't know how to map a charset name ('UTF-8', 'Latin-1', 'ISO-8859-15'...) to a CodePage (any hint ?).

I'm not sure wether all ISO encodings have a code page number, some definately have. IBM keeps a registry of all code pages they have defined:

http://www-03.ibm.com/servers/eserver/iseries/software/globalization/codepages.html

Also of use migth be the IANA registry of character sets, these are the encoding which are allowed to be used on the internet, which is mostly a subset of the ISO encodings combined with a subset of the IBM code pages.

http://www.iana.org/assignments/character-sets

So apparently there doesn't exist a portable API on top of this.

Not yet. Feel free to propose one.

Okay, noted.

Daniël

So apparently there doesn't exist a portable API on top of this.

There's APR:
http://apr.apache.org/
http://apr.apache.org/docs/apr-iconv/

See fpc\packages\base\httpd\httpd-2.2\apriconv directory. There are APR DLLs on Windows which you can link with.

HTH
TOndrej

On MS Windows, I found WideCharToMultiByte and MultiByteToWideChar, which take a numerical CodePage as parameter. But I don't know how to map a charset name ('UTF-8', 'Latin-1', 'ISO-8859-15'...) to a CodePage (any hint ?).

Iconv exists for windows too. (called iconv.dll) I am using it in my Synapse. unit synaicnv.pas is interface for iconv.dll, and synachar.pas is unit with various charset transpation functions. You can use it on windwos or linux/unix. Delphi, kylix, Freepascal. Just download Synapse and use it. :-) (You can found iconv.dll on Synapse web page too...)

Hi,

I attached conversion fpc program i had written earlier. Maybe you will find something usefull in it.

{$mode delphi}
uses
  windows, sysutils, classes, koduotes;
var
  w: widestring;
  p: string;
  l: integer;
  s: string;
  cp: string;
  cp2: string;
  st: TFileStream;
  tf: string;
begin
  cp:= paramstr(1);
  s:= paramstr(2);
  cp2:= paramstr(3);
  tf:= paramstr(4);
  if (not FileExists(s)) or (cp='') or (cp2='') or (tf='') then
  begin
    writeln('Naudojimas: koduok.exe koduote1 duomenu_failas koduote2
rezultatu_failas');
    writeln('Galimos koduote1 ir koduote2 reiksmes: ');
    Israsyk_Koduotes;
    exit;
  end;
  st:= TFileStream.Create(s, fmOpenRead);
  try
    setlength(p, st.size);
    l:= st.size;
    if st.size=0 then
    begin
      Writeln('Nera ko kuoduoti');
      Exit;
    end;
    st.read(p[1], st.size);
  finally
    st.free;
  end;
  setlength(w, l);
  l:= multibytetowidechar(strtoint(cp), 0, pchar(p), l, pwidechar(w), l);
  setlength(w, l);
  if l=0 then
  begin
    Writeln('Nieko neperkoduota');
    Exit;
  end;
  setlength(p, l*4);
  l:= widechartomultibyte(strtoint(cp2), 0, pwidechar(w), l, pchar(p), l*4,
nil,nil);
  setlength(p, l);
  if l=0 then
  begin
    Writeln('Nieko neperkoduota 2');
    Exit;
  end;
  st:= TFileStream.Create(tf, fmCreate);
  try
    st.write(p[1], l);
  finally
    st.free;
  end;
end.
{$mode delphi}
unit koduotes;
interface
uses
  classes, sysutils, windows;
const
_037  = 'IBM EBCDIC - U.S./Canada';
_437  = 'DOS - United States';
_500  = 'IBM EBCDIC - International';
_708  = 'Arabic - ASMO 708';
_709  = 'Arabic - ASMO 449+, BCON V4';
_710  = 'Arabic - Transparent Arabic';
_720  = 'Arabic - Transparent ASMO';
_737  = 'DOS 737- Graiku (formerly 437G)';
_775  = 'DOS 775 - Baltijos saliu';
_850  = 'DOS 850 - Multilingual Latin I';
_852  = 'DOS 852 - Latin II';
_855  = 'DOS 855 - Kirilica (primarily Russian)';
_857  = 'DOS 857 - Turku';
_858  = 'DOS 858 - Multlingual Latin I + Euro symbol';
_860  = 'DOS 860 - Portugalu';
_861  = 'DOS 861 - Islandijos';
_862  = 'DOS 862 - Hebraju';
_863  = 'DOS 863 - Kanados-prancuzu';
_864  = 'DOS 864 - Arabu';
_865  = 'DOS 865 - Nordic';
_866  = 'DOS 866 - Rusu';
_869  = 'DOS 869 - Modern Greek';
_870  = 'IBM EBCDIC 870 - Multilingual/ROECE (Latin-2)';
_874  = 'Windows/DOS 874 - Thai (same as 28605, ISO 8859-15)';
_875  = 'IBM EBCDIC 875 - Modern Greek';
_932  = 'Windows/DOS 932 - Japanese, Shift-JIS';
_936  = 'Windows/DOS 936 - Simplified Chinese (PRC, Singapore)';
_949  = 'Windows/DOS 949 - Korean (Unified Hangeul Code)';
_950  = 'Windows/DOS 950 - Traditional Chinese (Taiwan; Hong Kong SAR,
PRC)';
_1026  = 'IBM EBCDIC 1026 - Turkish (Latin-5)';
_1047  = 'IBM EBCDIC 1047 - Latin 1/Open System';
_1140  = 'IBM EBCDIC 1140 - U.S./Canada (037 + Euro symbol)';
_1141  = 'IBM EBCDIC 1141 - Germany (20273 + Euro symbol)';
_1142  = 'IBM EBCDIC 1142 - Denmark/Norway (20277 + Euro symbol)';
_1143  = 'IBM EBCDIC 1143 - Finland/Sweden (20278 + Euro symbol)';
_1144  = 'IBM EBCDIC 1144 - Italy (20280 + Euro symbol)';
_1145  = 'IBM EBCDIC 1145 - Latin America/Spain (20284 + Euro symbol)';
_1146  = 'IBM EBCDIC 1146 - United Kingdom (20285 + Euro symbol)';
_1147  = 'IBM EBCDIC 1147 - France (20297 + Euro symbol)';
_1148  = 'IBM EBCDIC 1148 - International (500 + Euro symbol)';
_1149  = 'IBM EBCDIC 1149 - Icelandic (20871 + Euro symbol)';
_1200  = 'Unicode UCS-2 Little-Endian (BMP of ISO 10646)';
_1201  = 'Unicode UCS-2 Big-Endian';
_1250  = 'Windows 1250- Vidurio Europos';
_1251  = 'Windows 1251- Kirilica';
_1252  = 'Windows 1252- Latin I';
_1253  = 'Windows 1253- Graiku';
_1254  = 'Windows 1254- Turku';
_1255  = 'Windows 1255- Hebraju';
_1256  = 'Windows 1256- Arabu';
_1257  = 'Windows 1257- Baltijos saliu';
_1258  = 'Windows/DOS 1258 - Vietnamieciu';
_1361  = 'Korean (Johab)';
_10000  = 'MAC 10000- Roman';
_10001  = 'MAC 10001- Japanese';
_10002  = 'MAC 10002- Traditional Chinese (Big5)';
_10003  = 'MAC 10003- Korean';
_10004  = 'MAC 10004- Arabic';
_10005  = 'MAC 10005- Hebrew';
_10006  = 'MAC 10006- Greek I';
_10007  = 'MAC 10007- Cyrillic';
_10008  = 'MAC 10008- Simplified Chinese (GB 2312)';
_10010  = 'MAC 10010- Romania';
_10017  = 'MAC 10017- Ukraine';
_10021  = 'MAC 10021- Thai';
_10029  = 'MAC 10029- Latin II';
_10079  = 'MAC 10079- Icelandic';
_10081  = 'MAC 10081- Turkish';
_10082  = 'MAC 10082- Croatia';
_12000  = 'Unicode UCS-4 Little-Endian';
_12001  = 'Unicode UCS-4 Big-Endian';
_20000  = 'CNS - Taiwan';
_20001  = 'TCA - Taiwan';
_20002  = 'Eten - Taiwan';
_20003  = 'IBM5550 - Taiwan';
_20004  = 'TeleText - Taiwan';
_20005  = 'Wang - Taiwan';
_20105  = 'IA5 IRV International Alphabet No. 5 (7-bit)';
_20106  = 'IA5 German (7-bit)';
_20107  = 'IA5 Swedish (7-bit)';
_20108  = 'IA5 Norwegian (7-bit)';
_20127  = 'US-ASCII (7-bit)';
_20261  = 'T.61';
_20269  = 'ISO 6937 Non-Spacing Accent';
_20273  = 'IBM EBCDIC - Germany';
_20277  = 'IBM EBCDIC - Denmark/Norway';
_20278  = 'IBM EBCDIC - Finland/Sweden';
_20280  = 'IBM EBCDIC - Italy';
_20284  = 'IBM EBCDIC - Latin America/Spain';
_20285  = 'IBM EBCDIC - United Kingdom';
_20290  = 'IBM EBCDIC - Japanese Katakana Extended';
_20297  = 'IBM EBCDIC - France';
_20420  = 'IBM EBCDIC - Arabic';
_20423  = 'IBM EBCDIC - Greek';
_20424  = 'IBM EBCDIC - Hebrew';
_20833  = 'IBM EBCDIC - Korean Extended';
_20838  = 'IBM EBCDIC - Thai';
_20866  = 'Russian - KOI8-R';
_20871  = 'IBM EBCDIC - Icelandic';
_20880  = 'IBM EBCDIC - Cyrillic (Russian)';
_20905  = 'IBM EBCDIC - Turkish';
_20924  = 'IBM EBCDIC - Latin-1/Open System (1047 + Euro symbol)';
_20932  = 'JIS X 0208-1990 & 0121-1990';
_20936  = 'Simplified Chinese (GB2312)';
_20949  = 'x-cp20949 Korean Wansung';
_21025  = 'IBM EBCDIC - Cyrillic (Serbian, Bulgarian)';
_21027  = 'Extended Alpha Lowercase';
_21866  = 'Ukrainian (KOI8-U)';
_28591  = 'ISO 8859-1 Latin I';
_28592  = 'ISO 8859-2 Vidurio Europos';
_28593  = 'ISO 8859-3 Latin 3';
_28594  = 'ISO 8859-4 Baltijos saliu';
_28595  = 'ISO 8859-5 Kirilica';
_28596  = 'ISO 8859-6 Arabu';
_28597  = 'ISO 8859-7 Graiku';
_28598  = 'ISO 8859-8 Hebraju';
_28599  = 'ISO 8859-9 Latin 5';
_28600  = 'ISO 8859-10';
_28601  = 'ISO 8859-11';
_28602  = 'ISO 8859-12';
_28603  = 'ISO 8859-13 Baltijos saliu';
_28604  = 'ISO 8859-14';
_28605  = 'ISO 8859-15 Latin 9';
_29001  = 'Europa 3';
_38598  = 'ISO 8859-8 Hebrew';
_50220  = 'ISO 2022 Japanese with no halfwidth Katakana';
_50221  = 'ISO 2022 Japanese with halfwidth Katakana';
_50222  = 'ISO 2022 Japanese JIS X 0201-1989';
_50225  = 'ISO 2022 Korean';
_50227  = 'ISO 2022 Simplified Chinese';
_50229  = 'ISO 2022 Traditional Chinese';
_50930  = 'Japanese (Katakana) Extended';
_50931  = 'US/Canada and Japanese';
_50933  = 'Korean Extended and Korean';
_50935  = 'Simplified Chinese Extended and Simplified Chinese';
_50936  = 'Simplified Chinese';
_50937  = 'US/Canada and Traditional Chinese';
_50939  = 'Japanese (Latin) Extended and Japanese';
_51932  = 'EUC - Japanese';
_51936  = 'EUC - Simplified Chinese';
_51949  = 'EUC - Korean';
_51950  = 'EUC - Traditional Chinese';
_52936  = 'HZ-GB2312 Simplified Chinese';
_54936  = 'Windows XP: GB18030 Simplified Chinese (4 Byte)';
_57002  = 'ISCII Devanagari';
_57003  = 'ISCII Bengali';
_57004  = 'ISCII Tamil';
_57005  = 'ISCII Telugu';
_57006  = 'ISCII Assamese';
_57007  = 'ISCII Oriya';
_57008  = 'ISCII Kannada';
_57009  = 'ISCII Malayalam';
_57010  = 'ISCII Gujarati';
_57011  = 'ISCII Punjabi';
_65000  = 'Unicode UTF-7';
_65001  = 'Unicode UTF-8';
cp_str  = 'Nezinoma koduote MS ID: %s';
procedure Israsyk_Koduotes;
var
  cp_list: TStringList;
implementation
function EnumCP(p: pchar): integer; stdcall;
begin
  cp_list.add(string(p));
  result:=1;
end;
procedure Israsyk_Koduotes;
var
  i: integer;
  a: integer;
begin
  for i:= 0 to cp_list.Count-1 do
  begin
    a:= strtoint(cp_list[i]);
    case a of
      037   : writeln(cp_list[i], ' ', _037);
      437   : writeln(cp_list[i], ' ', _437);
      500   : writeln(cp_list[i], ' ', _500);
      708   : writeln(cp_list[i], ' ', _708);
      709   : writeln(cp_list[i], ' ', _709);
      710   : writeln(cp_list[i], ' ', _710);
      720   : writeln(cp_list[i], ' ', _720);
      737   : writeln(cp_list[i], ' ', _737);
      775   : writeln(cp_list[i], ' ', _775);
      850   : writeln(cp_list[i], ' ', _850);
      852   : writeln(cp_list[i], ' ', _852);
      855   : writeln(cp_list[i], ' ', _855);
      857   : writeln(cp_list[i], ' ', _857);
      858   : writeln(cp_list[i], ' ', _858);
      860   : writeln(cp_list[i], ' ', _860);
      861   : writeln(cp_list[i], ' ', _861);
      862   : writeln(cp_list[i], ' ', _862);
      863   : writeln(cp_list[i], ' ', _863);
      864   : writeln(cp_list[i], ' ', _864);
      865   : writeln(cp_list[i], ' ', _865);
      866   : writeln(cp_list[i], ' ', _866);
      869   : writeln(cp_list[i], ' ', _869);
      870   : writeln(cp_list[i], ' ', _870);
      874   : writeln(cp_list[i], ' ', _874);
      875   : writeln(cp_list[i], ' ', _875);
      932   : writeln(cp_list[i], ' ', _932);
      936   : writeln(cp_list[i], ' ', _936);
      949   : writeln(cp_list[i], ' ', _949);
      950   : writeln(cp_list[i], ' ', _950);
      1026  : writeln(cp_list[i], ' ', _1026);
      1047  : writeln(cp_list[i], ' ', _1047);
      1140  : writeln(cp_list[i], ' ', _1140);
      1141  : writeln(cp_list[i], ' ', _1141);
      1142  : writeln(cp_list[i], ' ', _1142);
      1143  : writeln(cp_list[i], ' ', _1143);
      1144  : writeln(cp_list[i], ' ', _1144);
      1145  : writeln(cp_list[i], ' ', _1145);
      1146  : writeln(cp_list[i], ' ', _1146);
      1147  : writeln(cp_list[i], ' ', _1147);
      1148  : writeln(cp_list[i], ' ', _1148);
      1149  : writeln(cp_list[i], ' ', _1149);
      1200  : writeln(cp_list[i], ' ', _1200);
      1201  : writeln(cp_list[i], ' ', _1201);
      1250  : writeln(cp_list[i], ' ', _1250);
      1251  : writeln(cp_list[i], ' ', _1251);
      1252  : writeln(cp_list[i], ' ', _1252);
      1253  : writeln(cp_list[i], ' ', _1253);
      1254  : writeln(cp_list[i], ' ', _1254);
      1255  : writeln(cp_list[i], ' ', _1255);
      1256  : writeln(cp_list[i], ' ', _1256);
      1257  : writeln(cp_list[i], ' ', _1257);
      1258  : writeln(cp_list[i], ' ', _1258);
      1361  : writeln(cp_list[i], ' ', _1361);
      10000 : writeln(cp_list[i], ' ', _10000);
      10001 : writeln(cp_list[i], ' ', _10001);
      10002 : writeln(cp_list[i], ' ', _10002);
      10003 : writeln(cp_list[i], ' ', _10003);
      10004 : writeln(cp_list[i], ' ', _10004);
      10005 : writeln(cp_list[i], ' ', _10005);
      10006 : writeln(cp_list[i], ' ', _10006);
      10007 : writeln(cp_list[i], ' ', _10007);
      10008 : writeln(cp_list[i], ' ', _10008);
      10010 : writeln(cp_list[i], ' ', _10010);
      10017 : writeln(cp_list[i], ' ', _10017);
      10021 : writeln(cp_list[i], ' ', _10021);
      10029 : writeln(cp_list[i], ' ', _10029);
      10079 : writeln(cp_list[i], ' ', _10079);
      10081 : writeln(cp_list[i], ' ', _10081);
      10082 : writeln(cp_list[i], ' ', _10082);
      12000 : writeln(cp_list[i], ' ', _12000);
      12001 : writeln(cp_list[i], ' ', _12001);
      20000 : writeln(cp_list[i], ' ', _20000);
      20001 : writeln(cp_list[i], ' ', _20001);
      20002 : writeln(cp_list[i], ' ', _20002);
      20003 : writeln(cp_list[i], ' ', _20003);
      20004 : writeln(cp_list[i], ' ', _20004);
      20005 : writeln(cp_list[i], ' ', _20005);
      20105 : writeln(cp_list[i], ' ', _20105);
      20106 : writeln(cp_list[i], ' ', _20106);
      20107 : writeln(cp_list[i], ' ', _20107);
      20108 : writeln(cp_list[i], ' ', _20108);
      20127 : writeln(cp_list[i], ' ', _20127);
      20261 : writeln(cp_list[i], ' ', _20261);
      20269 : writeln(cp_list[i], ' ', _20269);
      20273 : writeln(cp_list[i], ' ', _20273);
      20277 : writeln(cp_list[i], ' ', _20277);
      20278 : writeln(cp_list[i], ' ', _20278);
      20280 : writeln(cp_list[i], ' ', _20280);
      20284 : writeln(cp_list[i], ' ', _20284);
      20285 : writeln(cp_list[i], ' ', _20285);
      20290 : writeln(cp_list[i], ' ', _20290);
      20297 : writeln(cp_list[i], ' ', _20297);
      20420 : writeln(cp_list[i], ' ', _20420);
      20423 : writeln(cp_list[i], ' ', _20423);
      20424 : writeln(cp_list[i], ' ', _20424);
      20833 : writeln(cp_list[i], ' ', _20833);
      20838 : writeln(cp_list[i], ' ', _20838);
      20866 : writeln(cp_list[i], ' ', _20866);
      20871 : writeln(cp_list[i], ' ', _20871);
      20880 : writeln(cp_list[i], ' ', _20880);
      20905 : writeln(cp_list[i], ' ', _20905);
      20924 : writeln(cp_list[i], ' ', _20924);
      20932 : writeln(cp_list[i], ' ', _20932);
      20936 : writeln(cp_list[i], ' ', _20936);
      21025 : writeln(cp_list[i], ' ', _21025);
      21027 : writeln(cp_list[i], ' ', _21027);
      21866 : writeln(cp_list[i], ' ', _21866);
      28591 : writeln(cp_list[i], ' ', _28591);
      28592 : writeln(cp_list[i], ' ', _28592);
      28593 : writeln(cp_list[i], ' ', _28593);
      28594 : writeln(cp_list[i], ' ', _28594);
      28595 : writeln(cp_list[i], ' ', _28595);
      28596 : writeln(cp_list[i], ' ', _28596);
      28597 : writeln(cp_list[i], ' ', _28597);
      28598 : writeln(cp_list[i], ' ', _28598);
      28599 : writeln(cp_list[i], ' ', _28599);
      28603 : writeln(cp_list[i], ' ', _28603);
      28605 : writeln(cp_list[i], ' ', _28605);
      29001 : writeln(cp_list[i], ' ', _29001);
      38598 : writeln(cp_list[i], ' ', _38598);
      50220 : writeln(cp_list[i], ' ', _50220);
      50221 : writeln(cp_list[i], ' ', _50221);
      50222 : writeln(cp_list[i], ' ', _50222);
      50225 : writeln(cp_list[i], ' ', _50225);
      50227 : writeln(cp_list[i], ' ', _50227);
      50229 : writeln(cp_list[i], ' ', _50229);
      50930 : writeln(cp_list[i], ' ', _50930);
      50931 : writeln(cp_list[i], ' ', _50931);
      50933 : writeln(cp_list[i], ' ', _50933);
      50935 : writeln(cp_list[i], ' ', _50935);
      50936 : writeln(cp_list[i], ' ', _50936);
      50937 : writeln(cp_list[i], ' ', _50937);
      50939 : writeln(cp_list[i], ' ', _50939);
      51932 : writeln(cp_list[i], ' ', _51932);
      51936 : writeln(cp_list[i], ' ', _51936);
      51949 : writeln(cp_list[i], ' ', _51949);
      51950 : writeln(cp_list[i], ' ', _51950);
      52936 : writeln(cp_list[i], ' ', _52936);
      54936 : writeln(cp_list[i], ' ', _54936);
      57002 : writeln(cp_list[i], ' ', _57002);
      57003 : writeln(cp_list[i], ' ', _57003);
      57004 : writeln(cp_list[i], ' ', _57004);
      57005 : writeln(cp_list[i], ' ', _57005);
      57006 : writeln(cp_list[i], ' ', _57006);
      57007 : writeln(cp_list[i], ' ', _57007);
      57008 : writeln(cp_list[i], ' ', _57008);
      57009 : writeln(cp_list[i], ' ', _57009);
      57010 : writeln(cp_list[i], ' ', _57010);
      57011 : writeln(cp_list[i], ' ', _57011);
      65000 : writeln(cp_list[i], ' ', _65000);
      65001 : writeln(cp_list[i], ' ', _65001);
    else
      Writeln (cp_list[i], ' ', Format(cp_str,[cp_list[i]]));
    end;
  end;
end;
initialization
  cp_list:= TStringList.Create;
  EnumSystemCodePages(@EnumCP, CP_INSTALLED);
finalization
  cp_list.Free;
end.

Lot of thanks to alll for your answers !
I'll now try to make best use of them.

Dominique.