Can I offer the attached patch for reading utf8 strings from resource files? It simply converts the utf string to a normal string by discarding the high bits, but I had some files from Delphi or Kylix that were otherwise unreadable.
Colin
Mailing lists / fpc-devel / 2005-03
utf8 reading
17 messages · 6 people · –
- Florian Klaempfl 5
- Uberto Barbini 5
- C Western 4
- DrDiettrich 1
- Marc Weustink 1
- Tomas Hajny 1
Reply by email Subscribe to this list Your subscription
Read-only archive. Use j and k to move between messages.
C Western wroteCan I offer the attached patch for reading utf8 strings from resource files? It simply converts the utf string to a normal string by discarding the high bits,
What would be the correct solution?
Show quoted text 82 lines
but I had some files from Delphi or Kylix that
were otherwise unreadable.
Colin
------------------------------------------------------------------------
diff -uNr fpc/rtl/objpas/classes/classesh.inc fpc.w/rtl/objpas/classes/classesh.inc
--- fpc/rtl/objpas/classes/classesh.inc 2005-03-07 20:43:48.487255015 +0000
+++ fpc.w/rtl/objpas/classes/classesh.inc 2005-03-07 20:53:11.494604156 +0000
@@ -777,7 +777,7 @@
TValueType = (vaNull, vaList, vaInt8, vaInt16, vaInt32, vaExtended,
vaString, vaIdent, vaFalse, vaTrue, vaBinary, vaSet, vaLString,
- vaNil, vaCollection, vaSingle, vaCurrency, vaDate, vaWString, vaInt64);
+ vaNil, vaCollection, vaSingle, vaCurrency, vaDate, vaWString, vaInt64, vaUTF8String);
TFilerFlag = (ffInherited, ffChildPos, ffInline);
TFilerFlags = set of TFilerFlag;
diff -uNr fpc/rtl/objpas/classes/classes.inc fpc.w/rtl/objpas/classes/classes.inc
--- fpc/rtl/objpas/classes/classes.inc 2005-03-07 20:43:48.462262468 +0000
+++ fpc.w/rtl/objpas/classes/classes.inc 2005-03-07 20:53:11.496603567 +0000
@@ -877,6 +877,41 @@
{$endif HASWIDESTRING}
end;
+ function ReadUTF8Str: String;
+ var
+ len, f, t: Integer;
+ begin
+ len := Input.ReadDWord;
+ SetLength(Result, len);
+ if len > 0 then begin
+ Input.Read(Result[1], len);
+ { For now simply take bottom 8 bits of Unicode character }
+ t := 1;
+ f := 1;
+ while f <= len do begin
+ if (Ord(Result[f]) and $80) 0 then begin
+ if (Ord(Result[f]) and %11100000) = %11000000 then
+ Inc(f)
+ else if (Ord(Result[f]) and %11110000) = %11100000 then
+ Inc(f,2)
+ else if (Ord(Result[f]) and %11111000) = %11110000 then
+ Inc(f,3)
+ else if (Ord(Result[f]) and %11111100) = %11111000 then
+ Inc(f,4)
+ else if (Ord(Result[f]) and %11111110) = %11111100 then
+ Inc(f,5)
+ else
+ WriteLn('Bad UTF8 Sequence');
+ Result[t] := Char((Ord(Result[f]) and %111111) or ((Ord(Result[f-1]) and %11) shl 6));
+ end else
+ Result[t] := Result[f];
+ Inc(f);
+ Inc(t);
+ end;
+ SetLength(Result, t-1);
+ end;
+ end;
+
procedure ReadPropList(indent: String);
procedure ProcessValue(ValueType: TValueType; Indent: String);
@@ -988,6 +1023,10 @@
{vaSingle: begin OutLn('!!Single!!'); exit end;
vaCurrency: begin OutLn('!!Currency!!'); exit end;
vaDate: begin OutLn('!!Date!!'); exit end;}
+ vaUTF8String: begin
+ OutString(ReadUTF8Str);
+ OutLn('');
+ end;
else
Stop(IntToStr(Ord(ValueType)));
end;
------------------------------------------------------------------------
_______________________________________________
fpc-devel maillist -
http://lists.freepascal.org/mailman/listinfo/fpc-devel
Florian Klaempfl wrote 8 lines
C Western wrote:
> Can I offer the attached patch for reading utf8 strings from resource
> files? It simply converts the utf string to a normal string by
> discarding the high bits,
What would be the correct solution?
I am showing my ignorance of multi byte characater handling here, but I think it would be to write an escaped sequence as is currently done for wide strings. I don't think the current OutChars function is quite up to it, as utf8 characters can be 32 bits?
Colin
C Western wrote 17 lines
Florian Klaempfl wrote:
> C Western wrote:
>
>> Can I offer the attached patch for reading utf8 strings from
>> resource files? It simply converts the utf string to a normal string
>> by discarding the high bits,
>
>
>
> What would be the correct solution?
I am showing my ignorance of multi byte characater handling here, but
I think it would be to write an escaped sequence as is currently done
for wide strings. I don't think the current OutChars function is quite
up to it, as utf8 characters can be 32 bits?
As a bit of background information as to why some work may ve necessary, the problem I had was with a copyright character in a string property, which some version of Delphi had decided to write out as utf8. Colin
Show quoted text 10 lines
>> What would be the correct solution?
>
> I am showing my ignorance of multi byte characater handling here, but
> I think it would be to write an escaped sequence as is currently done
> for wide strings. I don't think the current OutChars function is quite
> up to it, as utf8 characters can be 32 bits?
As a bit of background information as to why some work may ve necessary,
the problem I had was with a copyright character in a string property,
which some version of Delphi had decided to write out as utf8.
I think that the ultimate solution is to implement Unicode strings, I mean not escaped Unicode, and then convert utf-8 to unicode string. I'm not sure how Delphi manage that, Java has Unicode string.
Bye Uberto
Uberto Barbini wrote 15 lines
>>>What would be the correct solution?
>>
>>I am showing my ignorance of multi byte characater handling here, but
>>I think it would be to write an escaped sequence as is currently done
>>for wide strings. I don't think the current OutChars function is quite
>>up to it, as utf8 characters can be 32 bits?
>
>As a bit of background information as to why some work may ve necessary,
>the problem I had was with a copyright character in a string property,
>which some version of Delphi had decided to write out as utf8.
I think that the ultimate solution is to implement Unicode strings, I mean not
escaped Unicode, and then convert utf-8 to unicode string.
I'm not sure how Delphi manage that, Java has Unicode string.
utf-8 is a unicode string. What do you mean with "not escaped" Unicode? Memory wasting utf-32?
On Thu, 10 Mar 2005 08:43:07 +0100, Florian Klaempfl wrote
Show quoted text 15 lines
Uberto Barbini wrote:
>>>>What would be the correct solution?
>>>
>>>I am showing my ignorance of multi byte characater handling here, but
>>>I think it would be to write an escaped sequence as is currently done
>>>for wide strings. I don't think the current OutChars function is quite
>>>up to it, as utf8 characters can be 32 bits?
>>
>>As a bit of background information as to why some work may ve necessary,
>>the problem I had was with a copyright character in a string property,
>>which some version of Delphi had decided to write out as utf8.
>
>
> I think that the ultimate solution is to implement Unicode strings, I mean
not
Show quoted text 5 lines
> escaped Unicode, and then convert utf-8 to unicode string.
> I'm not sure how Delphi manage that, Java has Unicode string.
utf-8 is a unicode string. What do you mean with "not escaped"
Unicode? Memory wasting utf-32?
I'd see UCS2/UCS4 as not escaped Unicode.
Tomas
> utf-8 is a unicode string. What do you mean with "not escaped" > Unicode? Memory wasting utf-32? I'd see UCS2/UCS4 as not escaped Unicode.
Exactly.
I think that UCS2 will suffice to everyone not interested in Vogon poetry! ;)) This means to have 2 byte chars and related strings, all string functions made compatible and function to import/export to utf-8 and charset strings.
Another way is to use a flag to force all string to be compiled as UCS2 (or UCS4 for Vogon poets).
Using natively utf-8 I think is impossible, because the encoding.
Please note that at every Borland conference there is someone asking for Unicode support since Delphi2...
There are several opensource library for managing unicode strings in delphi but they are implemented as standard classes, not refcounted first class citizen as long-string.
Bye Uberto
Uberto Barbini wrote 10 lines
>>utf-8 is a unicode string. What do you mean with "not escaped"
>>Unicode? Memory wasting utf-32?
>
>I'd see UCS2/UCS4 as not escaped Unicode.
Exactly.
I think that UCS2 will suffice to everyone not interested in Vogon poetry! ;))
This means to have 2 byte chars and related strings, all string functions made
compatible and function to import/export to utf-8 and charset strings.
UCS-2 or UTF-16 how it called by the unicode consortium is "escaped" as well and you've to take care of it in your code. Even in utf-32 you've to take care of surrogate pairs.
Show quoted text 5 lines
Another way is to use a flag to force all string to be compiled as UCS2 (or
UCS4 for Vogon poets).
Using natively utf-8 I think is impossible, because the encoding.
Why?
Show quoted text 7 lines
Please note that at every Borland conference there is someone asking for
Unicode support since Delphi2...
There are several opensource library for managing unicode strings in delphi
but they are implemented as standard classes, not refcounted first class
citizen as long-string.
UCS-2 or UTF-16 how it called by the unicode consortium is "escaped" as well and you've to take care of it in your code.
mmh, no.
UCS-2 is different from utf-16 (which is escaped), but you cannot represent all utf characters (see the case of Vogon poetry).
See:
http://www.uazone.com/multiling/unicode/ucs2.html
http://lists.samba.org/archive/jcifs/2002-July/000969.html
Even in utf-32 you've to take care of surrogate pairs.
In utf-32 yes, in UCS-4 no.
Teorically we could have a UCS-8 in the future, but for the next hundred years this is not very likely.
> Using natively utf-8 I think is impossible, because the encoding. Why?
Because every simple function on strings (like copy) should require to start reading the string from beginning
Bye Uberto
From:
Behalf Of Uberto
Barbini
Sent: donderdag 10 maart 2005 19:30
Show quoted text 5 lines
> UCS-2 or UTF-16 how it called by the unicode consortium is "escaped" as
> well and you've to take care of it in your code.
mmh, no.
yes :)
UCS-2 is different from utf-16 (which is escaped),
nope. UCS originally wasn't escaped, but they changed that "recently"
Marc
Marc Weustink wrote 18 lines
From:
Behalf Of Uberto
Barbini
Sent: donderdag 10 maart 2005 19:30
>>UCS-2 or UTF-16 how it called by the unicode consortium is "escaped" as
>>well and you've to take care of it in your code.
>
>mmh, no.
yes :)
>UCS-2 is different from utf-16 (which is escaped),
nope. UCS originally wasn't escaped, but they changed that "recently"
Indeed, there is virtually no difference between ucs-2 and utf-16.
From: Behalf Of Uberto Barbini Sent: donderdag 10 maart 2005 19:30 yes :) nope. UCS…
>UCS-2 is different from utf-16 (which is escaped), nope. UCS originally wasn't escaped, but they changed that "recently"
Could you point me to a official document?
In the Unicode site I don't find anything.
Anyway with escaping UCS2 when UCS2 definition is utf-16 without encoding?
Bye Uberto
Uberto Barbini wrote 7 lines
> >UCS-2 is different from utf-16 (which is escaped),
>
> nope. UCS originally wasn't escaped, but they changed that "recently"
Could you point me to a official document?
In the Unicode site I don't find anything.
Anyway with escaping UCS2 when UCS2 definition is utf-16 without encoding?
ok, I found it in the FAQ at Unicode site.
UCS2 is the old standard, before the need for escaping, now is considered obsolete by utf-16.
So one can use UCS2 and do renunce to some asian alphabeths or use UCS4 that is not escaped, at least in Unicode version 4.1 and in foreable future. Now Unicode use 21 bits of 32.
Bye Uberto
This discussion has inspired me to think a little more about what should be done in this specfic context, and I think the attached patch should do it. It is a more drastic patch than the previous one, and I don't have any widestring test material, but it passes some simple tests. I have copied some code from the utf8bidi unit - the functionality I needed was not exported from the unit (but should be?) Colin
Exactly. I think that UCS2 will suffice to everyone not interested in Vogon poetry! ;))…
Uberto Barbini wroteUsing natively utf-8 I think is impossible, because the encoding.
Support might be implmemented like/in MBCS support.
Please note that at every Borland conference there is someone asking for Unicode support since Delphi2...
Not only for Delphi ;-)
There are several opensource library for managing unicode strings in delphi but they are implemented as standard classes, not refcounted first class citizen as long-string.
It's not easy to find a solution suitable for everybody. There exist so many character encodings, a single class or data type hardly will cover all of them. Windows users may be happy with utf-16 (WideChar/WideString) because that's supported by the OS and some of its standard controls, but other OS have different models and support.
As long as strings are created, used, and stored by an application, I'd suggest to use utf-8 for the external (disk file) representation, and WideString in the application. Then only two procedures are required to convert between utf-8 and WideString. Strings from other sources then have to be converted by the appropriate procedure into WideString, where the coder is responsible for the selection of the appropriate conversion; then a general library of such conversion procedures can be created and maintained for use in Pascal programs, or the coder can use his preferred opensource library.
Some coders may prefer WideString also in disk files, if utf-8 files would be bigger in their natural or preferred language. Of course an application also can continue to use AnsiString instead of WideString, if Unicode support is not required. All these selections are up to the coder, the required data types and conversions are already supported.
A compiler may support an Unicode switch, that maps the general data types Char and String into either AnsiChar/AnsiString or WideChar/WideString, in order to support easily portable code. The switch may be extended to map WideChar into utf-16, utf-32, utf-64, or whatever will become available in future compiler versions. A similar effect can be achieved by user-defined data types TChar and TString in portable code, with the possible problem that there exists no standard unit where these data types can be defined unambiguously, throughout whole projects.
DoDi
This discussion has inspired me to think a little more about what should be done in this…
C Western wrote:
Applied.