ChatGPT is saying I can print unicode scalars like that but i don’t see it works and no compiler warnings even. Did it make this up or did I do something wrong?

  Writeln('Unicode scalar 1F496: ', #$1F496); // 💖
  Writeln('Unicode scalar 1F496: ', WideChar($1F496));  // 💖

Regards,
Ryan Joseph

This works for me under Linux (Fedora), if I include the unit cwstring. Haven't tested other platforms.

  Writeln('Unicode scalar 1F496: ', WideChar($1F496));  // 💖

This one doesn't work. WideChar is 16-bit, and $1F496 doesn't fit. It also produces a warning:

 Warning: range check error while evaluating constants (128150 must be between 0 and 65535)

This is because a WideChar represents a single UTF-16 code unit. To encode $1F496, you need two UTF-16 code units (high and low surrogate):

  Writeln('Unicode scalar 1F496: ', WideChar((($1F496-$10000) shr 10)+$D800)+WideChar((($1F496-$10000) and $3FF)+$DC00));  // 💖

More info here:

https://en.wikipedia.org/wiki/UTF-16#Code_points_from_U+010000_to_U+10FFFF

Nikolay

What people call "Unicode", even compiler manuals, is not "Unicode". I repeat it again and again, it is not "Unicode" but so-called "Unicode". Microsoft, and those who want to be compatible with it, use UTF-16 <https://en.wikipedia.org/wiki/UTF-16> treated as if it were UCS-2 <https://en.wikipedia.org/wiki/Universal_Coded_Character_Set>

They call that "Unicode", which is plain nonsense. In the real world, one can not stuff 21-bits into 16-bits.

For heaven's sake, let's stop talking about so-called "Unicode" and instead use UTF-8 <https://en.wikipedia.org/wiki/UTf-8> or UTF-32 <https://en.wikipedia.org/wiki/UCS-4>.

Regards,

Adriaan van Os

Here's how Free Pascal types map to Unicode terminology:

WideChar = UTF-16 code unit

UnicodeString = UTF-16 encoded string

WideString = UTF-16 encoded string. On Windows it's not reference counted - used for COM compatibility. On other platforms, it's the same as UnicodeString.

UTF8String = UTF-8 encoded string. Defined as UTF8String=type AnsiString(CP_UTF8).

UTF16String = alias for UnicodeString

Hope this clears things up.

Another thing:

For conversions between different encodings to work (e.g. between UTF-8 and UTF-16), you need to install a widestring manager. Some platforms (like Windows) always include one by default, but other platforms (e.g. Linux) don't, in order to reduce bloat, for programs that don't need it. For these, you may need to include unit cwstring or something like that.

Nikolay

On Dec 1, 2024 at 2:23:08 PM, Nikolay Nikolov via fpc-pascal <

Including that unit is sneaky, seems you need it anytime dealing with unicode. Not sure how it even knows to change the meaning of those character constants.

Using the term “char” was maybe a mistake. This misleads people into thinking it’s a “character” as they perceive it but really it’s just a code point. Why isn’t there a “UnicodeChar” type which is 4 bytes and hold a full UTF-8 character? That’s probably what most people are expecting when they think “unicode” and “character”. Their are still compound characters which appear as one but actually multiple overlayed but still getting the component parts is useful.

Choosing UTF-16 for UnicodeString was probably a mistake too. It’s my understanding all websites are UTF-8 which means this encoding will dominate everything. I think UTF-8 is by far the most used right?

As a user I would expect to take a string constant and assigning it to a UnicodeString would let me iterate over UnicodeChar. That’s logical right? Maybe this is just left undone as of now. I don’t know.

var
  u: UnicodeChar;
  s: UnicodeString;
begin
  s := 'Hello, 🌎!';
  for u in s do
    writeln(u);

Regards,
Ryan Joseph

On Dec 1, 2024 at 2:02:06 PM, Adriaan van Os via fpc-pascal <

Good point, I read this very good article and he states unicode without encoding is nonsense. Why not UTF-16 though? I think next to UTF-8 it’s the most common right?

https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

Regards,
Ryan Joseph

Indeed you CAN print Unicode characters, but not with the normal windows command prompt, nor with powershell, that is where the problem is, not with FreePascal. But if you go to the Windows Store and download the free “Windows Terminal” there you can get it to work.

When you run the windows terminal, you get a powershell window, and ‘+’ gets you a new powershell window, but you can push the little down arrow next to the + and get a Command Prompt… there is a way to make the command prompt the default in the settings.

If you run your pascal program inside the command prompt window of Windows terminal, you will find that you get full color Unicode characters.

I just tested it on Windows 10 and the following work:
   Writeln('Unicode scalar 1F496: ', #$1F496); // 💖
   Writeln('Unicode scalar 1F496: 💖');  //
This one does not work:
   Writeln('Unicode scalar 1F496: ', WideChar($1F496));  // 💖

Note that these only work if you do NOT use the CRT unit.

James

There is nothing sneaky about this. This is simply how things work in FPC to avoid linking against the C-library (or including quite a load of Unicode data in case of fpwidestring instead of cwstring) when for much code it isn't necessary (just like the need to use unit cthreads on *nix-systems to install the threading manager).

Using the term “char” was maybe a mistake. This misleads people into thinking it’s a “character” as they perceive it but really it’s just a code point.

There isn't much choice here, cause that type name exists from old Pascal times and that will not change (well, okay, it will change in so far as when the Unicode RTL is enabled it will be Char = WideChar instead of Char = AnsiChar as it is now).

Why isn’t there a “UnicodeChar” type which is 4 bytes and hold a full UTF-8 character?

There is, it's called UCS4Char. Also it's not a "full UTF-8 character", but simply a "Unicode code point".

Choosing UTF-16 for UnicodeString was probably a mistake too.

Take that up with Borland, cause they termed it as "UnicodeString". Which is mainly because they only had to deal with Windows compatibility where there either were the single Byte encodings or the UTF-16 encoding.

It’s my understanding all websites are UTF-8 which means this encoding will dominate everything. I think UTF-8  is by far the most used right?

UTF-8 is usually used for textual encoding, because it is the most memory dense Unicode encoding, however many languages or runtimes including JavaScript, Java's JVM, the .Net CLR, Windows, Qt, UEFI and Delphi >= 2009 use UTF-16 internally.

Here you go:

=== code begin ===

program tstrenum;
{$codepage utf8}
{$mode objfpc}{$H+}
{$modeswitch advancedrecords}
type
   TUCS4CharUnicodeStrEnumerator = record
   private
     fStr: UnicodeString;
     fIndex: SizeInt;
     fCurrent: UCS4Char;
   public
     constructor Create(const aStr: UnicodeString);
     function MoveNext: Boolean;
     property Current: UCS4Char read fCurrent;
   end;
constructor TUCS4CharUnicodeStrEnumerator.Create(const aStr: UnicodeString);
begin
   fStr := aStr;
   fIndex := -1;
   fCurrent := 0;
end;
function TUCS4CharUnicodeStrEnumerator.MoveNext: Boolean;
begin
   Inc(fIndex);
   if fIndex > Length(fStr) then
     Exit(False);
   if (Ord(fStr[fIndex]) >= $D800) and (Ord(fStr[fIndex]) <= $DBFF) then
begin
     if fIndex < High(fStr) then begin
       if (Ord(fStr[fIndex + 1]) >= $DC00) and (Ord(fStr[fIndex + 1]) <=
$DFFF) then begin
         fCurrent := UCS4Char(Ord(fStr[fIndex]) - $D800) shl 10 +
UCS4Char(Ord(fStr[fIndex + 1])) - $DC00 + $10000;
         Inc(fIndex);
       end else
         fCurrent := Ord(fStr[fIndex]);
     end else
       fCurrent := Ord(fStr[fIndex]);
   end else
     fCurrent := Ord(fStr[fIndex]);
   Result := True;
end;
operator Enumerator(const aStr: UnicodeString):
TUCS4CharUnicodeStrEnumerator;
begin
   Result := TUCS4CharUnicodeStrEnumerator.Create(aStr);
end;
var
   s: UnicodeString;
   u: UCS4Char;
begin
   s := 'Hello, 🌎!';
   for u in s do
     Writeln(HexStr(Ord(u), 8));
end.

=== code end ===

Regards,
Sven