I wanted to get the status of WideString support. Specifically Multi-lingual (Ansi/UTF8 strings) using Threads.

I noticed that FPC is now automatically converting (encoding/decoding) the contents of a string during various operations. These conversions seem to be using system locale settings as a basis for selecting which code page it uses.

I did see that I can change the default code page, but that is not acceptable where incoming MIME data is concerned. With MIME messages - the content type and character set are passed along with each section of text. Blocks specifiy with which code page to employ.

The problem I have is that I process content of many different code pages in parallel. String manipulation occurs at a mult-threaded level so switching global variables is going to expose the current way of processing as dangerous.

What are the best practices for taking a chunks of string data (of various code pages), and converting to UTF8 given a multi-threaded application?

Supporting thread-safe system-wide code page string manipulation is a basic requirement moving forward.

If interested, I can contribute a converter I wrote that offers encoding/decoding of strings as perhaps some inspiration that may lead to FPC internal support for such a mechanism.

Thanks,

Andrew Brunner
Aurawin LLC

512.574.6298

https://aurawin.com

Andrew Brunner wrote

I did see that I can change the default code page, but that is not acceptable where incoming MIME data is concerned.

That global variable specifies the default code page for the plain ansistring type. It is not intended to be changed all the time in the first place. You can read your data into a RawByteString and then call SetCodePage(rawbytestr,codepagenr,false) to set its code page to whatever the code page of the data in that string is (without attempting to convert the data, which is what the last "false" parameter indicates). You can then convert it into UTF-8 by calling SetCodePage(rawbytestr,CP_UTF8), or the equivalent SetCodePage(rawbytestr,CP_UTF8,true) (the last parameter is a default parameter with value "true")

Jonas

Thanks Jonas, I'm relieved to see there is already mechanisms in place.

You can then convert it into UTF-8 by calling SetCodePage(rawbytestr,CP_UTF8), or the equivalent SetCodePage(rawbytestr,CP_UTF8,true) (the last parameter is a default parameter with value "true")

That looks exactly what is needed.

Thanks again!

What is supposed to happen when you uses such thingies a string "+" operation (to be added to each other, with a "normal" string or with a string constant ?

What happens when using it in a string "copy" or "pos" function ?

What happens when struing it in a TStringList ?

(OK I can try all this myself ...)
-Michael