Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2012-06

UTF8 JSON library: was: UTF-8 XML

6 messages · 3 people ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Reinier Olislagers

(Cross-posted to Lazarus and FPC lists)

On the Lazarus list,

Felipe Monteiro de Carvalho wrote 17 lines
Hello, I am using xmlread and dom from FPC to read a XML file and I got an unpleasant surprise. It is converting things like this: <mo>&#x00B1;<!-- &PlusMinus; --></mo> Into: <mo>±</mo> But encoded in ISO 8859-1 which is awful as I don't want my program to go into the dark ages of pre-unicode and all problems it has. I know that we have a XML reading library in Lazarus which uses UTF-8,

Related:

With the help of Ludo Brands - as usual ;) - I've converted the FPC fpjson library to return UTF8 data.

I use it for my twitter/Oauthv1 library/program; you can find it incorporated there at
https://bitbucket.org/reiniero/fpctwit/src
See e.g. fpjsonreadme.TXT
I've renamed the units so they shouldn't conflict with the FPC ones.

Note: I've only had to convert JSON strings to objects, not the other way round; haven't tested that.
No idea whether the RTTI stuff works, haven't used it.

As usual, comments/patches welcome ;)

Thanks,
Reinier

Reply by email

Luiz Americo Pereira Camara
to Reinier Olislagers
Reinier Olislagers wrote 17 lines
Related: With the help of Ludo Brands - as usual ;) - I've converted the FPC fpjson library to return UTF8 data. I use it for my twitter/Oauthv1 library/program; you can find it incorporated there at https://bitbucket.org/reiniero/fpctwit/src See e.g. fpjsonreadme.TXT I've renamed the units so they shouldn't conflict with the FPC ones. Note: I've only had to convert JSON strings to objects, not the other way round; haven't tested that. No idea whether the RTTI stuff works, haven't used it. As usual, comments/patches welcome ;)

Hi,

Thanks for your work. I plan to look at OAuth library soon

About fpjson and UTF8, can you post an example showing the problem?

Maybe i can help.

Are you using fpc 2.7.1?

Luiz

Reply by email

Reinier Olislagers
to Luiz Americo Pereira Camara
Luiz Americo Pereira Camara wrote 22 lines
Em 24/6/2012 07:58, Reinier Olislagers escreveu: > > Related: > > With the help of Ludo Brands - as usual ;) - I've converted the FPC > fpjson library to return UTF8 data. > > I use it for my twitter/Oauthv1 library/program; you can find it > incorporated there at > https://bitbucket.org/reiniero/fpctwit/src > See e.g. fpjsonreadme.TXT > I've renamed the units so they shouldn't conflict with the FPC ones. > > Note: I've only had to convert JSON strings to objects, not the other > way round; haven't tested that. > No idea whether the RTTI stuff works, haven't used it. > > As usual, comments/patches welcome ;) Hi, Thanks for your work. I plan to look at OAuth library soon

No problems, if you have questions, please feel free to ask.

About fpjson and UTF8, can you post an example showing the problem?

The symptom is that the tweets, which are UTF8, get converted to my ANSI codepage on Windows. This means e.g. Greek or Japanese tweets can't be displayed or stored correctly.

As mentioned in
https://bitbucket.org/reiniero/fpctwit/src/efde73d5b1b3/fpjsonreadme.TXT I applied your patch from bug 22197.

Then, adapted line jsonscannerutf8.pp line 244 (FPC 2.7.1 jsonscanner.pp line 238/239), function DoFetchToken
to convert incoming json string data to UTF8 instead of system codepage: e.g.
from:
// Takes care of conversion...
S:=WideChar(StrToInt('$'+S));
to:
// Convert from Unicode codepoint in hex to UTF8... via UTF16: S:=Utf8Encode(WideString(WideChar(StrToInt('$'+S))));

Maybe i can help.

It's working for me right now ;) Just wanted to note that I haven't tested converting objects to JSON output, just JSON strings to objects.

Of course, if you have improvement suggestions, they're very welcome.

Are you using fpc 2.7.1?

I'm using 2.6.1 - however I forked the UTF8 fpjson code from the trunk r21678 version as described in the readme.

Thanks,
Reinier

Reply by email

Luiz Americo Pereira Camara
to Reinier Olislagers
Reinier Olislagers wrote 6 lines
On 24-6-2012 17:05, Luiz Americo Pereira Camara wrote: > Hi, > > Thanks for your work. I plan to look at OAuth library soon No problems, if you have questions, please feel free to ask.

I will

Show quoted text 9 lines
> About fpjson and UTF8, can you post an example showing the problem? The symptom is that the tweets, which are UTF8, get converted to my ANSI codepage on Windows. This means e.g. Greek or Japanese tweets can't be displayed or stored correctly. As mentioned in https://bitbucket.org/reiniero/fpctwit/src/efde73d5b1b3/fpjsonreadme.TXT I applied your patch from bug 22197.

I already read but got curious since that change was not supposed to handle this kind of situation

Show quoted text 10 lines
Then, adapted line jsonscannerutf8.pp line 244 (FPC 2.7.1 jsonscanner.pp line 238/239), function DoFetchToken to convert incoming json string data to UTF8 instead of system codepage: e.g. from: // Takes care of conversion... S:=WideChar(StrToInt('$'+S)); to: // Convert from Unicode codepoint in hex to UTF8... via UTF16: S:=Utf8Encode(WideString(WideChar(StrToInt('$'+S))));

This seems correct

Show quoted text 5 lines
> Maybe i can help. It's working for me right now ;) Just wanted to note that I haven't tested converting objects to JSON output, just JSON strings to objects. Of course, if you have improvement suggestions, they're very welcome.

I isolated the problem and reported the issue. See http://bugs.freepascal.org/view.php?id=22310

Luiz

Reply by email

Mattias Gaertner
to Luiz Americo Pereira Camara
Show quoted text 33 lines
Em 24/6/2012 12:22, Reinier Olislagers escreveu: > On 24-6-2012 17:05, Luiz Americo Pereira Camara wrote: > >> Hi, >> >> Thanks for your work. I plan to look at OAuth library soon > No problems, if you have questions, please feel free to ask. I will > >> About fpjson and UTF8, can you post an example showing the problem? > The symptom is that the tweets, which are UTF8, get converted to my ANSI > codepage on Windows. This means e.g. Greek or Japanese tweets can't be > displayed or stored correctly. > > As mentioned in > https://bitbucket.org/reiniero/fpctwit/src/efde73d5b1b3/fpjsonreadme.TXT > I applied your patch from bug 22197. I already read but got curious since that change was not supposed to handle this kind of situation > Then, adapted line jsonscannerutf8.pp line 244 (FPC 2.7.1 jsonscanner.pp > line 238/239), function DoFetchToken > to convert incoming json string data to UTF8 instead of system codepage: > e.g. > from: > // Takes care of conversion... > S:=WideChar(StrToInt('$'+S)); > to: > // Convert from Unicode codepoint in hex to UTF8... via UTF16: > S:=Utf8Encode(WideString(WideChar(StrToInt('$'+S))));

or more direct:

using lazutf8:

S:=UnicodeToUTF8(StrToInt('$'+S));

Mattias

Reply by email

Reinier Olislagers
to Mattias Gaertner
Mattias Gaertner wrote 16 lines
On Sun, 24 Jun 2012 15:14:36 -0300 Luiz Americo Pereira Camara wrote: > Em 24/6/2012 12:22, Reinier Olislagers escreveu: >> On 24-6-2012 17:05, Luiz Americo Pereira Camara wrote: >> // Takes care of conversion... >> S:=WideChar(StrToInt('$'+S)); >> to: >> // Convert from Unicode codepoint in hex to UTF8... via UTF16: >> S:=Utf8Encode(WideString(WideChar(StrToInt('$'+S)))); or more direct: using lazutf8: S:=UnicodeToUTF8(StrToInt('$'+S));

Thanks, Mattias & Luiz!

Reply by email