Hi,

Following up on bug 22310
http://bugs.freepascal.org/view.php?id=22310

I enabled the use of UTF8 in the FPC JSON support.

The constructors of the JSON parser/scanner now accept an extra argument UseUTF8 which tells them to convert JSON strings to UTF8, not the system codepage.

This should make the jsonscannerUTF8 unit that is floating around redundant.

Michael.

I don't understand why you want to keep buggy behavior for backwards compatibility. I explain:
StringToJSONString doesn't do any character conversion except for some special characters. The json spec rfc4627 par 3 says: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8." Since no conversion is done logic says that fpjson expects unicode. Since TJSONStringType = AnsiString, UTF-8 is the only unicode encoding possible. So I don't understand why writing a TJSONStringType has to be utf8 to be compliant with the spec and with the outside world and when reading the same string back it is converted in system encoding.
If system encoding support is needed then StringToJSONString should do also a system to utf8 encoding.

Ludo

Because the old behaviour is not "buggy".

It simply did not support Unicode, and does the next best thing, in casu: it transforms to the system codepage.

A car without ABS and SAT-Nav is not buggy.
It just doesn't support features which are nowadays called standard. You can perfectly drive it.

Just as Unicode is nowadays standard, N years ago, it was not.

Since backwards compatibility is very important, we must offer the option.

Michael.

An old car without a seat belt is buggy although when it was built seat belts where not standard.

The Json standard specifies unicode. Why have utf8 default off? Old cars are retro-fitted with seat belts. Why sell new cars with seat belts as an option?

Ludo

I still wouldn't call the car buggy.
For me a car must be able to drive. That is its essential function. If it doesn't start, or stops every 5 minutes, then it is buggy. All the rest are options. Whether they are legally required or not is irrelevant to the car-ness of the car.

But I drive a motorcycle, so the point is moot ;-)

For backwards compatibility by default.

Old cars are retro-fitted with seat belts. Why sell new cars with seat belts as an option?

So you'd reverse the constructor boolean argument to specify Utf8 as default, and let the user choose the old behaviour if he needs it ?

Michael.

So you'd reverse the constructor boolean argument to specify Utf8 as default, and let the user choose the old behaviour if he needs it ?

If that is "unthinkable" then define new contructors TJSONParser.Create2(...,AUseUTF8 : Boolean = True) or Create2(...,AUseUTF8 : Boolean = True) and mark the Create(...) as deprecated. Or deprecate TJSONParser and make it a descendant of TJSONParser2 that has Create(...,AUseUTF8 : Boolean = True) constructors.

Ludo

Nothing is unthinkable. The other constructs are very ugly.

I reversed the argument default value to True. UTF8 is now the default.

Michael.

A very wise decision;)

Ludo

I must be getting older :)

Seriously: when json support was written, Unicode/UTF8 support in FPC was makes sense to change it, but backward compatibility must be preserved as much as possible.

I have added a section to

http://wiki.freepascal.org/index.php?title=User_Changes_Trunk

to notify people of this change.

Michael.

Could you change it to use the same format/template as the other entries?

Jonas

We can always trust Jonas to notice such things :-)

Done.

Michael.

In our previous episode, Michael Van Canneyt said:

Jonas remark reminded me of something else. Delphi uses TEncoding everywhere as parameter (specially load/savetofile/stream like methods) to select codepage. Maybe do that too here?

Michael Van Canneyt wrote

I reversed the argument default value to True. UTF8 is now the default.

I'm very happy with this, thanks Michael!

Regards,
Reinier (looking forward to dumping my private copy of FPJSON)

Agree with Ludo

Luiz

It's.

With the old behavior, in an system with a system code page UTF8, if i try to show the parsed value of "\u4E01" in e.g. a LCL app will get garbage.

I would expect to work correctly in any enviroment

Luiz

Luiz Americo Pereira Camara wrote

With the old behavior, in an system with a system code page UTF8, if i try to show the parsed value of "\u4E01" in e.g. a LCL app will get garbage. I would expect to work correctly in any enviroment

this means that some environments will end up with "garbage" for those UTF-8 characters that cannot be translated back to the local codepage... i've been running headlong into this with another project and needing to convert from UTF-8 back to at least CP437... there are more than 255 characters in UTF-8 and there's no way i know of to translate them all back to 255 characters... even with trying to use multiples like ae for æ ( alt-145 in CP437 i think realizing that this editor can do whatever it wants to :/ )... the doublet and the character i typed the ones i was thinking of for this example, though...

And your point is?
This change moves the conversion problem from with the JSON library to your responsibility. Please note that these same characters in a JSON string would have ended up as garbage already in the old situation.

In the new situation, you have some more control over it. If you don't like the new behaviour, you can always set the conversion flag in the constructor to false instead of its default true... and move back to the earlier "conversion in a black box" method.

However, if you don't have a need to convert to system codepage, as you say, you could lose information because the conversion is lossy. That has been fixed (and correctly so, IMO).
For instance, I can now get a UTF8 JSON string, and write it out to a UTF8 File, process it with the FPC UTF8 functions, show it in a Lazarus TEdit or grid, etc.

Thanks,
Reinier

In the previous behavior (conversion UTF16 -> system code page) you will get a meaningless "character" anyway, i.e., those unicode characters are not correctly translated to the system code page correctly since is impossible.

BTW: the original issue is already fixed. Thanks Michael

Luiz

that there will be data loss when converting from UTF-8 to codepages with less characters than UTF-8 has... that is what i stated and i thought it was an answer to the poster's question/problem of conversion...

This change moves the conversion problem from with the JSON library to your responsibility. Please note that these same characters in a JSON string would have ended up as garbage already in the old situation.

and they'll still end up as garbage anyway... especially if they do not exist in the current codepage and there's no multiple characters format to represent them...

maybe we're talking the same basic point to each other? ;)

<snip>

maybe we're talking the same basic point to each other? ;)

Could be, but I had the feeling you failed to see the advantage of not doing an unicode=>codepage conversion inside the FPJSON code if you don't want codepage output.

Actually, it was me who had trouble with the original setup of FPJSON discarding information because it did the unicode=>codepage conversion regardless. I wanted to get UTF8 and could never get that.

My solution was to modify (a tiny bit) and fork FPJSON for my twitter/OAuth v1 library.

To my pleasant surprise, Luiz Americo posted bug 22310 based on that and fortunately Michael Van Canneyt adapted fpjson so I can use it for e.g. my twitter code... and retire the fork once FPC trunk has been released as stable and is widely available (solved it now with a conditional unit inclusion).

So thanks to Luiz & Michael, and as always comments/patches welcome: https://bitbucket.org/reiniero/fpctwit
... or new applications based on this; oauthv1 is apparently also used by Yahoo and other providers...

Leledumbo on the forum has e.g. written an application for plurk.com with the library:
https://bitbucket.org/leledumbo/fpctwit
... and he said he intended to adapt it for use with Facebook sometime...

Thanks,
Reinier