Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2019-08

Tests results of several pascal based JSON parsers

19 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Anthony Walter

I've posted a new page that tests the speed and correctness of several pascal based JSON parsers.

https://www.getlazarus.org/json/tests/

In full disclosure I am the author of the new open source JsonTools library, and even though my parser seems to a big improvement over the other alternatives, my tests were not biased.

If anyone would like help in replication the tests, let me know and I'll see what I can do.

Also, to be thorough, you should read through both the article I posted at the top this message, and my original page <https://www.getlazarus.org/json> which has been updated with more information. Both pages took some time to write, and I promise if you read through them some of your questions will be answered without having to ask others for help or insight.

Thanks,
Anthony

Reply by email

Michael Van Canneyt
to Anthony Walter: I've posted a new page that tests the speed and correctness of several pascal based JSON…
Anthony Walter wrote 8 lines
I've posted a new page that tests the speed and correctness of several pascal based JSON parsers. https://www.getlazarus.org/json/tests/ In full disclosure I am the author of the new open source JsonTools library, and even though my parser seems to a big improvement over the other alternatives, my tests were not biased.

Test are always biased in the sense that they depend on the proficiency of the coder. you must ask someone with sufficient knowledge to write the code.

The shootout benchmarks for example are dismally coded for FPC with as a result that they perform badly.

So it could well be that the fpJSON results for example can be improved a lot by changing the method used to a more efficient one.

Also, not every library is designed with the same goals. fpjson could probably be made smaller if the goal was more focused. it has a factory pattern, added on behalf of users who asked for this. jsontools does not have this.
You can format the output. JSONTools does not have this. fpJSON was designed to be pluggable wrt. parsing. JSONTools is not.

In short fpjson can do a lot more than JSONtools.
Some things come at a price, others not.

In that sense, tests like this compare apples with pears.

Show quoted text 8 lines
If anyone would like help in replication the tests, let me know and I'll see what I can do. Also, to be thorough, you should read through both the article I posted at the top this message, and my original page <https://www.getlazarus.org/json> which has been updated with more information. Both pages took some time to write, and I promise if you read through them some of your questions will be answered without having to ask others for help or insight.

Can you please send me the testcode you used for your speed & correctness tests ?

I'm a bit surprised to see fpJSON fail in unicode correctness. It's been tested extensively, maybe your code contains errors (or maybe fpjson does, I'll need to check).

Also please explain what 'Handling duplicate key names correctly' means to you. Saying that a library fails a test without specifying what the test is, is strange.

Michael.

Reply by email

Michael Van Canneyt
Anthony Walter wrote 6 lines
With regards to duplicate key names, some libraries allow for the same key to be parsed resulting in multiple child nodes of the same name. Others throw an exception when parsing an object with a duplicate key name. The correct way to handle duplicate keys is to overwrite the existing key when a duplicate is encountered.
There you go. I think the "correct way" is to raise an error;
not to override and thus inadvertently lose previous data.

I won't argue on who is correct, since it is a matter of opinion.

But this is a prime example of 'biased' tests.
You're testing an opinion, not actual functionality.

so IMHO it would be only fair to remove it from your comparison.

For speed & correctness, I repeat my request:
please provide your test code.

Michael.

Reply by email

Alan Krause
to Anthony Walter: I've posted a new page that tests the speed and correctness of several pascal based JSON…

I'd recommend that you also list what license each is licensed under, as that can be an extremely large deciding factor in whether or not a given implementation can even be considered.

Alan

AM Anthony Walter wrote 22 lines
I've posted a new page that tests the speed and correctness of several pascal based JSON parsers. https://www.getlazarus.org/json/tests/ In full disclosure I am the author of the new open source JsonTools library, and even though my parser seems to a big improvement over the other alternatives, my tests were not biased. If anyone would like help in replication the tests, let me know and I'll see what I can do. Also, to be thorough, you should read through both the article I posted at the top this message, and my original page <https://www.getlazarus.org/json> which has been updated with more information. Both pages took some time to write, and I promise if you read through them some of your questions will be answered without having to ask others for help or insight. Thanks, Anthony

Reply by email

Anthony Walter
to Alan Krause

Alan, oh that's a good idea. I will do that as well as add a few more parser libraries as requested by a few people in other non mailing lists threads. I will also try to find out what's going on the unicode strings as it might be a problem with the compiler.

Michael,

I am on Linux as well, but I will test under Windows and Mac too.

Reply by email

Michael Van Canneyt
to Anthony Walter
Anthony Walter wrote 8 lines
Alan, oh that's a good idea. I will do that as well as add a few more parser libraries as requested by a few people in other non mailing lists threads. I will also try to find out what's going on the unicode strings as it might be a problem with the compiler. Michael, I am on Linux as well, but I will test under Windows and Mac too.

To show that my argument of 'coding proficiency' influence on algorithm speed is not complete nonsense, I quickly cooked up the following test:

{$mode objfpc}
{$h+}
uses DateUtils, Sysutils,Classes, fpjson, jsonparser;
var
   FN : String;
   i,aCount : Integer;
   S : TJSONStringType;
   T : TMemoryStream;
   N : TDateTime;
procedure ReadJSON;
begin
   T:=TMemoryStream.Create;
   T.LoadFromFile(FN);
   SetLength(S,T.Size);
   T.ReadBuffer(S[1],T.Size);
   T.Position:=0;
end;
begin
   if ParamCount2 then Halt(1);
   aCount:=StrToInt(Paramstr(1));
   FN:=ParamStr(2);
   ReadJSON;
   try
     Writeln('Reading string ',aCount,' times');
     N:=Now;
     for I:=1 to aCount do
       GetJSON(S).Free;
     Writeln('Msecs : ',MillisecondsBetween(Now,N));
     Writeln('Reading stream ',aCount,' times');
     N:=Now;
     for I:=1 to aCount do
       begin
       GetJSON(T).Free;
       T.Position:=0;
       end;
     Writeln('Msecs : ',MillisecondsBetween(Now,N));
   finally
     T.Free;
   end;
end.

When you run this:

home:~/fpc/packages/fcl-json/tests> ./testjsonspeedread 100 ./testdata.json Reading string 100 times
Msecs : 2972
Reading stream 100 times
Msecs : 1203

(file of 260Kb, 500 lines)

Not using a string (as you do) but a stream already gives a factor of 2.x faster. The speed gain is there both for trunk as for 3.0.4.

So, I'm fairly confident that I can probably speed up your test results as well, when you send me the sources.

That said, this is not to say that there is no room for speed improvements in fpjson.

I've already identified 2 places where speed gains can be made in the fpJSON codebase, I'll improve the codebase this weekend and post results.

Michael.

Reply by email

Anthony Walter

Okay, so I turned on my Windows VM with a different version of FPC and ran VerifyUnicodeChars with both FPJson and JsonTools. The resutls are the same. JsonTools sees the unicode correctly, and something is wrong when using FPJson. I don't know what the problem is, but other people are noticing similar issues, so it would seem there is definitely a problem resulting in a failure for FPJson.

Michael, you have all the information needed to find out what's wrong and I'd be curious to learn why it's not working.

Reply by email

Anthony Walter

For those tracking the unicode issue, could you please verify the problem does not present in my JsonTools library on compiler revisions and platforms? I always get true (passed) with my library, but not with any other library. Here is the relevant test:

function VerifyUnicodeChars: Boolean;
const
  UnicodeChars = '{ "name": "Joe®Schmoe", "occupation": "bank teller \u00Ae
" }';
var
  N: TJsonNode;
begin
  N := TJsonNode.Create;
  N.Parse(UnicodeChars);
  Result := (N.Child(0).AsString = 'Joe®Schmoe') and (N.Child(1).AsString =
'bank teller ® ');
  N.Free;
end;
begin
  WriteLn('Handles unicode characters correctly: ', VerifyUnicodeChars);
end.

Reply by email

Ben Grasset
to Michael Van Canneyt: Test are always biased in the sense that they depend on the proficiency of the coder. you…
AM Michael Van Canneyt wrote
The shootout benchmarks for example are dismally coded for FPC with as a result that they perform badly.

Not all of them! At least, not anymore:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/binarytrees.html

More recently I also managed to bring the execution time for "Mandelbrot" up to 8.52 seconds for FPC, whereas the version that had been around for a long time before that ran in 22 seconds:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-fpascal-8.html

I'm hoping to figure out how to bring it up even more, but I think I might need to wait for 3.2.0 for that (loop unrolling on by default at -O3, e.t.c.)

Reply by email

Michael Van Canneyt
to Anthony Walter: Okay, so I turned on my Windows VM with a different version of FPC and ran…
Anthony Walter wrote 9 lines
Okay, so I turned on my Windows VM with a different version of FPC and ran VerifyUnicodeChars with both FPJson and JsonTools. The resutls are the same. JsonTools sees the unicode correctly, and something is wrong when using FPJson. I don't know what the problem is, but other people are noticing similar issues, so it would seem there is definitely a problem resulting in a failure for FPJson. Michael, you have all the information needed to find out what's wrong and I'd be curious to learn why it's not working.

Allow me to correct you, I don't have all information:

1. Did you run my provided test program on linux ? (the first one I sent, not the one for the speed test)

If my test program also shows different results on your machine, indeed something strange is going on.

The reason I insist on the us of my test program and not yours, is that the result can be influenced by some unknown units or whatnot in yours, and my program is "bare bones".

On the assumption my private computer can be somehow 'compromised' by years of FPC development I even copied my program to 2 other linux machines, used for production work, and the result is 'True' on both. (see log below for one of them, this is with standard ubuntu installed compiler)

I can believe the program would output something different on Windows, but not on another linux box.

2. Where is the source code of your test program(s) ? At least the ones for fpjson and jsontools.

Without your actual source code, I cannot give advice or investigate properly.

I'd like to see this cleared up. fpjson has been in use in many REST services in large production sites for at least 8 years, these services definitely use UTF8 content outside the basic ASCII or even ANSI codepages.

So the failures you see are highly surprising to me, to say the least.

Michael.

Copy&paste from a quick sompile session.
----
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-34-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

* Keen to learn Istio? It's included in the single-package MicroK8s.

https://snapcraft.io/microk8s

* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at: https://ubuntu.com/livepatch
_____
/ ___/___ _ _ _____ _ ___ ___
| | / _ \| \| |_ _/ \ | _ )/ _ \
| |__| (_) | .` | | |/ _ \| _ \ (_) |
\____\___/|_|\_| |_/_/ \_|___/\___/

Welcome!

This server is hosted by Contabo. If you have any questions or need help, please don't hesitate to contact us at

Last login: Tue Aug 20 14:23:39 2019 from 81.82.199.218 fpc twalter.pas -S2
Free Pascal Compiler version 3.0.4+dfsg-18ubuntu1 [2018/07/02] for x86_64 Copyright (c) 1993-2017 by Florian Klaempfl and others Target OS: Linux for x86-64
Compiling twalter.pas
Linking twalter
/usr/bin/ld.bfd: warning: link.res contains output sections; did you forget -T? 26 lines compiled, 0.4 sec
./twalter
Handles unicode chars correctly: >{ "name": "Joe®Schmoe", "occupation": "bank teller \u00Ae " }< TRUE

Reply by email

Benito van der Zander
to Michael Van Canneyt: Test are always biased in the sense that they depend on the proficiency of the coder. you…

Hi,

when I need maximal speed, I use jsonscanner of fpJSON. It should be vastly faster than a complete parser by not allocating memory or validating the json.

Although it decodes all the strings in the JSON, which require some allocations, even if you do not need every string. It would be faster, if it would just set a (pchar, length) pair for each string. The CurTokenString property could be changed from returning the already decoded string to a method decoding the string only when the property is read.

Best,
Benito

Michael Van Canneyt wrote 66 lines
On Fri, 30 Aug 2019, Anthony Walter wrote: > I've posted a new page that tests the speed and correctness of several > pascal based JSON parsers. > > https://www.getlazarus.org/json/tests/ > > In full disclosure I am the author of the new open source JsonTools > library, and even though my parser seems to a big improvement over the > other alternatives, my tests were not biased. Test are always biased in the sense that they depend on the proficiency of the coder. you must ask someone with sufficient knowledge to write the code. The shootout benchmarks for example are dismally coded for FPC with as a result that they perform badly. So it could well be that the fpJSON results for example can be improved a lot by changing the method used to a more efficient one. Also, not every library is designed with the same goals. fpjson could probably be made smaller if the goal was more focused. it has a factory pattern, added on behalf of users who asked for this. jsontools does not have this. You can format the output. JSONTools does not have this. fpJSON was designed to be pluggable wrt. parsing. JSONTools is not. In short fpjson can do a lot more than JSONtools. Some things come at a price, others not. In that sense, tests like this compare apples with pears. > If anyone would like help in replication the tests, let me know and I'll > see what I can do. > > Also, to be thorough, you should read through both the article I > posted at > the top this message, and my original page > <https://www.getlazarus.org/json> > which has been updated with more information. Both pages took some > time to > write, and I promise if you read through them some of your questions > will > be answered without having to ask others for help or insight. Can you please send me the testcode you used for your speed & correctness tests ? I'm a bit surprised to see fpJSON fail in unicode correctness. It's been tested extensively, maybe your code contains errors (or maybe fpjson does, I'll need to check). Also please explain what 'Handling duplicate key names correctly' means to you. Saying that a library fails a test without specifying what the test is, is strange. Michael. _______________________________________________ fpc-pascal maillist  -  https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Reply by email

Michael Van Canneyt
to Benito van der Zander
Benito van der Zander wrote 12 lines
Hi, when I need maximal speed, I use jsonscanner of fpJSON. It should be vastly faster than a complete parser by not allocating memory or validating the json. Although it decodes all the strings in the JSON, which require some allocations, even if you do not need every string. It would be faster, if it would just set a (pchar, length) pair for each string. The CurTokenString property could be changed from returning the already decoded string to a method decoding the string only when the property is read.

Indeed, that is also why the JSONParser/JSONReader exists.

It allows a SAX-like approach to JSON, which is really important when dealing with huge documents. If you need to insert data at high speed in a database, you don't need the whole JSON Document which would allocate huge amounts of memory (at least twice the original JSON stream).

You use JSONSReader and handle record by record.

As I said, the scope of fpJSON is much bigger than the JSONtools approach. It simply became so out of necessity.

I'm currently implementing a speed improvement, but your suggestion is interesting and will probably result in even more speed enhancements.

Michael.

Reply by email

Anthony Walter
Could you include https://github.com/BeRo1985/pasjson in the comparison?

Sure. I also have a few other people have requested. I will also list the license of each in the first table.

Regarding a huge gigabytes of JSON in a file, I know a small portion of programmers of people might be inclined use it as an offline database format much like CSV. Even though by far most JSON is used with XMLHttpRequest, REST APIS, or storing settings and configurations, there are bound to be endless requests for use cases with JSON.

For example to accommodate the reading a huge files as indoividual records a helper class operating outside the definition of a JSON parser could accomplish this goal. For example, it would be relatively easy to write in a separate file:

type
  TJsonStreamReader = class
  public
    constructor Create(Stream: TStream; OwnsStream: Boolean = False);
    constructor CreateFromFile(const FileName: string);
    destructor Destroy;
    function Read(out Parser: TSomeJsonParser): Boolean;
  end;

Then use as ...

var
  R: TJsonStreamReader;
  P: TSomeJsonParser;
begin
  R := TJsonStreamReader.Create(InputStreamOrFileName);
  try
    while R.Read(P) do
      // Read JSON record here
  finally
    R.Free;
  end;
end;

And in this way a large file could be read in small blocks and given back to the user as a parser to allow for processing of individual records. The benefit of breaking this into its own class is that you do not need to mix in every possible use case into the parser. You can simply write separate use cases into their own independent units, rather than trying to make a super class which handles every possible concern.

For example if wanted to store object state using RTTI in a JSON file, create a separate TJsonObjectState class to handle this for you. Or if you wanted to create a database table from a JSON file, or create a JSON file from a database table, then again write this into its own class.

The point is, saying this JSON class does lots of these things is the wrong approach (IMO), as these use case scenarios as likely endless and would add unnecessary cruft to a parser. Even designing a plug in or other extensible seems unnecessary, when simple separate classes to add functionality works as well without all the complexity.

Reply by email

Michael Van Canneyt
to Anthony Walter
Anthony Walter wrote 5 lines
> Could you include https://github.com/BeRo1985/pasjson in the comparison? Sure. I also have a few other people have requested. I will also list the license of each in the first table.

[snip]

For example if wanted to store object state using RTTI in a JSON file, create a separate TJsonObjectState class to handle this for you. Or if you wanted to create a database table from a JSON file, or create a JSON file from a database table, then again write this into its own class.

Not sure I understand what you mean.

It seems to me that in that case you will repeat your scanner/parser code all over the place. in case of an error, you need to fix it in as many places.

I can of course be wrong.

The current fpjson scanner/parser can be used anywhere. You don't need to use fpjson data structures to be able to use the scanner or reader: It's perfectly possible to use the scanner/parser to create TJSONNode from JSONTools.

But general usability comes indeed at the price of some speed loss.

That said, your classes are markedly slower when it comes to data manipulation.

The following is 100 (!) times slower than fpjson code:

{$mode objfpc}
{$h+}
uses dateutils,sysutils, jsontools;
Var
   I,N : Integer;
   D,E : TJSONNode;
   B : double;
   NT : TDateTime;
begin
   N:=10000000;
   D:=TJSONNode.Create;
   D.Parse('{ "d": 12345678.3 }');
   E:=D.Child(0);
   NT:=Now;
   B:=1;
   for i:=0 to N  do
     B:=E.AsNumber * 2;
   Writeln('Time ',MillisecondsBetween(Now,NT));
   D.Free;
end.

home:~> ./tb
Time 3888

Same program in fpJSON:

home:~> ./tb2
Time 32

This is because when accessing the value, you must do the conversion to float. Every time.

This is true for JSON string values as well: you must re-decode the JSON on every access. And you do it over and over again, each time the data is accessed.

No doubt you can easily fix this by storing the value in the proper type, but this will slow down your parser.

So:
if you use the resulting JSON a lot, code will run faster in fpJSON.

It thus boils down to a choice: do you need fast processing or fast parsing ?

In the end it will probably not matter: most likely all the nodes will be traversed in a typical use case, and the overall time for your and my approach will be similar.

This is the danger of benchmarks. They focus on 1 aspect. In real life, all aspects are usually present.

Anyway.

While coding this small test, I noticed that this also does not work in jsontools:

D : TJSONNode;
begin
   D.Parse('true'); // or D.Parse('12345678.3');
end.

An unhandled exception occurred at $00000000004730B0: EJsonException: Root node must be an array or object

if you look at browser specs, this is supposed to work as well:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse

Also frequently encountered is omitting "" around property names. JSON is a subset of Javascript:

D.Parse('{ d: 12345678.3 }');

Results in:

An unhandled exception occurred at $0000000000473075: EJsonException: Error while parsing text

Both are things which are supported in fpJSON. No doubt you can fix this easily.

So you see, with some extra study, the picture of what is "better", jsontool or fpjson is not so clear as it may seem. In the end it all boils down to some choices.

Michael.

PS.
With 2 relatively simple changes, I took 40% off the parsing time of fpJSON. No doubt more gain can be achieved, for example I didn't do the suggestion by Benito yet.

Reply by email

Ben Grasset
to Anthony Walter: Sure. I also have a few other people have requested. I will also list the license of each…
AM Anthony Walter wrote 5 lines
> Could you include https://github.com/BeRo1985/pasjson in the comparison? Sure. I also have a few other people have requested. I will also list the license of each in the first table.

Note that I'm not sure if it's FPC-compatible, but I've found https://github.com/neslib/Neslib.Json to be *by far* the fastest JSON library for Delphi.

So if you can get it to compile with FPC (probably with {$mode DelphiUnicode} amongst other things) it would be an interesting addition to your tests as well.

Reply by email

Anthony Walter
to Ben Grasset

Ben,

After this hurricane Dorian situation is over (I live in Cape Canaveral Florida), I will be adding many parsers to the original test page I linked at the top of this thread. I will be revising the numbers and charts, adding more notes per this thread and some more which I feel are notable, and adding library license information.

Also I will be posting a couple of tools that use a JSON parser to perform things like text to speech, image recognition, as well as a generic AWS, Azure, and Google API to bridge authentication with those services in an abstract manner using pascal code and JSON.

Reply by email

Michael Van Canneyt
to Anthony Walter
Anthony Walter wrote 12 lines
Ben, After this hurricane Dorian situation is over (I live in Cape Canaveral Florida), I will be adding many parsers to the original test page I linked at the top of this thread. I will be revising the numbers and charts, adding more notes per this thread and some more which I feel are notable, and adding library license information. Also I will be posting a couple of tools that use a JSON parser to perform things like text to speech, image recognition, as well as a generic AWS, Azure, and Google API to bridge authentication with those services in an abstract manner using pascal code and JSON.

Looking forward to that. My guess is you're actually 3 persons operating under 1 account. With such a programme, I cannot imagine you every sleeping... ;)

Michael.

Reply by email