- Notifications
You must be signed in to change notification settings - Fork6
saleyn/simdjsone
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
An implementation of the fastest JSON parser for Erlang/Elixir using the C++simdjson NIF library. The decoding speedof this parser is about 2.5 times faster thanjiffy.
Seefull documentation of the C++ library.
Only a subset of functionality is implemented:
- Ability to decode JSON terms using a main scheduler or a dirty schedulerbased on the size of JSON binary input.
- Ability to cache the decoded term, and access its key/value pairs usinga NIF based
simdjson:get/2function. - The resources stored in the cache will get automatically garbage collectedwhen the owner process dies.
For small JSON objectssimdjsone is about twice faster thanjiffy and for large JSON objects, it's about30% faster thanjiffy.
The following decoding options are supported indecode(String, Options):
return_maps- decode JSON object as map (this is default)object_as_tuple- decode JSON object as a proplist wrapped in a tuplededup_keys- eliminate duplicate keys from a JSON objectuse_nil- decode JSON "null" asnil{null_term, V}- use the given valueVfor a JSON "null"
The following encoding options are supported inencode(String, Options):
uescape- escape UTF-8 sequences to produce a 7-bit clean outputpretty- return JSON using two-space indentationuse_nil- encode the atomnilasnullescape_fwd_slash- escape the/character (useful when encoding URLs){bytes_per_red, N}- whereN>= 0 - This controls the number of bytesthat the NIF library will process as an equivalent to a reduction. Each 20reductions we consume 1% of our allocated time slice for the current process.When the Erlang VM indicates we need to return from the NIF.
NOTE: Since the simdjson library currently doen't have an implementation ofa JSON encoder, the encoding implementation is the jiffy's modified encoderoptimized for speed when encoding integers.
The implementation includessimdjson:int_to_bin/1 function that is about 30%faster thanerlang:integer_to_binary/1, but it's limited to integers in range:(-1 bsl 63) <= I <= (1 bsl 62).
Serge Aleynikov
Include the following dependency in your project.
Erlang (rebar.config):
{deps, [{simdjsone,"0.5.0"}]}.Elixir (mix.exs):
defdeps()do[{:simdjsone,"~> 0.5.0"}]end
Erlang:
$ make deps compile
Elixir
$ MIX_ENV=dev make deps compile
1>simdjson:decode("{\"a\": [1,2,3],\"b\": 123,\"c\": 12.234}").#{<<"a">>=> [1,2,3],<<"b">>=>123,<<"c">>=>12.234}
After calling thesimdjson:parse/1, the functionsimdjson:get/2returns the value stored in a given path:
1>Ref=simdjson:parse("{\"a\": [1,2,3],\"b\": 123,\"c\": 12.234}").#Ref<0.1852532992.2458255361.217635>2>simdjson:get(Ref,"/a").[1,2,3]3>simdjson:get(Ref,"/b").1234>simdjson:get(Ref,"/a/c").12.234ok
1>simdjson:encode(#{a=> [1,2,3], <<"b">>=>123,c=>12.234}).<<"{\"b\":123,\"a\":[1,2,3],\"c\":12.234}">>2>simdjson:encode({[{a, [1,2,3]}, {<<"b">>,123}, {c,12.234}]}).<<"{\"a\":[1,2,3],\"b\":123,\"c\":12.234}">>
To run the performance benchmark ofsimdjsone againstjiffy andthoasdo the following (prefix the command withCXX=clang++ for using Clang C++compiler):
$ make benchmark=== Benchmark (file size: 616.7K) === simdjsone: 2526.420us euneus: 5314.820us thoas: 5452.380us json: 5541.130us jiffy: 10182.650us=== Benchmark (file size: 1.3K) === simdjsone: 6.940us thoas: 8.270us euneus: 9.790us json: 10.000us jiffy: 16.640us=== Benchmark (file size: 0.1K) === simdjsone: 1.930us euneus: 2.300us json: 2.500us jiffy: 2.770us thoas: 2.830usIf you have Elixir installed, the project also includes a benchmark for thejason andpoison Elixir parsers. The Elixir benchmarksare more exhaustive:
$ MIX_ENV=test make benchmark=== Benchmark (file size: 616.7K) ===Name ips average deviation median 99th %simdjsone 344.00 2.91 ms ±18.13% 2.63 ms 4.82 mspoison 154.88 6.46 ms ±10.31% 6.25 ms 9.96 msjason 153.87 6.50 ms ±11.37% 6.30 ms 10.18 msthaos 147.71 6.77 ms ±11.53% 6.52 ms 10.64 mseuneus 142.47 7.02 ms ±22.71% 6.50 ms 13.47 msjiffy 78.74 12.70 ms ±12.83% 12.52 ms 21.65 msComparison:simdjsone 344.00poison 154.88 - 2.22x slower +3.55 msjason 153.87 - 2.24x slower +3.59 msthaos 147.71 - 2.33x slower +3.86 mseuneus 142.47 - 2.41x slower +4.11 msjiffy 78.74 - 4.37x slower +9.79 msMemory usage statistics:Name Memory usagesimdjsone 0.00153 MBpoison 1.84 MB - 1200.09x memory usage +1.84 MBjason 1.81 MB - 1182.38x memory usage +1.81 MBthaos 1.81 MB - 1182.31x memory usage +1.81 MBeuneus 1.87 MB - 1219.47x memory usage +1.87 MBjiffy 3.19 MB - 2077.98x memory usage +3.19 MB**All measurements for memory usage were the same**=== Benchmark (file size: 1.3K) ===Name ips average deviation median 99th %simdjsone 177.77 K 5.63 μs ±187.84% 4.90 μs 16.90 μseuneus 101.90 K 9.81 μs ±94.81% 9 μs 22.30 μspoison 98.29 K 10.17 μs ±87.43% 9.40 μs 23.30 μsjason 95.18 K 10.51 μs ±99.89% 9.50 μs 30.70 μsjiffy 88.49 K 11.30 μs ±259.91% 9.30 μs 32.60 μsthaos 84.36 K 11.85 μs ±57.49% 10.90 μs 25.90 μsComparison:simdjsone 177.77 Keuneus 101.90 K - 1.74x slower +4.19 μspoison 98.29 K - 1.81x slower +4.55 μsjason 95.18 K - 1.87x slower +4.88 μsjiffy 88.49 K - 2.01x slower +5.68 μsthaos 84.36 K - 2.11x slower +6.23 μsMemory usage statistics:Name Memory usagesimdjsone 1.57 KBeuneus 5.22 KB - 3.32x memory usage +3.65 KBpoison 5.57 KB - 3.55x memory usage +4 KBjason 5.29 KB - 3.37x memory usage +3.72 KBjiffy 1.55 KB - 0.99x memory usage -0.01563 KBthaos 5.22 KB - 3.32x memory usage +3.65 KB**All measurements for memory usage were the same**=== Benchmark (file size: 0.1K) ===Name ips average deviation median 99th %simdjsone 820.20 K 1.22 μs ±1445.57% 1.10 μs 3.30 μspoison 576.53 K 1.73 μs ±1017.41% 1.40 μs 4.00 μsthaos 489.07 K 2.04 μs ±1003.95% 1.80 μs 4.70 μseuneus 483.75 K 2.07 μs ±624.18% 1.80 μs 4.70 μsjason 435.99 K 2.29 μs ±752.86% 2 μs 5.70 μsjiffy 311.21 K 3.21 μs ±652.57% 2.50 μs 9.10 μsComparison:simdjsone 820.20 Kpoison 576.53 K - 1.42x slower +0.52 μsthaos 489.07 K - 1.68x slower +0.83 μseuneus 483.75 K - 1.70x slower +0.85 μsjason 435.99 K - 1.88x slower +1.07 μsjiffy 311.21 K - 2.64x slower +1.99 μsMemory usage statistics:Name Memory usagesimdjsone 0.50 KBpoison 1.32 KB - 2.64x memory usage +0.82 KBthaos 1.20 KB - 2.41x memory usage +0.70 KBeuneus 1.20 KB - 2.41x memory usage +0.70 KBjason 1.27 KB - 2.55x memory usage +0.77 KBjiffy 1.46 KB - 2.92x memory usage +0.96 KB**All measurements for memory usage were the same**- Add support for
iterator - Add support for
iterate_manyandparse_many