Repository files navigation

A hyper-fast, safe Python module to read and write JSON data. Works as adrop-in replacement for Python's built-injson module.This is alpha software and there will be bugs, so maybe don't deploy to productionjust yet. 😉

This project is not actively maintained.orjson is likely the better alternative.

pip install hyperjson

hyperjson is meant as a drop-in replacement for Python'sjsonmodule:

>>>importhyperjson>>>hyperjson.dumps([{"key":"value"},81,True])'[{"key":"value"},81,true]'>>>hyperjson.loads("""[{"key": "value"}, 81, true]""")[{u'key':u'value'},81,True]

Parsing JSON is a solved problem; so, no need to reinvent the wheel, right?
Well, unless you care aboutperformance and safety.

Turns out, parsing JSONcorrectly is ahard problem. Thanks to Rust however, we can minimize the risk of running intostack overflows or segmentation faults however.

hyperjson is a thin wrapper around Rust'sserde-json andpyo3. It is compatible with Python 3 (and 2 on a best-effort basis).

For a more in-depth discussion,watch the talk about this project recorded at the Rust Cologne Meetup in August 2018.

• Compatibility: Support the full feature-set of Python'sjson module.
• Safety: No segfaults, panics, or overflows.
• Performance: Significantly faster thanjson and as fast asujson (both written in C).

• Support ujson and simplejson extensions:
  Custom extensions likeencode(),__json__(), ortoDict() are notsupported. The reason is, that they go against PEP8 (e.g.dunder methodsare restricted to the standard library, camelCase is not Pythonic) and are notavailable in Python'sjson module.
• Whitespace preservation: Whitespace in JSON strings is not preserved.Mainly because JSON is a whitespace-agnostic format andserde-json stripsthem out by default. In practice this should not be a problem, since yourapplication must not depend on whitespace padding, but it's something to beaware of.

We arenot fast yet. That said, we haven't done any big optimizations.In the long-term we might explore features of newer CPUs like multi-core and SIMD.That's one area other (C-based) JSON extensions haven't touched yet, because it mightmake code harder to debug and prone to race-conditions. In Rust, this is feasible due to crates likefaster orrayon.

So there's a chance that the following measurements might improve soon.
If you want to help, check the instructions in theDevelopment Environment section below.

Test machine:
MacBook Pro 15 inch, Mid 2015 (2,2 GHz Intel Core i7, 16 GB RAM) Darwin 17.6.18

If you would like to hack on hyperjson, here's what needs to be done:

• Implementloads()
• Implementload()
• Implementdumps()
• Implementdump()
• Benchmark againstjson andujson (see#1)
• Add a CI/CD pipeline for easier testing (see#2)
• Create a proper pip package from it, to make installing easier (see#3).
• Profile and optimize performance (see#16)
• Add remainingkeyword-only arguments to methods

Just pick one of the open tickets. We can provide mentorship if you like. 😃

This project usespoetry for managing the development environment. If you don't have it installed, run

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | pythonexport PATH="$HOME/.poetry/bin:$PATH"

The project requires thenightly version of Rust.

Install it viarustup:

rustup install nightly

If you have already installed thenightly version, make sure it is up-to-date:

rustup update nightly

After that, you can compile the current version of hyperjson and execute all tests and benchmarks with the following commands:

make installmake testmake bench

🤫 Pssst!... runmake help to learn more.

In order to recreate the benchmark histograms, you first need a few additional prerequisites:

• Matplotlib
• Numpy

On macOS, please also add the following to your~/.matplotlib/matplotlibrc (reference):

backend: TkAgg

After that, run the following:

make plot

hyperjson is licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE orhttp://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT orhttp://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submittedfor inclusion in hyperjson by you, as defined in the Apache-2.0 license, shallbe dual licensed as above, without any additional terms or conditions.