getml/reflect-cppPublic

NotificationsYou must be signed in to change notification settings
Fork149
Star1.7k

A C++20 library for fast serialization, deserialization and validation using reflection. Supports JSON, Avro, BSON, Cap'n Proto, CBOR, CSV, flexbuffers, msgpack, parquet, TOML, UBJSON, XML, YAML / msgpack.org[C++20]

License

MIT license

1.7k stars 149 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 745 Commits
.github/workflows		.github/workflows
benchmarks		benchmarks
docs		docs
include		include
scripts		scripts
src		src
tests		tests
vcpkg @ 7213cf8		vcpkg @ 7213cf8
.clang-format		.clang-format
.clang-tidy		.clang-tidy
.gitignore		.gitignore
.gitmodules		.gitmodules
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
README.md		README.md
banner1.png		banner1.png
conanfile.py		conanfile.py
mkdocs.yaml		mkdocs.yaml
pyproject.toml		pyproject.toml
reflectcpp-config.cmake.in		reflectcpp-config.cmake.in
vcpkg.json		vcpkg.json

Repository files navigation

reflect-cpp

📖 Documentation:https://rfl.getml.com

reflect-cpp is a C++-20 library forfast serialization, deserialization and validation using reflection, similar topydantic in Python,serde in Rust,encoding in Go oraeson in Haskell.

Moreover, reflect-cpp is the basis forsqlgen, amodern, type-safe ORM and SQL query generator for C++20, inspired by Python's SQLAlchemy/SQLModel and Rust's Diesel. It provides a fluent, composable interface for database operations with compile-time type checking and SQL injection protection.

reflect-cpp and sqlgen fill important gaps in C++ development. They reduce boilerplate code and increase code safety. Together, they enable reliable and efficient ETL pipelines.

Design principles for reflect-cpp include:

Close integration withcontainers from the C++ standard library
Close adherence to C++ idioms
Out-of-the-box support forJSON
Simpleinstallation
Simple extendability toother serialization formats
Simple extendability tocustom classes
Being one of the fastest serialization libraries in existence, as demonstrated by ourbenchmarks

Serialization formats

reflect-cpp provides a unified reflection-based interface across different serialization formats. It is deliberately designed in a very modular way, usingconcepts, to make it as easy as possible to interface various C or C++ libraries related to serialization. Refer to thedocumentation for details.

The following table lists the serialization formats currently supported by reflect-cpp and the underlying libraries used:

Format	Library	Version	License	Remarks
JSON	yyjson	>= 0.8.0	MIT	out-of-the-box support, included in this repository
Avro	avro-c	>= 1.11.3	Apache 2.0	Schemaful binary format
BSON	libbson	>= 1.25.1	Apache 2.0	JSON-like binary format
Cap'n Proto	capnproto	>= 1.0.2	MIT	Schemaful binary format
CBOR	jsoncons	>= 0.176.0	BSL 1.0	JSON-like binary format
CSV	Apache Arrow	>= 21.0.0	Apache 2.0	Tabular textual format
flexbuffers	flatbuffers	>= 23.5.26	Apache 2.0	Schema-less version of flatbuffers, binary format
msgpack	msgpack-c	>= 6.0.0	BSL 1.0	JSON-like binary format
parquet	Apache Arrow	>= 21.0.0	Apache 2.0	Tabular binary format
TOML	toml++	>= 3.4.0	MIT	Textual format with an emphasis on readability
UBJSON	jsoncons	>= 0.176.0	BSL 1.0	JSON-like binary format
XML	pugixml	>= 1.14	MIT	Textual format used in many legacy projects
YAML	yaml-cpp	>= 0.8.0	MIT	Textual format with an emphasis on readability

Support for more serialization formats is in development. Refer to theissues for details.

Please also refer to theconanfile.py orvcpkg.json in this repository.

Feature Overview

Simple Example

#include<rfl/json.hpp>#include<rfl.hpp>structPerson {  std::string first_name;  std::string last_name;int age;};constauto homer =    Person{.first_name ="Homer",           .last_name ="Simpson",           .age =45};// We can now write into and read from a JSON string.const std::string json_string = rfl::json::write(homer);auto homer2 = rfl::json::read<Person>(json_string).value();

The resulting JSON string looks like this:

{"first_name":"Homer","last_name":"Simpson","age":45}

You can transform the field names fromsnake_case tocamelCase like this:

const std::string json_string =   rfl::json::write<rfl::SnakeCaseToCamelCase>(homer);auto homer2 =   rfl::json::read<Person, rfl::SnakeCaseToCamelCase>(json_string).value();

The resulting JSON string looks like this:

{"firstName":"Homer","lastName":"Simpson","age":45}

Or you can use another format, such as YAML.

#include<rfl/yaml.hpp>// ... (same as above)const std::string yaml_string = rfl::yaml::write(homer);auto homer2 = rfl::yaml::read<Person>(yaml_string).value();

The resulting YAML string looks like this:

first_name:Homerlast_name:Simpsonage:45

This will work for just about any example in the entire documentationand any of the following formats, except where explicitly noted otherwise:

rfl::avro::write(homer);rfl::bson::write(homer);rfl::capnproto::write(homer);rfl::cbor::write(homer);rfl::flexbuf::write(homer);rfl::msgpack::write(homer);rfl::toml::write(homer);rfl::ubjson::write(homer);rfl::xml::write(homer);rfl::avro::read<Person>(avro_bytes);rfl::bson::read<Person>(bson_bytes);rfl::capnproto::read<Person>(capnproto_bytes);rfl::cbor::read<Person>(cbor_bytes);rfl::flexbuf::read<Person>(flexbuf_bytes);rfl::msgpack::read<Person>(msgpack_bytes);rfl::toml::read<Person>(toml_string);rfl::ubjson::read<Person>(ubjson_bytes);rfl::xml::read<Person>(xml_string);

More Comprehensive Example

#include<iostream>#include<rfl/json.hpp>#include<rfl.hpp>// Age must be a plausible number, between 0 and 130. This will// be validated automatically.using Age = rfl::Validator<int, rfl::Minimum<0>, rfl::Maximum<130>>;structPerson {  rfl::Rename<"firstName", std::string> first_name;  rfl::Rename<"lastName", std::string> last_name ="Simpson";  std::string town ="Springfield";  rfl::Timestamp<"%Y-%m-%d"> birthday;  Age age;  rfl::Email email;  std::vector<Person> children;};constauto bart = Person{.first_name ="Bart",                         .birthday ="1987-04-19",                         .age =10,                         .email ="bart@simpson.com"};constauto lisa = Person{.first_name ="Lisa",                         .birthday ="1987-04-19",                         .age =8,                         .email ="lisa@simpson.com"};constauto maggie = Person{.first_name ="Maggie",                           .birthday ="1987-04-19",                           .age =0,                           .email ="maggie@simpson.com"};constauto homer =    Person{.first_name ="Homer",           .birthday ="1987-04-19",           .age =45,           .email ="homer@simpson.com",           .children = std::vector<Person>({bart, lisa, maggie})};// We can now transform this into a JSON string.const std::string json_string = rfl::json::write(homer);std::cout << json_string << std::endl;// We can also directly write into std::cout (or any other std::ostream).rfl::json::write(homer, std::cout) << std::endl;

This results in the following JSON string:

{"firstName":"Homer","lastName":"Simpson","town":"Springfield","birthday":"1987-04-19","age":45,"email":"homer@simpson.com","children":[{"firstName":"Bart","lastName":"Simpson","town":"Springfield","birthday":"1987-04-19","age":10,"email":"bart@simpson.com","children":[]},{"firstName":"Lisa","lastName":"Simpson","town":"Springfield","birthday":"1987-04-19","age":8,"email":"lisa@simpson.com","children":[]},{"firstName":"Maggie","lastName":"Simpson","town":"Springfield","birthday":"1987-04-19","age":0,"email":"maggie@simpson.com","children":[]}]}

We can also create structs from the string:

auto homer2 = rfl::json::read<Person>(json_string).value();// Fields can be accessed like this:std::cout <<"Hello, my name is" << homer.first_name() <<""          << homer.last_name() <<"." << std::endl;// Since homer2 is mutable, we can also change the values like this:homer2.first_name ="Marge";std::cout <<"Hello, my name is" << homer2.first_name() <<""          << homer2.last_name() <<"." << std::endl;

Tabular data

reflect-cpp also supports tabular data formats, like CSV or Parquet:

#include<rfl/csv.hpp>#include<rfl/parquet.hpp>structPerson {    std::string first_name;    std::string last_name ="Simpson";    std::string town ="Springfield";int age;    rfl::Email email;};constauto people =  std::vector<Person>({Person{.first_name ="Bart",                              .birthday ="1987-04-19",                              .age =10,                              .email ="bart@simpson.com"},                       Person{.first_name ="Lisa",                              .birthday ="1987-04-19",                              .age =8,                              .email ="lisa@simpson.com"},                       Person{.first_name ="Maggie",                              .birthday ="1987-04-19",                              .age =0,                              .email ="maggie@simpson.com"},                       Person{.first_name ="Homer",                              .birthday ="1987-04-19",                              .age =45,                              .email ="homer@simpson.com"}});constauto csv_string = rfl::csv::write(people);constauto bytestring = rfl::parquet::write(people);

This will resulting CSV will look like this:

"first_name","last_name","town","birthday","age","email""Bart","Simpson","Springfield",1987-04-19,10,"bart@simpson.com""Lisa","Simpson","Springfield",1987-04-19,8,"lisa@simpson.com""Maggie","Simpson","Springfield",1987-04-19,0,"maggie@simpson.com""Homer","Simpson","Springfield",1987-04-19,45,"homer@simpson.com"

Error messages

reflect-cpp returns clear and comprehensive error messages:

const std::string faulty_json_string =R"({"firstName":"Homer","lastName":12345,"town":"Springfield","birthday":"04/19/1987","age":145,"email":"homer(at)simpson.com"})";constauto result = rfl::json::read<Person>(faulty_json_string);

Yields the following error message:

Found 5 errors:1) Failed to parse field 'lastName': Could not cast to string.2) Failed to parse field 'birthday': String '04/19/1987' did not match format '%Y-%m-%d'.3) Failed to parse field 'age': Value expected to be less than or equal to 130, but got 145.4) Failed to parse field 'email': String 'homer(at)simpson.com' did not match format 'Email': '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'.5) Field named 'children' not found.

JSON schema

reflect-cpp also supports generating JSON schemata:

structPerson {  std::string first_name;  std::string last_name;  rfl::Description<"Must be a proper email in the form xxx@xxx.xxx.",                   rfl::Email>      email;  rfl::Description<"The person's children. Pass an empty array for no children.",      std::vector<Person>>      children;float salary;};const std::string json_schema = rfl::json::to_schema<Person>();

The resulting JSON schema looks like this:

{"$schema":"https://json-schema.org/draft/2020-12/schema","$ref":"#/$defs/Person","$defs":{"Person":{"type":"object","properties":{"children":{"type":"array","description":"The person's children. Pass an empty array for no children.","items":{"$ref":"#/$defs/Person"}},"email":{"type":"string","description":"Must be a proper email in the form xxx@xxx.xxx.","pattern":"^[a-zA-Z0-9._%+\\-]+@[a-zA-Z0-9.\\-]+\\.[a-zA-Z]{2,}$"},"first_name":{"type":"string"},"last_name":{"type":"string"},"salary":{"type":"number"}},"required":["children","email","first_name","last_name","salary"]}}}

Note that this is currently supported for JSON only, since most other formats do not support schemata in the first place.

Enums

reflect-cpp supports scoped enumerations:

enumclassShape { circle, square, rectangle };enumclassColor { red =256, green =512, blue =1024, yellow =2048 };structItem {float pos_x;float pos_y;  Shape shape;  Color color;};constauto item = Item{.pos_x =2.0,                         .pos_y =3.0,                       .shape = Shape::square,                       .color = Color::red | Color::blue};rfl::json::write(item);

This results in the following JSON string:

{"pos_x":2.0,"pos_y":3.0,"shape":"square","color":"red|blue"}

You can also directly convert between enumerator values and strings withrfl::enum_to_string() andrfl::string_to_enum(), orobtain list of enumerator name and value pairs withrfl::get_enumerators<EnumType>() orrfl::get_enumerator_array<EnumType>().

Algebraic data types

reflect-cpp supports Pydantic-style tagged unions, which allow you to form algebraic data types:

structCircle {double radius;};structRectangle {double height;double width;};structSquare {double width;};using Shapes = rfl::TaggedUnion<"shape", Circle, Square, Rectangle>;const Shapes r = Rectangle{.height =10, .width =5};constauto json_string = rfl::json::write(r);

This results in the following JSON string:

{"shape":"Rectangle","height":10.0,"width":5.0}

Other forms of tagging are supported as well. Refer to thedocumentation for details.

Extra fields

If you don't know all of your fields at compile time, no problem. Just userfl::ExtraFields:

structPerson {  std::string first_name;  std::string last_name ="Simpson";  rfl::ExtraFields<rfl::Generic> extra_fields;};auto homer = Person{.first_name ="Homer"};homer.extra_fields["age"] =45;homer.extra_fields["email"] ="homer@simpson.com";homer.extra_fields["town"] ="Springfield";

This results in the following JSON string:

{"firstName":"Homer","lastName":"Simpson","age":45,"email":"homer@simpson.com","town":"Springfield"}

Reflective programming

Beyond serialization and deserialization, reflect-cpp also supports reflective programming in general.

For instance:

structPerson {  std::string first_name;  std::string last_name ="Simpson";  std::string town ="Springfield";unsignedint age;  std::vector<Person> children;};for (constauto& f : rfl::fields<Person>()) {  std::cout <<"name:" << f.name() <<", type:" << f.type() << std::endl;}

You can also create a view and then access these fields usingstd::get orrfl::get, or iterate over the fields at compile-time:

auto lisa = Person{.first_name ="Lisa", .last_name ="Simpson", .age =8};constauto view = rfl::to_view(lisa);// view.values() is a std::tuple containing// pointers to the original fields.// This will modify the struct `lisa`:*std::get<0>(view.values()) ="Maggie";// All of this is supported as well:*view.get<1>() ="Simpson";*view.get<"age">() =0;*rfl::get<0>(view) ="Maggie";*rfl::get<"first_name">(view) ="Maggie";view.apply([](constauto& f) {// f is an rfl::Field pointing to the original field.  std::cout << f.name() <<":" <<rfl::json::write(*f.value()) << std::endl;});

It also possible to replace fields:

structPerson {  std::string first_name;  std::string last_name;  std::vector<Person> children;};constauto lisa = Person{.first_name ="Lisa", .last_name ="Simpson"};// Returns a deep copy of "lisa" with the first_name replaced.constauto maggie = rfl::replace(    lisa, rfl::make_field<"first_name">(std::string("Maggie")));

Or you can create structs from other structs:

structA {  std::string f1;  std::string f2;};structB {  std::string f3;  std::string f4;};structC {  std::string f1;  std::string f2;  std::string f4;};constauto a = A{.f1 ="Hello", .f2 ="World"};constauto b = B{.f3 ="Hello", .f4 ="World"};// f1 and f2 are taken from a, f4 is taken from b, f3 is ignored.constauto c = rfl::as<C>(a, b);

You can also replace fields in structs using fields from other structs:

constauto a = A{.f1 ="Hello", .f2 ="World"};constauto c = C{.f1 ="C++", .f2 ="is", .f4 ="great"};// The fields f1 and f2 are replaced with the fields f1 and f2 in a.constauto c2 = rfl::replace(c, a);

Support for containers

C++ standard library

reflect-cpp supports the following containers from the C++ standard library:

std::array
std::deque
std::chrono::duration
std::filesystem::path
std::forward_list
std::map
std::multimap
std::multiset
std::list
std::optional
std::pair
std::set
std::shared_ptr
std::span
std::string
std::string_view
std::tuple
std::unique_ptr
std::unordered_map
std::unordered_multimap
std::unordered_multiset
std::unordered_set
std::variant
std::vector
std::wstring

Additional containers

In addition, it supports the following custom containers:

rfl::Binary: Used to express numbers in binary format.
rfl::Box: Similar tostd::unique_ptr, but (almost) guaranteed to never be null.
rfl::Bytestring: An alias forstd::vector<std::byte>. Supported by Avro, BSON, Cap'n Proto, CBOR, flexbuffers, msgpack and UBJSON.
rfl::Generic: A catch-all type that can represent (almost) anything.
rfl::Hex: Used to express numbers in hex format.
rfl::Literal: An explicitly enumerated string.
rfl::NamedTuple: Similar tostd::tuple, but with named fields that can be retrieved via their name at compile time.
rfl::Object: A map-like type representing a object with field names that are unknown at compile time.
rfl::Oct: Used to express numbers in octal format.
rfl::Ref: Similar tostd::shared_ptr, but (almost) guaranteed to never be null.
rfl::Result: Allows for exception-free programming.
rfl::TaggedUnion: Similar tostd::variant, but with explicit tags that make parsing more efficient.
rfl::Tuple: An alternative tostd::tuple that compiles considerably faster.
rfl::Validator: Allows for automatic input validation.
rfl::Variant: An alternative tostd::variant that compiles considerably faster.

Custom classes

Finally, it is very easy to extend full support to your own classes, refer to thedocumentation for details.

Installation

The following compilers are supported:

GCC 11.4 or higher
Clang 14.0 or higher
MSVC 17.8 (19.38) or higher

Using vcpkg

https://vcpkg.io/en/package/reflectcpp

Using Conan

https://conan.io/center/recipes/reflect-cpp

Compilation using cmake

This will compile reflect-cpp with JSON support only. You can then include reflect-cpp in your project and link to the binary.

cmake -S. -B build -DCMAKE_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Releasecmake --build build -j 4# gcc, clangcmake --build build --config Release -j 4# MSVC

To install all supported serialization formats, first install vcpkg:

git submodule update --init./vcpkg/bootstrap-vcpkg.sh# Linux, macOS./vcpkg/bootstrap-vcpkg.bat# Windows# You may be prompted to install additional dependencies.

Then, compile the library:

cmake -S. -B build -DCMAKE_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Release -DREFLECTCPP_ALL_FORMATS=ONcmake --build build -j 4# gcc, clangcmake --build build --config Release -j 4# MSVC

For other installation methods, refer to thedocumentation.

The team behind reflect-cpp

reflect-cpp has been developed bygetML (Code17 GmbH), a company specializing in software engineering and machine learning for enterprise applications. reflect-cpp is currently maintained by Patrick Urbanke and Manuel Bellersen, with major contributions coming from the community.

Related projects

reflect-cpp was originally developed forgetml-community, the fastest open-source tool for feature engineering on relational data and time series. If you are interested in Data Science and/or Machine Learning, please check it out.

Professional C++ Support

For comprehensive C++ support beyond the scope of GitHub discussions, we’re here to help! Reach out atsupport@getml.com to discuss any technical challenges or project requirements. We’re excited to support your work as independent software consultants.

License

reflect-cpp is released under the MIT License. Refer to the LICENSE file for details.

reflect-cpp includesYYJSON, the fastest JSON library currently in existence. YYJSON is written by YaoYuan and also released under the MIT License.

reflect-cpp includescompile-time-regular-expressions. CTRE is written by Hana Dusíková and released under the Apache-2.0 License with LLVM exceptions.

reflect-cpp includesenchantum. enchantum is written by ZXShady and also released under the MIT License.