Repository files navigation

Bitproto is a fast, lightweight and easy-to-use bit level datainterchange format for serializing data structures.

The protocol describing syntax looks like the greatprotocol buffers,but in bit level:

proto examplemessage Data {    uint3 the = 1    uint3 bit = 2    uint5 level = 3    uint4 data = 4    uint11 interchange = 6    uint6 format = 7}  // 32 bits => 4B

TheData above is called a message, it consists of 7 fields and will occupy a totalof 4 bytes after encoding.

This image shows the layout of data fields in the encoded bytes buffer:

Code example to encode bitproto message in C:

structDatadata= {.the=7,                    .bit=7,                    .level=31,                    .data=15,                    .interchange=2047,                    .format=63};unsignedchars[BYTES_LENGTH_DATA]= {0};EncodeData(&data,s);// length of s is 4, and the hex format is// 0xFF 0xFF 0xFF 0xFF

And the decoding example:

structDatad= {0};DecodeData(&d,s);// values of d's fields is now:// 7 7 31 15 2047 63

Simple and green, isn't it?

Code patterns of bitproto encoding are exactly similar in C, Go and Python.

• Supports bit level data serialization,born for embedded development.
• Supports protocolextensiblity , for forward-compatibility.
• Easy to start, syntax is similar to the well-known protobuf.
• Supports languages:C (without dynamic memory allocation),Go,Python.
• Blazing fast encoding/decoding,benchmark.
• We canclearly know the size and arrangement of encoded data, fields are compact without a single bit gap.

An example for a simple overview of the bitproto schema grammar:

proto pen// Constant valueconst PEN_ARRAY_SIZE = 2 * 3;// Bit level enum.enum Color : uint3 {    COLOR_UNKNOWN = 0    COLOR_RED = 1    COLOR_BLUE = 2    COLOR_GREEN = 3}// Type aliastype Timestamp = int64// Composite structuremessage Pen {    Color color = 1    Timestamp produced_at = 2    uint3 number = 3    uint13 value = 4}message Box {    // Fixed-size array    Pen[PEN_ARRAY_SIZE] pens = 1;}

Run the bitproto compiler to generate C files:

$ bitproto c pen.bitproto

Which generates two files:pen_bp.h andpen_bp.c.

We can have an overview of the generated code for the C language:

// Constant value#definePEN_ARRAY_SIZE 6// Bit level enum.typedefuint8_tColor;// 3bit#defineCOLOR_UNKNOWN 0#defineCOLOR_RED 1#defineCOLOR_BLUE 2#defineCOLOR_GREEN 3// Type aliastypedefint64_tTimestamp;// 64bit// Number of bytes to encode struct Pen#defineBYTES_LENGTH_PEN 11// Composite structurestructPen {Colorcolor;// 3bitTimestampproduced_at;// 64bituint8_tnumber;// 3bituint16_tvalue;// 13bit};// Number of bytes to encode struct Box#defineBYTES_LENGTH_BOX 63structBox {// Fixed-size arraystructPenpens[6];// 498bit};

You can checkout directoryexample for a larger example.

There is protobuf, why bitproto?

The bitproto was originally made when I'm working with embedded programs onmicro-controllers. Where usually exists many programming constraints:

• tight communication size.
• limited compiled code size.
• better no dynamic memory allocation.

Protobuf does not live on embedded field natively,it doesn't target ANSI C out of box.

It's recommended to use bitproto over protobuf when:

• Working on or with microcontrollers.
• Wants bit-level message fields.
• Wants to know clearly how many bytes the encoded data will occupy.

For scenarios other than the above, I recommend to use protobuf over bitproto.

The differences between bitproto and protobuf are:

• bitproto supports bit level data serialization, like thebit fields in C.

• bitproto doesn't use any dynamic memory allocations. Few ofprotobuf C implementationssupport this, exceptnanopb.

• bitproto doesn't support varying sized data, all types are fixed sized.

  bitproto won't encode typing or size reflection information into the buffer.It only encodes the data itself, without any additional data,the encoded datais arranged like it's arranged in the memory, with fixed size, without paddings,think settingaligned attribute to 1on structs in C.

• Protobuf works good onforward compatibility.For bitproto, this is the main shortcome of bitproto serialization until v0.4.0, since this version, it supports message'sextensiblity by adding two bytes indicatingthe message size at head of the message's encoded buffer. This breaks thetraditional data layout design by encoding some minimal reflectionsize information in, so this is designed as an optional feature.

• bitproto doesn't support varying sized types. For example, aunit37 always occupies37 bits even you assign it a small value like1.

  Which means there will be lots of zero bytes if the meaningful data occupies little onthis type. For instance, there will ben-1 bytes left zero if only one byte of atype withn bytes size is used.

  Generally, we actually don't care much about this, since there are not so many bytesin communication with embedded devices. The protocol itself is meant to be designedtight and compact. Consider to wrap a compression mechanism likezlibon the encoded buffer if you really care.

• bitproto can't providebest encoding performancewithextensibility.

  There's anoptimization mode designed in bitprototo generate plain encoding/decoding statements directly at code-generation time, since alltypes in bitproto are fixed-sized, how-to-encode can be determined earlier at code-generationtime. This mode gives a huge performance improvement, but I still haven't found a way tomake it work with bitproto's extensibility mechanism together.

Documentation:

• Website:https://bitproto.readthedocs.io
• Documentation in Chinese:https://bitproto.readthedocs.io/zh/latest
• Quick start tutorial
• Grammar guide, in one page

Editor syntax highlighting plugins:

• Vim Plugin
• PyCharm Settings
• VS Code Extension

Faq:

• What’s the advantage of this over a bit field?

Blog posts:

• Dev notes in Chinese:https://hit9.dev/post/bitproto-notes

BSD3