Repository files navigation

Arbitrary base binary-to-text encoder (any base to any base), in Python.

In short,basest isbased on (pundefinitely intended 😉) the concept of binary-to-text conversion, that is where binary or 8-bit data is converted or serialised into a text-based representation format that can be safely passed through a medium that would otherwise destroy or corrupt the meaning of the binary data.

This concept is very commonly used in areas such as Email, the PDF format and Public Key Cryptography, to name but a few.

There are many different formats and schemes for serilising binary data to text, employing different alphabet sizes and different printable ASCII characters used for various different reasons.

It is also not just 8-bit binary data that could be serialised. Any collection of symbols declared to be in a given number base or alphabet size can be serialised into any other, provided an encoding ratio between the two symbols can be established and the input and output symbols defined.

This library is my implementation of a generic, base-to-base converter which addresses this last point. An encoder and decoder for every binary-to-text format currently existing can be created and used with this library, requiring only for the details of the desired format to be given. Due to its flexibility, the library also makes it trivial to invent new wonderful and interesting base-to-base serialisation/conversion formats (I myself plan to work on and release one that translates binary files into a purely emoji-based format!).

One limitation of the library is that it cannot encode data from a smaller input base to a larger output base with padding on the input (i.e. if you're encoding from base 2 to base 1000, you need to ensure that the number of input symbols exactly matches the encoding ratio you're using). This is an accepted limitation due to the complexities of implementing a padding system that works in the same manner as base-64 and others but which can be extended to any arbitrary base.

So, I hope you find this library fun, useful or both!

This library is designed to work withPython 2.7 andPython 3.3or greater.

It is tested against Python2.7,3.5,3.6 andPyPy 3.5.

💡Help Wanted

If you have tried or want to try this out on any other Python implementations, your feedback would be greatly appreciated!

Open an issue if you are interested.

pip install basest

pip install git+git://github.com/saxbophone/basest-python@develop

Here is a short overview of the interfaces defined in this library, where to import them from and how to use them.

There is a functional interface and a class-based interface (the class-based one piggy-backs on the functional one, but will also be used to add additional features in the future).

To use the class-based interface, you will need to create a subclass ofbasest.encoders.Encoder and override attributes of the class, as shown below (using base64 as an example):

frombasest.encodersimportEncoderclassCustomEncoder(Encoder):input_base=256output_base=64input_ratio=3output_ratio=4# these attributes are only required if using decode() and encode()input_symbol_table= [chr(c)forcinrange(256)]output_symbol_table= [sforsin'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'    ]padding_symbol='='

Note: You must subclassEncoder, you cannot use it directly!

Subclasses ofEncoder have the following public methods available:

encode() will encode an iterable of symbols in the class'input symbol table into an iterable of symbols in the class'output symbol table, observing the chosen encoding ratios and padding symbol.

encoder=CustomEncoder()encoder.encode(['c','a','b','b','a','g','e','s'])# -> ['Y', '2', 'F', 'i', 'Y', 'm', 'F', 'n', 'Z', 'X', 'M', '=']

encode_raw() works just likeencode(), except that symbols are not interpreted. Instead, plain integers within range 0->(base - 1) should be used. the value of the base is used as the padding symbol.

encoder=CustomEncoder()encoder.encode_raw([1,2,3,4,5,6,7])# -> [0, 16, 8, 3, 1, 0, 20, 6, 1, 48, 64, 64]

decode() works in the exact same way asencode(), but in the inverse.

encoder=CustomEncoder()encoder.decode(['Y','2','F','i','Y','m','F','n','Z','X','M','='])# -> ['c', 'a', 'b', 'b', 'a', 'g', 'e', 's']

decode_raw() works just likedecode(), except that symbols are not interpreted. Instead, plain integers within range 0->(base - 1) should be used. the value of the base is used as the padding symbol.

encoder=CustomEncoder()encoder.decode_raw([0,16,8,3,1,0,20,6,1,48,64,64])# -> [1, 2, 3, 4, 5, 6, 7]

For a giveninput base,input symbol table,output base,output symbol table,output padding,input ratio,output ratio and theinput data (as an iterable composed of items which are defined ininput symbol table):Return the input data, encoded into the specified base using the specified encoding ratio and symbol tables (and the suppliedoutput padding symbol used if needed).Returns the output data as a list of items that are guaranteed to be in theoutput symbol table, or theoutput padding symbol.

importbasestbasest.core.encode(input_base=256,input_symbol_table=[chr(c)forcinrange(256)],output_base=64,output_symbol_table=[sforsin'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'    ],output_padding='=',input_ratio=3,output_ratio=4,input_data='falafel')# -> ['Z', 'm', 'F', 's', 'Y', 'W', 'Z', 'l', 'b', 'A', '=', '=']

Similar to the function above,basest.core.encode_raw will encode one base into another, but only accepts and returns arrays of integers (e.g. bytes would be passed as integers between 0-255, not asbyte objects). As such, it omits thepadding andsymbol table arguments, but is otherwise identical in function and form toencode.

importbasestbasest.core.encode_raw(input_base=256,output_base=85,input_ratio=4,output_ratio=5,input_data=[99,97,98,98,97,103,101,115])# -> [31, 79, 81, 71, 52, 31, 25, 82, 13, 76]

For a giveninput base,input symbol table,input padding,output base,output symbol table,input ratio,output ratio and theinput data (as an iterable composed of items which are defined ininput symbol table), return the input data, decoded from the base it was encoded into.Returns the output data as a list of items that are guaranteed to be in theoutput symbol table, with no padding.

This is essentially the inverse ofencode()

importbasestbasest.core.decode(input_base=64,input_symbol_table=[sforsin'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'    ],input_padding='=',output_base=256,output_symbol_table=[chr(c)forcinrange(256)],input_ratio=4,output_ratio=3,input_data='YWJhY3VzIFpaWg==')# -> ['a', 'b', 'a', 'c', 'u', 's', ' ', 'Z', 'Z', 'Z']

Similar to the function above,basest.core.decode_raw will decode from one base to another, but only accepts and returns arrays of integers (e.g. base64 would be passed as integers between 0-65 (65 is for the padding symbol), not asstr objects). As such, it omits thepadding andsymbol table arguments, but is otherwise identical in function and form todecode.

importbasestbasest.core.decode_raw(input_base=85,output_base=256,input_ratio=5,output_ratio=4,input_data=[31,79,81,71,52,31,25,82,13,76])# -> [99, 97, 98, 98, 97, 103, 101, 115]

For a giveninput base (e.g. base-256 / 8-bit Bytes), a given desiredoutput base (e.g. base 94)OR a given range of acceptableoutput bases and a range ofchunk sizes to consider using for the input (amount of bytes/symbols processed at once), return the most efficient output base and encoding ratio to use (in terms of input base to output base).

Returns tuples containing an integer as the first item (representing the output base that is most efficient) and a tuple as the second, containing two integers representing the ratio ofinput base symbols tooutput base symbols.

importbasestbasest.core.best_ratio(input_base=256,output_bases=[94],chunk_sizes=range(1,256))# -> (94, (68, 83))basest.core.best_ratio(input_base=256,output_bases=[94],chunk_sizes=range(1,512))# -> (94, (458, 559))basest.core.best_ratio(input_base=256,output_bases=range(2,95),chunk_sizes=range(1,256))# -> (94, (68, 83))basest.core.best_ratio(input_base=256,output_bases=range(2,334),chunk_sizes=range(1,256))# -> (333, (243, 232))