Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-146192: Add base32 support to binascii#146193

Merged
serhiy-storchaka merged 8 commits intopython:mainfrom
kangtastic:base32-accel
Mar 22, 2026
Merged

gh-146192: Add base32 support to binascii#146193
serhiy-storchaka merged 8 commits intopython:mainfrom
kangtastic:base32-accel

Conversation

@kangtastic
Copy link
Contributor

@kangtastickangtastic commentedMar 20, 2026
edited
Loading

Synopsis

Add base32 encoder and decoder functions implemented in C tobinascii and use them to greatly improve the performance and reduce the memory usage of the existing base32 codec functions inbase64.

No API or documentation changes are necessary with respect to any functions inbase64, and all existing unit tests for those functions continue to pass without modification.

Resolves:gh-146192

Discussion

The base32-related functions inbase64 are now wrappers for the new functions inbinascii, as envisioned in thedocs:

Thebinascii module contains a number of methods to convert between binary and various ASCII-encoded binary representations. Normally, you will not use these functions directly but use wrapper modules likeuu orbase64 instead. Thebinascii module contains low-level functions written in C for greater speed that are used by the higher-level modules.

Comments and questions are welcome.

Benchmarks

Benchmark script

# bench_b32.py# Note: Can be EXTREMELY SLOW on unmodified mainline CPython.importbase64importsysimporttimeitimporttracemallocfuncs= [(base64.b64encode,base64.b64decode),# sanity check/comparison         (base64.b32encode,base64.b32decode),         (base64.b32hexencode,base64.b32hexdecode)]defmb(n):returnf"{n/1024/1024:.3f}"defstats(func,data,t,m):name,n,bps=func.__qualname__,len(data),len(data)/tprint(f"{name:<16}{n:<16}{t:<11.3f}{mb(bps):<13}{mb(m)}")if__name__=="__main__":print(f"Python{sys.version}\n")print(f"function        processed (b)   time (s)   avg (MB/s)   mem (MB)\n")data=b"a"*int(sys.argv[1])*1024*1024forfenc,fdecinfuncs:tracemalloc.start()enc=fenc(data)menc=tracemalloc.get_traced_memory()[1]-len(enc)tracemalloc.stop()tenc=timeit.timeit("fenc(data)",number=1,globals=globals())stats(fenc,data,tenc,menc)tracemalloc.start()dec=fenc(enc)mdec=tracemalloc.get_traced_memory()[1]-len(dec)tracemalloc.stop()tdec=timeit.timeit("fdec(enc)",number=1,globals=globals())stats(fdec,enc,tdec,mdec)

Unmodified mainline CPython

$./python bench_b32.py 16Python 3.15.0a7+ (heads/main:d357a7dbf38, Mar 19 2026, 23:22:25) [GCC 15.2.0]function        processed (b)   time (s)   avg (MB/s)   mem (MB)b64encode       16777216        0.015      1088.370     0.000b64decode       22369624        0.017      1264.389     0.000b32encode       16777216        2.308      6.933        17.382b32decode       26843552        3.389      7.553        27.787b32hexencode    16777216        2.338      6.843        17.379b32hexdecode    26843552        3.388      7.557        27.787

With this PR

$./python bench_b32.py 16Python 3.15.0a7+ (heads/base32-accel:72fd0f0302a, Mar 20 2026, 00:04:23) [GCC 15.2.0]function        processed (b)   time (s)   avg (MB/s)   mem (MB)b64encode       16777216        0.015      1084.957     0.000b64decode       22369624        0.016      1363.524     0.000b32encode       16777216        0.017      967.528      0.000b32decode       26843552        0.016      1581.002     0.000b32hexencode    16777216        0.016      995.277      0.000b32hexdecode    26843552        0.016      1588.353     0.000

Encoding performance is improved by ~150x, decoding performance is improved by ~200x,
and no auxiliary memory is used.


📚 Documentation preview 📚:https://cpython-previews--146193.org.readthedocs.build/

serhiy-storchaka reacted with hooray emoji
Add base32 encoder and decoder functions implemented inC to `binascii` and use them to greatly improve theperformance and reduce the memory usage of the existingbase32 codec functions in `base64`.No API or documentation changes are necessary withrespect to any functions in `base64`, and all existingunit tests for those functions continue to pass withoutmodification.Resolves:pythongh-146192
@serhiy-storchaka
Copy link
Member

You can now update your PR,@kangtastic.

@kangtastic
Copy link
ContributorAuthor

@serhiy-storchaka Already on it 😄

- Use the new `alphabet` parameter in `binascii`- Remove `binascii.a2b_base32hex()` and `binascii.b2a_base32hex()`- Change value for `.. versionadded::` ReST directive in docs for  new `binascii` functions to "next" instead of "3.15"
@kangtastickangtastic marked this pull request as ready for reviewMarch 20, 2026 16:03
Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I added some suggestions, but the core LGTM.

Please add assertions for new alphabets in test_constants.

kangtastic reacted with thumbs up emoji

.. function:: b2a_base32(data, /, *, alphabet=BASE32_ALPHABET)

Convert binary data to a line(s) of ASCII characters in base32 coding,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It is a single line.

I will addwrapcol in a separate issue.

kangtastic reacted with thumbs up emoji

Convert base32 data back to binary and return the binary data.

Valid base32 data:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This list is incomplete and redundant. I think it is better to follow the example of ascii85 and base85 (with a reference to the RFC). Mention that the mapping is case-sensitive and no optional mapping of the digit "0" and "1" to letters "O", "I" or "l" is used.

kangtastic reacted with thumbs up emoji

.. data:: BASE32_ALPHABET

The base32 alphabet according to :rfc:`4648`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
Thebase32 alphabet according to:rfc:`4648`.
TheBase 32 alphabet according to:rfc:`4648`.


.. data:: BASE32HEX_ALPHABET

The "Extended Hex" base32hex alphabet according to :rfc:`4648`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
The "Extended Hex"base32hex alphabet according to:rfc:`4648`.
The "Extended Hex"Base 32 alphabet according to:rfc:`4648`.

These are the names used in the table 3 and 4 captions in RFC 4648.

Oh, we can even refer directly to the table:

Suggested change
The "Extended Hex"base32hexalphabet according to:rfc:`4648`.
The "Extended Hex"Base 32alphabet according to:rfc:`4648`, table 4.

Add this also for Base 64 alphabets if you choose this variant.

Copy link
ContributorAuthor

@kangtastickangtasticMar 21, 2026
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I was wondering if this would come up. RFC 4648 uses all four of the terms "Base 32", "Base32", "base 32", and "base32" to refer to this encoding at various points, but it also states e.g.:

This encoding may be referred to as "base32hex". This encoding should not be regarded as the same as the "base32" encoding and should not be referred to as only "base32".

and e.g.:

One property with this alphabet, which the base64 and base32 alphabets lack...

thus implying that "base32" and "base32hex" are preferred, even if the rest of the document doesn't adhere to the implication.

Anyway, I'll refer to it as "Base 32" in docs for now to fit what's already there, and not reference the table number or touch any Base64 stuff so as to keep the scope of this PR limited.

Lib/base64.py Outdated
Comment on lines 212 to 213
if len(s) % 8:
raise binascii.Error('Incorrect padding')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Should not this be handled in the C code?

kangtastic reacted with thumbs up emoji
_b32rev[alphabet] = {v: k for k, v in enumerate(alphabet)}

def _b32decode_prepare(s, casefold=False, map01=None):
s = _bytes_from_decode_data(s)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is only needed if map01 is not None.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Correction: it is also needed if casefold is true, for input like 'ß' or 'ffi'.

Lib/base64.py Outdated
if alphabet not in _b32rev:
_b32rev[alphabet] = {v: k for k, v in enumerate(alphabet)}

def _b32decode_prepare(s, casefold=False, map01=None):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I suggest to inline this function. map01 handling is only needed for standard alphabet, and the code for casefold is trivial.

kangtastic reacted with thumbs up emoji
*
alphabet: Py_buffer(c_default="{NULL, NULL}") = BASE32_ALPHABET

base32-code line of data.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
base32-codelineofdata.
Base32-codelineofdata.

- Update docs to refer to "Base 32" and "Base32"- Update docs to better explain `binascii.a2b_base32()`- Inline helper function in `base64`- Add forgotten tests for presence of alphabet module globals
Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please add also the What's New entry.

kangtastic reacted with thumbs up emoji
* Contains no excess data after padding (including excess padding, newlines, etc.).
* Does not start with padding.
.. note::
By default, this function does not map lowercase characters (which are

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Remove "by default". There are no options for non-default behavior.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Wouldn't the user specifying an alternative alphabet be a non-default behavior?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It does not allow to map several characters to the same code.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It would work for folding 100% lowercase input. Maybe "By itself" instead of "By default"? But I don't mind removing it since it's such a rare hypothetical.

_b32rev[alphabet] = {v: k for k, v in enumerate(alphabet)}

def _b32decode_prepare(s, casefold=False, map01=None):
s = _bytes_from_decode_data(s)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Correction: it is also needed if casefold is true, for input like 'ß' or 'ffi'.

- Revise docs- Add whatsnew entry- Minor whitespace change in tests
Referring to a group of 8 bytes as an "octet" may causeconfusion, because the term is already commonly used insome languages to refer to a group of 8 bits (i.e. a byte)."Octa" is a suitable preexisting alternative for a group of64 bits [1] (used by Knuth himself, at that). "Octad" wasconsidered, but it, too, historically refers to a byte.Also rename "quintet" to "quint". "Pentad" was considered,but it historically refers to a group of 5 bits.[1]https://en.wikipedia.org/wiki/Units_of_information
Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM. 👍

kangtastic reacted with heart emoji
- Reword NEWS.d entry to "Base32" instead of "base-32".  No prior entries have ever mentioned "base-64", etc.,  but they have mentioned "Base64", etc., so this is  more consistent.- Reword whatsnew entry to "Base32" instead of "Base 32".  No prior entries have ever mentioned "Base 64", etc.,  and there is an entry a little further up mentioning  "Ascii85, Base85, and Z85", so this is more consistent.- Add a whatsnew entry in Optimizations > base64 & binascii  section.- Whitespace change in `binascii.c`.
When decoding invalid length (1, 3 or 6 mod 8) + no padding,mention the invalid length instead of the improper padding inthe exception message to match what the base64 decoder does.Additionally, move the logic for setting the exception message(back) outside the "slow path" loop; if we do end up checkingcanonicity of decoder input, it will feel (subjectively) betterto have several checks grouped together after the loop.
Copy link
Member

@gpsheadgpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

nice work!

kangtastic reacted with heart emoji
@serhiy-storchakaserhiy-storchaka merged commitb4e5bc2 intopython:mainMar 22, 2026
50 of 51 checks passed
@kangtastic
Copy link
ContributorAuthor

@serhiy-storchaka,@gpshead, thanks for the quick review! Doing more of this sort of thing might be fun. Stay safe out there.

CuriousLearner added a commit to CuriousLearner/cpython that referenced this pull requestMar 23, 2026
…8577* 'main' of github.com:python/cpython:pythongh-146197: Run -m test.pythoninfo on the Emscripten CI (python#146332)pythongh-146325: Use `test.support.requires_fork` in test_fastpath_cache_cleared_in_forked_child (python#146330)pythongh-146197: Add Emscripten to CI (python#146198)pythongh-143387: Raise an exception instead of returning None when metadata file is missing. (python#146234)pythongh-108907: ctypes: Document _type_ codes (pythonGH-145837)pythongh-146175: Soft-deprecate outdated macros; convert internal usage (pythonGH-146178)pythongh-146056: Rework ref counting in treebuilder_handle_end() (python#146167)  Add a warning about untrusted input to `configparser` docs (python#146276)pythongh-145264: Do not ignore excess Base64 data after the first padded quad (pythonGH-145267)pythongh-146308: Fix error handling issues in _remote_debugging module (python#146309)pythongh-146192: Add base32 support to binascii (pythonGH-146193)pythongh-135953: Properly obtain main thread identifier in Gecko Collector (python#146045)pythongh-143414: Implement unique reference tracking for JIT, optimize unpacking of such tuples (pythonGH-144300)pythongh-146261: Fix bug in `_Py_uop_sym_set_func_version` (pythonGH-146291)pythongh-145144: Add more tests for UserList, UserDict, etc (pythonGH-145145)pythongh-143959: Fix test_datetime if _datetime is unavailable (pythonGH-145248)pythongh-146245: Fix reference and buffer leaks via audit hook in socket module (pythonGH-146248)pythongh-140049: Colorize exception notes in `traceback.py` (python#140051)  Update docs forpythongh-146056 (pythonGH-146213)
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadgpshead approved these changes

@serhiy-storchakaserhiy-storchakaserhiy-storchaka approved these changes

@AA-TurnerAA-TurnerAwaiting requested review from AA-TurnerAA-Turner is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

C accelerator for Base32 character encoding

3 participants

@kangtastic@serhiy-storchaka@gpshead

[8]ページ先頭

©2009-2026 Movatter.jp