This change has no impact on performance, even if the new public API allocates memory on the heap, instead of allocating on the stack. It uses a freelist to optimizePyBytesWriter_Create().

Microbenchmark on 3 functions, to compare the private_PyBytesWriter (ref) to the new publicPyBytesWriter (change):

• bytes(list)
• bytes.fromhex(str)
• binascii.b2a_uu(bytes)

importpyperfimportbinasciirunner=pyperf.Runner()runner.bench_func('from list 100',bytes,list(b'x'*100))runner.bench_func('from list 1,000',bytes,list(b'x'*1_000))runner.bench_func('from hex 100',bytes.fromhex,bytes(range(100)).hex())runner.bench_func('from hex 1,000',bytes.fromhex, (b'x'*1_000).hex())runner.bench_func('b2a_uu',binascii.b2a_uu,b'x'*45)

Result:

Benchmark hidden because not significant (1): from list 1,000

Benchmark comparingPyBytes_FromStringAndSize(NULL, length) (ref) toPyBytesWriter_Create() (change).

Benchmark:

importpyperfSIZES= (10,100,500)runner=pyperf.Runner()forsizeinSIZES:large_int= (2** (size*8)-1)runner.bench_func(f'to_bytes({size})',large_int.to_bytes,size)forsizeinSIZES:mem=memoryview(b'x'*size)runner.bench_func(f'memoryview({size}).tobytes()',mem.tobytes)

Result:

It's hard to beatPyBytes_FromStringAndSize(NULL, length) performance, sincePyBytesWriter_Create() is a wrapper built on top ofPyBytes_FromStringAndSize(NULL, length).

There is an overhead around10 ns when usingPyBytesWriter.

Could you please benchmark the following?

• ASCII, Latin1 and UTF-8 encoders. For ASCII-only and non-ASCII data.
• The backslashreplace and xmlcharrefreplace error handlers (encoding).
• PyBytes_FromFormat(). Especially with few % formats and large raw data between them.
• PyBytes_DecodeEscape().

I wrote a big PR to show how PEP 782 would look like and how it's being used. But if PEP 782 is accepted, I will only start by adding the API without using it. Then I will write separated changes to use the new API and run benchmarks on each change.

ASCII, Latin1 and UTF-8 encoders. For ASCII-only and non-ASCII data.

I didn't modify these encoders, they still use the private_PyBytesWriter API.

The backslashreplace and xmlcharrefreplace error handlers (encoding).

Same.

If I modify these encoders and error handlers later, I will run benchmarks to decide if it's acceptable to use the public API or not.

Microbenchmark onPyBytes_FromFormat() andPyBytes_DecodeEscape() functions.

importpyperfrunner=pyperf.Runner()importctypesfromctypesimportpythonapi,py_objectfromctypesimport (c_int,c_uint,c_long,c_ulong,c_size_t,c_ssize_t,c_char_p)PyBytes_FromFormat=pythonapi.PyBytes_FromFormatPyBytes_FromFormat.argtypes= (c_char_p,)PyBytes_FromFormat.restype=py_objectPyBytes_DecodeEscape=pythonapi.PyBytes_DecodeEscapePyBytes_DecodeEscape.argtypes= (c_char_p,c_size_t,c_char_p,c_size_t,c_char_p)PyBytes_DecodeEscape.restype=py_objectrunner.bench_func('Format hello world',PyBytes_FromFormat,b'Hello %s !',b'world')fmt= (b'Hell%c'+b' '*1024+b' %s')runner.bench_func('Format long format',PyBytes_FromFormat,fmt,c_int(ord('o')),b'world')s=b'abc\\ndef\\x40.'runner.bench_func('Decode simple',PyBytes_DecodeEscape,s,len(s),None,0,b'unused')s=b'x'*1024runner.bench_func('Decode long copy',PyBytes_DecodeEscape,s,len(s),None,0,b'unused')s=b'\\x40'*1024runner.bench_func('Decode long\\x40',PyBytes_DecodeEscape,s,len(s),None,0,b'unused')

Results:

Benchmark hidden because not significant (1): Format hello world

I'm not sure why PEP 782 is faster, but at least it's not slower :-)

I build Python withgcc -O3 (without PGO, LTO, CPU isolation).

I started to split this huge PR into smaller PRs, see PRs attached to the issue#129813.