Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-132983: Introduce_zstd bindings module#133027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
gpshead merged 59 commits intopython:mainfromemmatyping:3.14-zstd-c-code
May 4, 2025
Merged
Show file tree
Hide file tree
Changes from1 commit
Commits
Show all changes
59 commits
Select commitHold shift + click to select a range
9814e3b
Add _zstd module
emmatypingApr 26, 2025
fda87c8
Add _zstd to modules
emmatypingApr 26, 2025
887e564
Fix path for compression.zstd module
emmatypingApr 26, 2025
cdba656
Ignore _zstd module like _io
emmatypingApr 26, 2025
6b67e9b
Expand module state macros to improve code quality
emmatypingApr 26, 2025
a99a5d2
Remove backticks suggested in review
emmatypingApr 27, 2025
02cd17a
Use critical sections to lock object state
emmatypingApr 27, 2025
54eca74
Remove compress/decompress and mark module as not reliant on the GIL
emmatypingApr 27, 2025
f605956
Lift critical section to avoid clang warning
emmatypingApr 27, 2025
2eadc65
Respond to comments by picnixz
emmatypingApr 27, 2025
8eac354
Call out pyzstd explicitly in license description
emmatypingApr 27, 2025
26775be
Use a much more robust implementation...
emmatypingApr 27, 2025
eae460f
Use PyList_GetItemRef for thread safety purposes
emmatypingApr 27, 2025
2ab5e4a
Use a macro for the minimum supported version
emmatypingApr 27, 2025
d5bf1c1
remove const from primivite types
emmatypingApr 27, 2025
9e92b9f
Use PyMem_New in another spot
emmatypingApr 27, 2025
47f815a
Simplify error handling in _get_frame_size
emmatypingApr 27, 2025
6a4f7b8
Another simplification of error handling in get_frame_info
emmatypingApr 27, 2025
d7b3805
Rename _module_state to mod_state
emmatypingApr 27, 2025
c225ea6
Rewrite comment explaining the context of the code
emmatypingApr 28, 2025
6e8c61c
Add link to pyzstd
emmatypingApr 28, 2025
e52ad06
Add TODO about refactoring dict training code
emmatypingApr 28, 2025
2a1ad8b
Use PyModule_AddObjectRef over PyModule_AddObject
emmatypingApr 28, 2025
94473b9
Check result of OutputBufferGrow
emmatypingApr 28, 2025
e2b2515
Simplify return logic in `add_constant_to_type`
emmatypingApr 29, 2025
cd2f085
Ignore return value of _zstd_clear()
emmatypingApr 29, 2025
79e174f
Remove redundant comments
emmatypingApr 29, 2025
ce6f79c
Remove __reduce__ from ZstdDict
emmatypingApr 29, 2025
e15dd85
Use PyUnicode_FromFormat instead of a buffer
emmatypingApr 29, 2025
685a3d1
Don't use C constants/types in error messages
emmatypingApr 29, 2025
1b9f786
Make error messages easier to understand for Python users
emmatypingApr 29, 2025
40c653c
Lower minimum required version 1.4.0
emmatypingApr 30, 2025
428677d
Use casts and make slot function signatures correct
emmatypingApr 30, 2025
0962bbb
Be consistent with CPython on const usage
emmatypingApr 30, 2025
85efc18
Make else clauses in line with PEP 7
emmatypingApr 30, 2025
cadf6e4
Fix over-indented blocks in argument clinic
emmatypingApr 30, 2025
e45c22a
Merge branch 'main' into 3.14-zstd-c-code
emmatypingApr 30, 2025
b9415be
Merge branch 'main' into 3.14-zstd-c-code
emmatypingApr 30, 2025
6760545
Add critical section around ZSTD_DCtx_setParameter
emmatypingMay 1, 2025
c082d8a
Add a TODO about refactoring critical sections
emmatypingMay 1, 2025
e825285
Use Py_UNREACHABLE
emmatypingMay 1, 2025
f02ff5a
Move bytes operations out of Py_BEGIN_ALLOW_THREADS
emmatypingMay 2, 2025
0d69c8c
Add TODO about ensuring a lock is held
emmatypingMay 2, 2025
58b0008
Remove asserts that may not be correct
emmatypingMay 2, 2025
c786c3c
Add TODO to make ZstdDict and others GC objects
emmatypingMay 2, 2025
a9d57fc
Make objects GC tracked
emmatypingMay 2, 2025
cc9ab2a
Remove unused include
emmatypingMay 2, 2025
1f65d19
Fix some memory issues
emmatypingMay 2, 2025
552da2d
Fix refleaks on module and in ZstdDict
emmatypingMay 2, 2025
c0648d8
Update configure to check for ZDICT_finalizeDictionary
emmatypingMay 3, 2025
d149b2c
Merge branch 'main' into 3.14-zstd-c-code
emmatypingMay 3, 2025
a5ea379
Properly check version in configure
emmatypingMay 3, 2025
0d88d6d
exit(1) if check fails
emmatypingMay 3, 2025
4a0a33e
Use AC_RUN_IFELSE
emmatypingMay 3, 2025
30b3934
Use a define() to re-use version check
emmatypingMay 3, 2025
90b9172
Merge branch 'main' into 3.14-zstd-c-code
emmatypingMay 3, 2025
8fdc7f7
Actually properly set _zstd module status based on version
emmatypingMay 3, 2025
10e2e80
Merge branch 'main' into 3.14-zstd-c-code
emmatypingMay 3, 2025
d90b29d
Merge branch 'main' into 3.14-zstd-c-code
emmatypingMay 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
PrevPrevious commit
NextNext commit
Be consistent with CPython on const usage
  • Loading branch information
@emmatyping
emmatyping committedApr 30, 2025
commit0962bbb0dc04bcd23fef42fe26db1672a976446d
6 changes: 3 additions & 3 deletionsModules/_zstd/_zstdmodule.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -20,7 +20,7 @@ module _zstd
/* Format error message and set ZstdError. */
void
set_zstd_error(const _zstd_state* const state,
consterror_type type, const size_t zstd_ret)
error_type type, size_t zstd_ret)
{
char *msg;
assert(ZSTD_isError(zstd_ret));
Expand DownExpand Up@@ -610,7 +610,7 @@ add_parameters(PyObject *module)
static inline PyObject *
get_zstd_version_info(void)
{
constuint32_t ver = ZSTD_versionNumber();
uint32_t ver = ZSTD_versionNumber();
uint32_t major, minor, release;

major = ver / 10000;
Expand DownExpand Up@@ -723,7 +723,7 @@ add_type_to_module(PyObject *module, const char *name,
}

static inline int
add_constant_to_type(PyTypeObject *type, const char *name,constlong value)
add_constant_to_type(PyTypeObject *type, const char *name, long value)
{
PyObject *temp;

Expand Down
10 changes: 5 additions & 5 deletionsModules/_zstd/_zstdmodule.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -171,7 +171,7 @@ mt_continue_should_break(ZSTD_inBuffer *in, ZSTD_outBuffer *out) {
/* Format error message and set ZstdError. */
extern void
set_zstd_error(const _zstd_state* const state,
const error_type type,constsize_t zstd_ret);
const error_type type, size_t zstd_ret);

extern void
set_parameter_error(const _zstd_state* const state, int is_compress,
Expand All@@ -194,10 +194,10 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options);

extern PyObject *
decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,
constPy_ssize_t max_length,
constPy_ssize_t initial_size,
constdecompress_type type);
Py_ssize_t max_length,
Py_ssize_t initial_size,
decompress_type type);

extern PyObject *
compress_impl(ZstdCompressor *self, Py_buffer *data,
constZSTD_EndDirective end_directive);
ZSTD_EndDirective end_directive);
6 changes: 3 additions & 3 deletionsModules/_zstd/buffer.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -13,7 +13,7 @@ Python module.
Return -1 on failure */
static inline int
_OutputBuffer_InitAndGrow(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
constPy_ssize_t max_length)
Py_ssize_t max_length)
{
/* Ensure .list was set to NULL */
assert(buffer->list == NULL);
Expand All@@ -33,8 +33,8 @@ _OutputBuffer_InitAndGrow(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
Return -1 on failure */
static inline int
_OutputBuffer_InitWithSize(_BlocksOutputBuffer *buffer, ZSTD_outBuffer *ob,
constPy_ssize_t max_length,
constPy_ssize_t init_size)
Py_ssize_t max_length,
Py_ssize_t init_size)
{
Py_ssize_t block_size;

Expand Down
8 changes: 4 additions & 4 deletionsModules/_zstd/compressor.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -37,7 +37,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,

/* Integer compression level */
if (PyLong_Check(level_or_options)) {
constint level = PyLong_AsInt(level_or_options);
int level = PyLong_AsInt(level_or_options);
if (level == -1 && PyErr_Occurred()) {
PyErr_Format(PyExc_ValueError,
"Compression level should be an int value between %d and %d.",
Expand DownExpand Up@@ -75,7 +75,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,
return -1;
}

constint key_v = PyLong_AsInt(key);
int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Key of options dict should be a CParameter attribute.");
Expand All@@ -84,7 +84,7 @@ _PyZstd_set_c_parameters(ZstdCompressor *self, PyObject *level_or_options,

// TODO(emmatyping): check bounds when there is a value error here for better
// error message?
constint value_v = PyLong_AsInt(value);
int value_v = PyLong_AsInt(value);
if (value_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Value of option dict should be an int.");
Expand DownExpand Up@@ -399,7 +399,7 @@ _zstd_ZstdCompressor___init___impl(ZstdCompressor *self, PyObject *level,

PyObject *
compress_impl(ZstdCompressor *self, Py_buffer *data,
constZSTD_EndDirective end_directive)
ZSTD_EndDirective end_directive)
{
ZSTD_inBuffer in;
ZSTD_outBuffer out;
Expand Down
24 changes: 12 additions & 12 deletionsModules/_zstd/decompressor.c
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -87,7 +87,7 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)
}

/* Both key & value should be 32-bit signed int */
constint key_v = PyLong_AsInt(key);
int key_v = PyLong_AsInt(key);
if (key_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Key of options dict should be a DParameter attribute.");
Expand All@@ -96,7 +96,7 @@ _PyZstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)

// TODO(emmatyping): check bounds when there is a value error here for better
// error message?
constint value_v = PyLong_AsInt(value);
int value_v = PyLong_AsInt(value);
if (value_v == -1 && PyErr_Occurred()) {
PyErr_SetString(PyExc_ValueError,
"Value of options dict should be an int.");
Expand DownExpand Up@@ -269,9 +269,9 @@ _PyZstd_load_d_dict(ZstdDecompressor *self, PyObject *dict)
*/
PyObject *
decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,
constPy_ssize_t max_length,
constPy_ssize_t initial_size,
constdecompress_type type)
Py_ssize_t max_length,
Py_ssize_t initial_size,
decompress_type type)
{
size_t zstd_ret;
ZSTD_outBuffer out;
Expand DownExpand Up@@ -370,7 +370,7 @@ decompress_impl(ZstdDecompressor *self, ZSTD_inBuffer *in,

void
decompressor_reset_session(ZstdDecompressor *self,
constdecompress_type type)
decompress_type type)
{
/* Reset variables */
self->in_begin = 0;
Expand All@@ -392,7 +392,7 @@ decompressor_reset_session(ZstdDecompressor *self,

PyObject *
stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length,
constdecompress_type type)
decompress_type type)
{
Py_ssize_t initial_buffer_size = -1;
ZSTD_inBuffer in;
Expand DownExpand Up@@ -449,21 +449,21 @@ stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length
use_input_buffer = 1;

/* Unconsumed data size in input_buffer */
constsize_t used_now = self->in_end - self->in_begin;
size_t used_now = self->in_end - self->in_begin;
assert(self->in_end > self->in_begin);

/* Number of bytes we can append to input buffer */
constsize_t avail_now = self->input_buffer_size - self->in_end;
size_t avail_now = self->input_buffer_size - self->in_end;
assert(self->input_buffer_size >= self->in_end);

/* Number of bytes we can append if we move existing contents to
beginning of buffer */
constsize_t avail_total = self->input_buffer_size - used_now;
size_t avail_total = self->input_buffer_size - used_now;
assert(self->input_buffer_size >= used_now);

if (avail_total < (size_t) data->len) {
char *tmp;
constsize_t new_size = used_now + data->len;
size_t new_size = used_now + data->len;

/* Allocate with new size */
tmp = PyMem_Malloc(new_size);
Expand DownExpand Up@@ -537,7 +537,7 @@ stream_decompress(ZstdDecompressor *self, Py_buffer *data, Py_ssize_t max_length
self->in_end = 0;
}
} else {
constsize_t data_size = in.size - in.pos;
size_t data_size = in.size - in.pos;

self->needs_input = 0;

Expand Down

[8]ページ先頭

©2009-2025 Movatter.jp