Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
gh-132983: Introduce_zstd bindings module#133027
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes from1 commit
Commits
Show all changes
59 commits Select commitHold shift + click to select a range
9814e3b Add _zstd module
emmatypingfda87c8 Add _zstd to modules
emmatyping887e564 Fix path for compression.zstd module
emmatypingcdba656 Ignore _zstd module like _io
emmatyping6b67e9b Expand module state macros to improve code quality
emmatypinga99a5d2 Remove backticks suggested in review
emmatyping02cd17a Use critical sections to lock object state
emmatyping54eca74 Remove compress/decompress and mark module as not reliant on the GIL
emmatypingf605956 Lift critical section to avoid clang warning
emmatyping2eadc65 Respond to comments by picnixz
emmatyping8eac354 Call out pyzstd explicitly in license description
emmatyping26775be Use a much more robust implementation...
emmatypingeae460f Use PyList_GetItemRef for thread safety purposes
emmatyping2ab5e4a Use a macro for the minimum supported version
emmatypingd5bf1c1 remove const from primivite types
emmatyping9e92b9f Use PyMem_New in another spot
emmatyping47f815a Simplify error handling in _get_frame_size
emmatyping6a4f7b8 Another simplification of error handling in get_frame_info
emmatypingd7b3805 Rename _module_state to mod_state
emmatypingc225ea6 Rewrite comment explaining the context of the code
emmatyping6e8c61c Add link to pyzstd
emmatypinge52ad06 Add TODO about refactoring dict training code
emmatyping2a1ad8b Use PyModule_AddObjectRef over PyModule_AddObject
emmatyping94473b9 Check result of OutputBufferGrow
emmatypinge2b2515 Simplify return logic in `add_constant_to_type`
emmatypingcd2f085 Ignore return value of _zstd_clear()
emmatyping79e174f Remove redundant comments
emmatypingce6f79c Remove __reduce__ from ZstdDict
emmatypinge15dd85 Use PyUnicode_FromFormat instead of a buffer
emmatyping685a3d1 Don't use C constants/types in error messages
emmatyping1b9f786 Make error messages easier to understand for Python users
emmatyping40c653c Lower minimum required version 1.4.0
emmatyping428677d Use casts and make slot function signatures correct
emmatyping0962bbb Be consistent with CPython on const usage
emmatyping85efc18 Make else clauses in line with PEP 7
emmatypingcadf6e4 Fix over-indented blocks in argument clinic
emmatypinge45c22a Merge branch 'main' into 3.14-zstd-c-code
emmatypingb9415be Merge branch 'main' into 3.14-zstd-c-code
emmatyping6760545 Add critical section around ZSTD_DCtx_setParameter
emmatypingc082d8a Add a TODO about refactoring critical sections
emmatypinge825285 Use Py_UNREACHABLE
emmatypingf02ff5a Move bytes operations out of Py_BEGIN_ALLOW_THREADS
emmatyping0d69c8c Add TODO about ensuring a lock is held
emmatyping58b0008 Remove asserts that may not be correct
emmatypingc786c3c Add TODO to make ZstdDict and others GC objects
emmatypinga9d57fc Make objects GC tracked
emmatypingcc9ab2a Remove unused include
emmatyping1f65d19 Fix some memory issues
emmatyping552da2d Fix refleaks on module and in ZstdDict
emmatypingc0648d8 Update configure to check for ZDICT_finalizeDictionary
emmatypingd149b2c Merge branch 'main' into 3.14-zstd-c-code
emmatypinga5ea379 Properly check version in configure
emmatyping0d88d6d exit(1) if check fails
emmatyping4a0a33e Use AC_RUN_IFELSE
emmatyping30b3934 Use a define() to re-use version check
emmatyping90b9172 Merge branch 'main' into 3.14-zstd-c-code
emmatyping8fdc7f7 Actually properly set _zstd module status based on version
emmatyping10e2e80 Merge branch 'main' into 3.14-zstd-c-code
emmatypingd90b29d Merge branch 'main' into 3.14-zstd-c-code
emmatypingFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
Expand module state macros to improve code quality
Also removes module state references from the classes in the _zstdmodule and instead uses PyType_GetModuleState()
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commit6b67e9bac8dfe7f9c82f0d3e2cb0646c8f2ee251
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -256,8 +256,10 @@ _zstd__train_dict_impl(PyObject *module, PyBytesObject *samples_bytes, | ||
| /* Check zstd dict error */ | ||
| if (ZDICT_isError(zstd_ret)) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| if (_module_state != NULL) { | ||
| set_zstd_error(_module_state, ERR_TRAIN_DICT, zstd_ret); | ||
| } | ||
| goto error; | ||
| } | ||
| @@ -384,8 +386,10 @@ _zstd__finalize_dict_impl(PyObject *module, PyBytesObject *custom_dict_bytes, | ||
emmatyping marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| /* Check zstd dict error */ | ||
| if (ZDICT_isError(zstd_ret)) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state != NULL) { | ||
| set_zstd_error(_module_state, ERR_FINALIZE_DICT, zstd_ret); | ||
| } | ||
| goto error; | ||
| } | ||
| @@ -425,15 +429,19 @@ _zstd__get_param_bounds_impl(PyObject *module, int is_compress, | ||
| if (is_compress) { | ||
| bound = ZSTD_cParam_getBounds(parameter); | ||
| if (ZSTD_isError(bound.error)) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state != NULL) { | ||
| set_zstd_error(_module_state, ERR_GET_C_BOUNDS, bound.error); | ||
| } | ||
| return NULL; | ||
| } | ||
| } else { | ||
| bound = ZSTD_dParam_getBounds(parameter); | ||
| if (ZSTD_isError(bound.error)) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state != NULL) { | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| set_zstd_error(_module_state, ERR_GET_D_BOUNDS, bound.error); | ||
| } | ||
| return NULL; | ||
| } | ||
| } | ||
| @@ -462,13 +470,15 @@ _zstd_get_frame_size_impl(PyObject *module, Py_buffer *frame_buffer) | ||
| frame_size = ZSTD_findFrameCompressedSize(frame_buffer->buf, frame_buffer->len); | ||
| if (ZSTD_isError(frame_size)) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state != NULL) { | ||
| PyErr_Format(_module_state->ZstdError, | ||
| "Error when finding the compressed size of a zstd frame. " | ||
| "Make sure the frame_buffer argument starts from the " | ||
| "beginning of a frame, and its length not less than this " | ||
| "complete frame. Zstd error message: %s.", | ||
| ZSTD_getErrorName(frame_size)); | ||
| } | ||
| goto error; | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| } | ||
| @@ -508,12 +518,14 @@ _zstd__get_frame_info_impl(PyObject *module, Py_buffer *frame_buffer) | ||
| /* #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1) | ||
| #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2) */ | ||
| if (decompressed_size == ZSTD_CONTENTSIZE_ERROR) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state != NULL) { | ||
| PyErr_SetString(_module_state->ZstdError, | ||
| "Error when getting information from the header of " | ||
| "a zstd frame. Make sure the frame_buffer argument " | ||
| "starts from the beginning of a frame, and its length " | ||
| "not less than the frame header (6~18 bytes)."); | ||
| } | ||
| goto error; | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| } | ||
| @@ -553,7 +565,10 @@ _zstd__set_parameter_types_impl(PyObject *module, PyObject *c_parameter_type, | ||
| PyObject *d_parameter_type) | ||
| /*[clinic end generated code: output=a13d4890ccbd2873 input=3e7d0d37c3a1045a]*/ | ||
| { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| if (_module_state == NULL) { | ||
| return NULL; | ||
| } | ||
| if (!PyType_Check(c_parameter_type) || !PyType_Check(d_parameter_type)) { | ||
| PyErr_SetString(PyExc_ValueError, | ||
| @@ -562,13 +577,13 @@ _zstd__set_parameter_types_impl(PyObject *module, PyObject *c_parameter_type, | ||
| return NULL; | ||
| } | ||
| Py_XDECREF(_module_state->CParameter_type); | ||
| Py_INCREF(c_parameter_type); | ||
| _module_state->CParameter_type = (PyTypeObject*)c_parameter_type; | ||
| Py_XDECREF(_module_state->DParameter_type); | ||
| Py_INCREF(d_parameter_type); | ||
| _module_state->DParameter_type = (PyTypeObject*)d_parameter_type; | ||
| Py_RETURN_NONE; | ||
| } | ||
| @@ -598,20 +613,21 @@ _zstd_compress_impl(PyObject *module, Py_buffer *data, PyObject *level, | ||
| PyObject *options, PyObject *zstd_dict) | ||
| /*[clinic end generated code: output=0cca9399ca5c95cc input=e8a7c59073af923c]*/ | ||
| { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state == NULL) { | ||
| return NULL; | ||
| } | ||
emmatyping marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| PyObject *ret = NULL; | ||
| ZstdCompressor self = {0}; | ||
| /* Initialize & set ZstdCompressor */ | ||
| self.cctx = ZSTD_createCCtx(); | ||
| if (self.cctx == NULL) { | ||
| PyErr_SetString(_module_state->ZstdError, | ||
| "Unable to create ZSTD_CCtx instance."); | ||
| goto error; | ||
| } | ||
| if (level != Py_None && options != Py_None) { | ||
| PyErr_SetString(PyExc_RuntimeError, "Only one of level or options should be used."); | ||
| return NULL; | ||
| @@ -679,20 +695,21 @@ _zstd_decompress_impl(PyObject *module, Py_buffer *data, PyObject *zstd_dict, | ||
| Py_ssize_t initial_size; | ||
| ZstdDecompressor self = {0}; | ||
| ZSTD_inBuffer in; | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state == NULL) { | ||
| return NULL; | ||
| } | ||
| PyObject *ret = NULL; | ||
| /* Initialize & set ZstdDecompressor */ | ||
| self.dctx = ZSTD_createDCtx(); | ||
| if (self.dctx == NULL) { | ||
| PyErr_SetString(_module_state->ZstdError, | ||
| "Unable to create ZSTD_DCtx instance."); | ||
| goto error; | ||
| } | ||
| self.at_frame_edge = 1; | ||
| /* Load dictionary to decompression context */ | ||
| if (zstd_dict != Py_None) { | ||
| if (_PyZstd_load_d_dict(&self, zstd_dict) < 0) { | ||
| @@ -736,7 +753,7 @@ _zstd_decompress_impl(PyObject *module, Py_buffer *data, PyObject *zstd_dict, | ||
| char *extra_msg = (Py_SIZE(ret) == 0) ? "." : | ||
| ", if want to output these decompressed data, use " | ||
| "the ZstdDecompressor class to decompress."; | ||
| PyErr_Format(_module_state->ZstdError, | ||
| "Decompression failed: zstd data ends in an incomplete " | ||
| "frame, maybe the input data was truncated. Decompressed " | ||
| "data is %zd bytes%s", | ||
| @@ -916,8 +933,8 @@ add_vars_to_module(PyObject *module) | ||
| #define ADD_STR_TO_STATE_MACRO(STR) \ | ||
| do { \ | ||
| _module_state->str_##STR = PyUnicode_FromString(#STR); \ | ||
| if (_module_state->str_##STR == NULL) { \ | ||
| return -1; \ | ||
| } \ | ||
| } while(0) | ||
| @@ -959,17 +976,20 @@ add_constant_to_type(PyTypeObject *type, const char *name, const long value) | ||
| } | ||
| static int _zstd_exec(PyObject *module) { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state == NULL) { | ||
| return -1; | ||
| } | ||
| /* Reusable objects & variables */ | ||
| _module_state->empty_bytes = PyBytes_FromStringAndSize(NULL, 0); | ||
| if (_module_state->empty_bytes == NULL) { | ||
| return -1; | ||
| } | ||
| _module_state->empty_readonly_memoryview = | ||
| PyMemoryView_FromMemory((char*)_module_state, 0, PyBUF_READ); | ||
| if (_module_state->empty_readonly_memoryview == NULL) { | ||
| return -1; | ||
| } | ||
| @@ -979,59 +999,59 @@ static int _zstd_exec(PyObject *module) { | ||
| ADD_STR_TO_STATE_MACRO(write); | ||
| ADD_STR_TO_STATE_MACRO(flush); | ||
| _module_state->CParameter_type = NULL; | ||
| _module_state->DParameter_type = NULL; | ||
| /* Add variables to module */ | ||
| if (add_vars_to_module(module) < 0) { | ||
| return -1; | ||
| } | ||
| /* ZstdError */ | ||
| _module_state->ZstdError = PyErr_NewExceptionWithDoc( | ||
| "_zstd.ZstdError", | ||
| "Call to the underlying zstd library failed.", | ||
| NULL, NULL); | ||
| if (_module_state->ZstdError == NULL) { | ||
| return -1; | ||
| } | ||
| Py_INCREF(_module_state->ZstdError); | ||
| if (PyModule_AddObject(module, "ZstdError",_module_state->ZstdError) < 0) { | ||
| Py_DECREF(_module_state->ZstdError); | ||
| return -1; | ||
| } | ||
| /* ZstdDict */ | ||
| if (add_type_to_module(module, | ||
| "ZstdDict", | ||
| &zstddict_type_spec, | ||
| &_module_state->ZstdDict_type) < 0) { | ||
| return -1; | ||
| } | ||
| // ZstdCompressor | ||
| if (add_type_to_module(module, | ||
| "ZstdCompressor", | ||
| &zstdcompressor_type_spec, | ||
| &_module_state->ZstdCompressor_type) < 0) { | ||
| return -1; | ||
| } | ||
| // Add EndDirective enum to ZstdCompressor | ||
| if (add_constant_to_type(_module_state->ZstdCompressor_type, | ||
| "CONTINUE", | ||
| ZSTD_e_continue) < 0) { | ||
| return -1; | ||
| } | ||
| if (add_constant_to_type(_module_state->ZstdCompressor_type, | ||
| "FLUSH_BLOCK", | ||
| ZSTD_e_flush) < 0) { | ||
| return -1; | ||
| } | ||
| if (add_constant_to_type(_module_state->ZstdCompressor_type, | ||
| "FLUSH_FRAME", | ||
| ZSTD_e_end) < 0) { | ||
| return -1; | ||
| @@ -1041,7 +1061,7 @@ static int _zstd_exec(PyObject *module) { | ||
| if (add_type_to_module(module, | ||
| "ZstdDecompressor", | ||
| &ZstdDecompressor_type_spec, | ||
| &_module_state->ZstdDecompressor_type) < 0) { | ||
| return -1; | ||
| } | ||
| @@ -1051,48 +1071,54 @@ static int _zstd_exec(PyObject *module) { | ||
| static int | ||
| _zstd_traverse(PyObject *module, visitproc visit, void *arg) | ||
| { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state == NULL) { | ||
| return -1; | ||
| } | ||
| Py_VISIT(_module_state->empty_bytes); | ||
| Py_VISIT(_module_state->empty_readonly_memoryview); | ||
| Py_VISIT(_module_state->str_read); | ||
| Py_VISIT(_module_state->str_readinto); | ||
| Py_VISIT(_module_state->str_write); | ||
| Py_VISIT(_module_state->str_flush); | ||
| Py_VISIT(_module_state->ZstdDict_type); | ||
| Py_VISIT(_module_state->ZstdCompressor_type); | ||
| Py_VISIT(_module_state->ZstdDecompressor_type); | ||
| Py_VISIT(_module_state->ZstdError); | ||
| Py_VISIT(_module_state->CParameter_type); | ||
| Py_VISIT(_module_state->DParameter_type); | ||
| return 0; | ||
| } | ||
| static int | ||
| _zstd_clear(PyObject *module) | ||
| { | ||
| _zstd_state* const _module_state = PyModule_GetState(module); | ||
| if (_module_state == NULL) { | ||
| return -1; | ||
| } | ||
| Py_CLEAR(_module_state->empty_bytes); | ||
| Py_CLEAR(_module_state->empty_readonly_memoryview); | ||
| Py_CLEAR(_module_state->str_read); | ||
| Py_CLEAR(_module_state->str_readinto); | ||
| Py_CLEAR(_module_state->str_write); | ||
| Py_CLEAR(_module_state->str_flush); | ||
| Py_CLEAR(_module_state->ZstdDict_type); | ||
| Py_CLEAR(_module_state->ZstdCompressor_type); | ||
| Py_CLEAR(_module_state->ZstdDecompressor_type); | ||
| Py_CLEAR(_module_state->ZstdError); | ||
| Py_CLEAR(_module_state->CParameter_type); | ||
| Py_CLEAR(_module_state->DParameter_type); | ||
| return 0; | ||
| } | ||
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.