Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
gh-119127: functools.partial placeholders#119827
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes from1 commit
Commits
Show all changes
78 commits Select commitHold shift + click to select a range
ee7333c Initial Implementation
dg-pb8bcc462 serialization fix
dg-pbc67c9b4 bug fix
dg-pb680d900 Bug 2 fix
dg-pb9591ff5 Py_TPFLAGS_IMMUTABLETYPE added
dg-pb067e938 placeholder added to state as opposed to being used as global constant
dg-pb8af20b3 static removed
dg-pb607a0b1 creating sentinel via PyType_Spec
dg-pbf55801e more accurate variable name
dg-pb5894145 trailing trim bug and tests
dg-pb3722e07 Updated docs
dg-pba79c2af blurb
dg-pb12aaa72 Merge branch 'main' into implement-119127
92c767b minor edit
dg-pb496a9d2 doc fix
dg-pb38d9c11 better variable names and mini corrections
dg-pb707b957 Merge branch 'implement-119127' into implement-119127-again
dg-pb14b38ca review comments mostly
dg-pb32bca19 singleton sentinel and reduce
dg-pb8576493 python module sentinel better mimics the one of the extension
dg-pba3fd2d6 Emulated None behaviour, but using PyType_FromModuleAndSpec
dg-pb0852993 review feedback
dg-pb6fea348 included constant into tsv
dg-pbcaec6e8 documentation update
dg-pb115b8c5 review edits
dg-pb3f5f00b trailing placeholder prohibition and small changes
dg-pb202c929 change constant name in ignores
dg-pb2c16d38 PlaceholderType Hidden
dg-pb400ff55 support 4-arg pre-placeholder state
dg-pb8ccc38f better variable names
dg-pbe7c82c7 partialmethod impl
dg-pbc9b7ef3 fix tests
dg-pbe59d711 adjust inspect to partial Placeholders
dg-pb7bfc591 arg alignment
dg-pb7957a97 small fixes
dg-pb8aaee6a pickle compatibility ensured
dg-pbfe8e0ad trailing placeholder test for __setstate__
dg-pb00dd80e rough implementation rolled back and example of successive applicatio…
dg-pbd352cfa small doc edits
dg-pb9038ed5 example changes
dg-pb49b8c71 serialization issues addressed
dg-pbbc1fdbd delete redundant references
dg-pb3067221 simplify PyPlaceholder implementation
dg-pb1185510 more optimal python functools.partial
dg-pb266b4fa placeholder arg and pre-placeholder instance conversions to positional
dg-pbdd58a12 unittest.mock.ANY test for Placeholder
dg-pb5971fbb functools.partial.__get__
dg-pb9033650 Revert "functools.partial.__get__"
dg-pbd31e5d1 Merge branch 'main' into implement-119127-again
dg-pba3d39b0 review changes
dg-pb9e4c5df factor out repr. same to be used for partial and partialmethod
dg-pb16f12f8 microopt & args.count(Placeholder) can not be used as it uses __eq__
dg-pb82dd600 simplify preparation
dg-pbf9cb653 whatsnew + minor doc edit
dg-pbd255524 typo fixes
dg-pb404044e whatsnew edit
dg-pb800217b revert stylistic changes
dg-pb38ee450 factor out full __new__
dg-pb11f47db Merge branch 'main' into implement-119127-again
serhiy-storchaka3c872bd Merge branch 'main' into implement-119127-again
serhiy-storchakafd16189 CR part 1
dg-pba6c6ef2 CR Part 2
dg-pb1c8d73e remove ignored global var
dg-pba8bd3ae CR changes
dg-pb70e47ed small CR changes and doc updates
dg-pb2eacf5e push placeholder check to earlier place
dg-pbf78d8d3 more appropriate test functions and better doc example
dg-pb0a8640e minor fixes; message, doc polish
dg-pb6e3d282 better doc example and small test changes
dg-pb66c305d assertRaisesRegex exact match
dg-pb14bf68c doc nits
dg-pbee642d5 Add Placeholder to __all__
rhettinger8d6c28e Update copyright span
rhettinger8744bcb Minor doc edits and add doctests
rhettingerb896470 Merge branch 'main' into implement-119127-again
rhettinger4881ae6 Merge branch 'main' into implement-119127-again
rhettingerc3ad7d9 Having the separate dict was necessary to eliminate duplicate keywords
rhettinger5e5d484 Merge branch 'main' into implement-119127-again
rhettingerFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
NextNext commit
Initial Implementation
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commitee7333c124cfcf1abb9eccbc4b52969f443fa59d
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -273,6 +273,11 @@ def reduce(function, sequence, initial=_initial_missing): | ||
| ### partial() argument application | ||
| ################################################################################ | ||
| class Placeholder: | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| """placeholder for partial arguments""" | ||
| # Purely functional, no descriptor behaviour | ||
| class partial: | ||
| """New function with partial application of the given arguments | ||
| @@ -285,8 +290,33 @@ def __new__(cls, func, /, *args, **keywords): | ||
| if not callable(func): | ||
| raise TypeError("the first argument must be callable") | ||
| np = 0 | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| nargs = len(args) | ||
| if args: | ||
| while nargs and args[nargs-1] is Placeholder: | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| nargs -= 1 | ||
| args = args[:nargs] | ||
| np = args.count(Placeholder) | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| if isinstance(func, partial): | ||
| pargs = func.args | ||
| pnp = func.np | ||
| if pnp and args: | ||
| all_args = list(pargs) | ||
| nargs = len(args) | ||
| j, pos = 0, 0 | ||
| end = nargs if nargs < pnp else pnp | ||
| while j < end: | ||
| pos = all_args.index(Placeholder, pos) | ||
| all_args[pos] = args[j] | ||
| j += 1 | ||
| pos += 1 | ||
| if pnp < nargs: | ||
| all_args.extend(args[pnp:]) | ||
| np += pnp - end | ||
| args = tuple(all_args) | ||
| else: | ||
| np += pnp | ||
| args = func.args + args | ||
| keywords = {**func.keywords, **keywords} | ||
| func = func.func | ||
| @@ -295,11 +325,25 @@ def __new__(cls, func, /, *args, **keywords): | ||
| self.func = func | ||
| self.args = args | ||
| self.keywords = keywords | ||
| self.np = np | ||
| return self | ||
| def __call__(self, /, *args, **keywords): | ||
| if np := self.np: | ||
| if len(args) < np: | ||
| raise ValueError("unfilled placeholders in 'partial' call") | ||
| f_args = list(self.args) | ||
| j, pos = 0, 0 | ||
| while j < np: | ||
| pos = f_args.index(Placeholder, pos) | ||
| f_args[pos] = args[j] | ||
| j += 1 | ||
| pos += 1 | ||
| keywords = {**self.keywords, **keywords} | ||
| return self.func(*f_args, *args[np:], **keywords) | ||
| else: | ||
| keywords = {**self.keywords, **keywords} | ||
| return self.func(*self.args, *args, **keywords) | ||
| @recursive_repr() | ||
| def __repr__(self): | ||
| @@ -340,7 +384,7 @@ def __setstate__(self, state): | ||
| self.keywords = kwds | ||
| try: | ||
| from _functools import partial, Placeholder | ||
| except ImportError: | ||
| pass | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -41,13 +41,31 @@ get_functools_state(PyObject *module) | ||
| /* partial object **********************************************************/ | ||
| typedef struct { | ||
| PyObject_HEAD | ||
| } placeholderobject; | ||
| static PyTypeObject placeholder_type = { | ||
| .ob_base = PyVarObject_HEAD_INIT(NULL, 0) | ||
| .tp_name = "functools.Placeholder", | ||
| .tp_doc = PyDoc_STR("placeholder for partial arguments"), | ||
| .tp_basicsize = sizeof(placeholderobject), | ||
| .tp_itemsize = 0, | ||
| .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_DISALLOW_INSTANTIATION, | ||
| .tp_new = PyType_GenericNew, | ||
| }; | ||
| typedef struct { | ||
| PyObject_HEAD | ||
| PyObject *fn; | ||
| PyObject *args; | ||
| PyObject *kw; | ||
| PyObject *dict; /* __dict__ */ | ||
| PyObject *weakreflist; /* List of weak references */ | ||
| Py_ssize_t np; /* Number of placeholders */ | ||
| vectorcallfunc vectorcall; | ||
| } partialobject; | ||
| @@ -72,6 +90,7 @@ partial_new(PyTypeObject *type, PyObject *args, PyObject *kw) | ||
| { | ||
| PyObject *func, *pargs, *nargs, *pkw; | ||
| partialobject *pto; | ||
| Py_ssize_t pnp = 0; | ||
| if (PyTuple_GET_SIZE(args) < 1) { | ||
| PyErr_SetString(PyExc_TypeError, | ||
| @@ -98,6 +117,7 @@ partial_new(PyTypeObject *type, PyObject *args, PyObject *kw) | ||
| pargs = part->args; | ||
| pkw = part->kw; | ||
| func = part->fn; | ||
| pnp = part->np; | ||
| assert(PyTuple_Check(pargs)); | ||
| assert(PyDict_Check(pkw)); | ||
| } | ||
| @@ -120,11 +140,57 @@ partial_new(PyTypeObject *type, PyObject *args, PyObject *kw) | ||
| Py_DECREF(pto); | ||
| return NULL; | ||
| } | ||
| Py_ssize_t nnp = 0; | ||
| Py_ssize_t nnargs = PyTuple_GET_SIZE(nargs); | ||
| PyObject *item; | ||
| if (nnargs > 0){ | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| Py_ssize_t i; | ||
| for (i=nnargs; i > 0; i--) { | ||
| item = PyTuple_GET_ITEM(nargs, i); | ||
| if (!Py_Is(item, (PyObject *) &placeholder_type)) | ||
| break; | ||
| } | ||
| if (i != nnargs) | ||
| nnargs = i; | ||
| if (nnargs > 0){ | ||
| for (Py_ssize_t i=0; i < nnargs; i++){ | ||
| item = PyTuple_GET_ITEM(nargs, i); | ||
| nnp += Py_Is(item, (PyObject *) &placeholder_type); | ||
| } | ||
| } | ||
| } | ||
| if ((pnp > 0) & (nnargs > 0)) { | ||
| Py_ssize_t pnargs = PyTuple_GET_SIZE(pargs); | ||
| Py_ssize_t anargs = pnargs; | ||
| if (nnargs > pnp) | ||
| anargs += nnargs - pnp; | ||
| PyObject *aargs = PyTuple_New(anargs); | ||
| Py_ssize_t j = 0; | ||
| for (Py_ssize_t i=0; i < anargs; i++) { | ||
| if (i < pnargs) { | ||
| item = PyTuple_GET_ITEM(pargs, i); | ||
| if ((j < nnargs) & Py_Is(item, (PyObject *) &placeholder_type)){ | ||
| item = PyTuple_GET_ITEM(nargs, j); | ||
| j++; | ||
| pnp--; | ||
| } | ||
| } else { | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| item = PyTuple_GET_ITEM(nargs, j); | ||
| j++; | ||
| } | ||
| Py_INCREF(item); | ||
| PyTuple_SET_ITEM(aargs, i, item); | ||
| } | ||
| pto->args = aargs; | ||
| pto->np = pnp + nnp; | ||
| Py_DECREF(nargs); | ||
| } else if (pargs == NULL) { | ||
| pto->args = nargs; | ||
| pto->np = nnp; | ||
| } else { | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| pto->args = PySequence_Concat(pargs, nargs); | ||
| pto->np = pnp + nnp; | ||
| Py_DECREF(nargs); | ||
| if (pto->args == NULL) { | ||
| Py_DECREF(pto); | ||
| @@ -217,13 +283,19 @@ partial_vectorcall(partialobject *pto, PyObject *const *args, | ||
| size_t nargsf, PyObject *kwnames) | ||
| { | ||
| PyThreadState *tstate = _PyThreadState_GET(); | ||
| Py_ssize_t nargs = PyVectorcall_NARGS(nargsf); | ||
| /* pto->kw is mutable, so need to check every time */ | ||
| if (PyDict_GET_SIZE(pto->kw)) { | ||
| return partial_vectorcall_fallback(tstate, pto, args, nargsf, kwnames); | ||
| } | ||
| Py_ssize_t np = pto->np; | ||
| if (nargs < np) { | ||
| PyErr_SetString(PyExc_TypeError, | ||
| "unfilled placeholders in 'partial' call"); | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| return NULL; | ||
| } | ||
| Py_ssize_t nargs_total = nargs; | ||
| if (kwnames != NULL) { | ||
| nargs_total += PyTuple_GET_SIZE(kwnames); | ||
| @@ -251,6 +323,7 @@ partial_vectorcall(partialobject *pto, PyObject *const *args, | ||
| } | ||
| Py_ssize_t newnargs_total = pto_nargs + nargs_total; | ||
| newnargs_total = newnargs_total - np; | ||
| PyObject *small_stack[_PY_FASTCALL_SMALL_STACK]; | ||
| PyObject *ret; | ||
| @@ -267,12 +340,28 @@ partial_vectorcall(partialobject *pto, PyObject *const *args, | ||
| } | ||
| } | ||
| Py_ssize_t nargs_new; | ||
| if (np) { | ||
| nargs_new = pto_nargs + nargs - np; | ||
| Py_ssize_t j = 0; // Placeholder counter | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| for (Py_ssize_t i=0; i < pto_nargs; i++) { | ||
| if (Py_Is(pto_args[i], (PyObject *) &placeholder_type)){ | ||
| memcpy(stack + i, args + j, 1 * sizeof(PyObject*)); | ||
| j += 1; | ||
| } else { | ||
| memcpy(stack + i, pto_args + i, 1 * sizeof(PyObject*)); | ||
| } | ||
| } | ||
| if (nargs_total > np){ | ||
| memcpy(stack + pto_nargs, args + np, (nargs_total - np) * sizeof(PyObject*)); | ||
| } | ||
| } else { | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| nargs_new = pto_nargs + nargs; | ||
| /* Copy to new stack, using borrowed references */ | ||
| memcpy(stack, pto_args, pto_nargs * sizeof(PyObject*)); | ||
| memcpy(stack + pto_nargs, args, nargs_total * sizeof(PyObject*)); | ||
| } | ||
| ret = _PyObject_VectorcallTstate(tstate, pto->fn, stack, nargs_new, kwnames); | ||
| if (stack != small_stack) { | ||
| PyMem_Free(stack); | ||
| } | ||
| @@ -304,6 +393,14 @@ partial_call(partialobject *pto, PyObject *args, PyObject *kwargs) | ||
| assert(PyTuple_Check(pto->args)); | ||
| assert(PyDict_Check(pto->kw)); | ||
| Py_ssize_t nargs = PyTuple_GET_SIZE(args); | ||
| Py_ssize_t np = pto->np; | ||
| if (nargs < np) { | ||
| PyErr_SetString(PyExc_TypeError, | ||
| "unfilled placeholders in 'partial' call"); | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| return NULL; | ||
| } | ||
| /* Merge keywords */ | ||
| PyObject *kwargs2; | ||
| if (PyDict_GET_SIZE(pto->kw) == 0) { | ||
| @@ -328,8 +425,33 @@ partial_call(partialobject *pto, PyObject *args, PyObject *kwargs) | ||
| } | ||
| /* Merge positional arguments */ | ||
| PyObject *args2; | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| if (np) { | ||
| Py_ssize_t pto_nargs = PyTuple_GET_SIZE(pto->args); | ||
| Py_ssize_t nargs_new = pto_nargs + nargs - np; | ||
| args2 = PyTuple_New(nargs_new); | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| PyObject *pto_args = pto->args; | ||
| PyObject *item; | ||
| Py_ssize_t j = 0; // Placeholder counter | ||
| for (Py_ssize_t i=0; i < pto_nargs; i++) { | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| item = PyTuple_GET_ITEM(pto_args, i); | ||
| if (Py_Is(item, (PyObject *) &placeholder_type)){ | ||
| item = PyTuple_GET_ITEM(args, j); | ||
| j += 1; | ||
| } | ||
| PyTuple_SET_ITEM(args2, i, item); | ||
| } | ||
| if (nargs > np){ | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| for (Py_ssize_t i=pto_nargs; i < nargs_new; i++) { | ||
| item = PyTuple_GET_ITEM(args, j); | ||
| PyTuple_SET_ITEM(args2, i, item); | ||
| j += 1; | ||
| } | ||
| } | ||
| } else { | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| /* Note: tupleconcat() is optimized for empty tuples */ | ||
| args2 = PySequence_Concat(pto->args, args); | ||
| } | ||
| if (args2 == NULL) { | ||
| Py_XDECREF(kwargs2); | ||
| return NULL; | ||
| @@ -354,6 +476,8 @@ static PyMemberDef partial_memberlist[] = { | ||
| "tuple of arguments to future partial calls"}, | ||
| {"keywords", _Py_T_OBJECT, OFF(kw), Py_READONLY, | ||
| "dictionary of keyword arguments to future partial calls"}, | ||
| {"np", Py_T_PYSSIZET, OFF(np), Py_READONLY, | ||
dg-pb marked this conversation as resolved. OutdatedShow resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| "number of placeholders"}, | ||
| {"__weaklistoffset__", Py_T_PYSSIZET, | ||
| offsetof(partialobject, weakreflist), Py_READONLY}, | ||
| {"__dictoffset__", Py_T_PYSSIZET, | ||
| @@ -1497,6 +1621,9 @@ _functools_exec(PyObject *module) | ||
| if (PyModule_AddType(module, state->partial_type) < 0) { | ||
| return -1; | ||
| } | ||
| if (PyModule_AddType(module, &placeholder_type) < 0) { | ||
| return -1; | ||
| } | ||
| PyObject *lru_cache_type = PyType_FromModuleAndSpec(module, | ||
| &lru_cache_type_spec, NULL); | ||
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.