
This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.
Created on2016-01-05 07:39 byanilredshift, last changed2022-04-11 14:58 byadmin. This issue is nowclosed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| b.py | anilredshift,2016-01-05 07:39 | demonstration of bug | ||
| unpickle_broken_import.patch | serhiy.storchaka,2016-01-05 09:18 | review | ||
| Messages (9) | |||
|---|---|---|---|
| msg257524 -(view) | Author: Anil Kulkarni (anilredshift) | Date: 2016-01-05 07:39 | |
Pickles created with python 3.4.X will not load with python 3.5.X if they include a collections.OrderedDictTo reproduce this issue, simply create a pickle of an OrderedDict on python 3.4.3 with protocol=2 and try to open it on 3.5. I have included a simple script to demonstrate this issue.I believe this is related to this bug:https://bugs.python.org/issue18473As to the real-world implications: The python package Celery uses protocol=2 by default when serializing with pickle, and thus a celery web running 3.5 cannot receive the results of a worker running 3.4For celery specifically, there is a workaround by setting the PICKLE_PROTOCOL environment variable, but this is a core python issue.P.S. This is the first bug I've filed with python so please let me know if there's something else I should be including.Thanks! | |||
| msg257526 -(view) | Author: Serhiy Storchaka (serhiy.storchaka)*![]() | Date: 2016-01-05 09:18 | |
Thank you for your report Anil.Python 3.4.3 creates incorrect pickle with protocol 2 (it can't be load in Python 2).issue18473 fixed pickling with protocol 2 in Python 3, but broke loading broken pickles created in unpatched versions of Python 3.Here is a patch that allows to load such broken pickles.But it is too late to fix Python 3.4, it can get only security fixes. You have Celery specific workaround. More general workaround is to update _compact_pickle.IMPORT_MAPPING, e.g.:import _compact_pickle_compact_pickle.IMPORT_MAPPING.update({ 'UserDict': 'collections', 'UserList': 'collections', 'UserString': 'collections', 'whichdb': 'dbm', 'StringIO': 'io', 'cStringIO': 'io',})Note that you have to set mapping not just for one module name like 'UserList', but for all 'UserDict', 'UserList', and 'UserString', because it is not predicable to what module name 'collections' is mapped in Python 3.4.3. | |||
| msg257541 -(view) | Author: Anil Kulkarni (anilredshift) | Date: 2016-01-05 19:53 | |
Hi Serhiy,I have done some more investigation this morning and I have come across two distinct issues.The first is that pickles (specifically the pickle in my earlier message) created in python 3.0->3.4.3 do not load on python 2.7.10. The exception is 3.5.1, which produces a pickle that DOES load on 2.7.10This is unfortunate but not a regressionThe second issue is that pickles created by python 3.0->3.4.3 do NOT load in python 3.5This is a regression, in the sense that the pickle created by 3.0->3.4.3 is compatible with every other version of python (I haven't tested every single combination here, but several)From the language perspective, it may be that 3.4.X has incorrect code, but from a compatibility perspective, 3.5 is breaking the promise that pickles are compatible across versions of python.I write this as justification that python 3.5 should fix this regression in compatibility with 3.0 -> 3.4.3. At the very least 3.5 should have a shim behavior to fallback and allow it to import these pickles correctly.Thanks,Anil | |||
| msg257557 -(view) | Author: Serhiy Storchaka (serhiy.storchaka)*![]() | Date: 2016-01-05 21:23 | |
I agree, and the provided is purposed to fix this issue. In the previous message I just proposed a workaround for those who can't wait 3.5.2. | |||
| msg257558 -(view) | Author: Anil Kulkarni (anilredshift) | Date: 2016-01-05 21:57 | |
Ah, sorry I misunderstood. Thanks for the quick turnaround! | |||
| msg258540 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-01-18 19:36 | |
New changeset270add17f203 by Serhiy Storchaka in branch '3.5':Issue#26013: Added compatibility with broken protocol 2 pickles createdhttps://hg.python.org/cpython/rev/270add17f203New changeset35ff0976b211 by Serhiy Storchaka in branch 'default':Issue#26013: Added compatibility with broken protocol 2 pickles createdhttps://hg.python.org/cpython/rev/35ff0976b211 | |||
| msg258544 -(view) | Author: STINNER Victor (vstinner)*![]() | Date: 2016-01-18 20:21 | |
Buildbots are unhappy.http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/344/steps/test/logs/stdio======================================================================FAIL: test_reverse_import_mapping (test.test_pickle.CompatPickleTests) [('cStringIO', 'io')]----------------------------------------------------------------------Traceback (most recent call last): File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_pickle.py", line 353, in test_reverse_import_mapping (module3, module2))AssertionError: No reverse mapping from 'io' to 'cStringIO' | |||
| msg258545 -(view) | Author: Roundup Robot (python-dev)![]() | Date: 2016-01-18 20:34 | |
New changeset7215d13baa2d by Serhiy Storchaka in branch '3.5':Added exceptins for testing non-reversible import mapping for Issue#26013.https://hg.python.org/cpython/rev/7215d13baa2dNew changeset16cfc1652844 by Serhiy Storchaka in branch 'default':Added exceptins for testing non-reversible import mapping for Issue#26013.https://hg.python.org/cpython/rev/16cfc1652844 | |||
| msg258546 -(view) | Author: Serhiy Storchaka (serhiy.storchaka)*![]() | Date: 2016-01-18 20:46 | |
Thank you Victor. | |||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:25 | admin | set | github: 70201 |
| 2016-01-19 10:02:59 | serhiy.storchaka | set | status: open -> closed resolution: fixed |
| 2016-01-18 20:46:05 | serhiy.storchaka | set | messages: +msg258546 |
| 2016-01-18 20:34:43 | python-dev | set | messages: +msg258545 |
| 2016-01-18 20:21:54 | vstinner | set | status: closed -> open nosy: +vstinner messages: +msg258544 resolution: fixed -> (no value) |
| 2016-01-18 19:41:25 | serhiy.storchaka | set | status: open -> closed resolution: fixed stage: patch review -> resolved |
| 2016-01-18 19:36:33 | python-dev | set | nosy: +python-dev messages: +msg258540 |
| 2016-01-05 21:57:42 | anilredshift | set | messages: +msg257558 |
| 2016-01-05 21:23:59 | serhiy.storchaka | set | messages: +msg257557 |
| 2016-01-05 19:53:28 | anilredshift | set | messages: +msg257541 |
| 2016-01-05 09:18:35 | serhiy.storchaka | set | files: +unpickle_broken_import.patch assignee:serhiy.storchaka components: + Library (Lib) versions: + Python 3.6 keywords: +patch nosy: +alexandre.vassalotti,serhiy.storchaka,pitrou messages: +msg257526 stage: patch review |
| 2016-01-05 07:39:44 | anilredshift | create | |