Movatterモバイル変換


[0]ホーム

URL:


homepage

Issue25430

This issue trackerhas been migrated toGitHub, and is currentlyread-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title:speed up ipaddress __contain__ method
Type:performanceStage:resolved
Components:Library (Lib)Versions:Python 3.8
process
Status:closedResolution:fixed
Dependencies:Superseder:
Assigned To:Nosy List: gescheit, methane, ncoghlan, pitrou, pmoody, serhiy.storchaka, xtreak
Priority:normalKeywords:patch

Created on2015-10-17 11:57 bygescheit, last changed2022-04-11 14:58 byadmin. This issue is nowclosed.

Files
File nameUploadedDescriptionEdit
ipaddress_contains.patchgescheit,2015-10-17 11:57review
Pull Requests
URLStatusLinkedEdit
PR 1785mergedpython-dev,2017-05-24 10:56
Messages (8)
msg253126 -(view)Author: Aleksandr Balezin (gescheit)*Date: 2015-10-17 11:57
Current check "address in network" is seems a bit odd:int(self.network_address) <= int(other._ip) < int(self.broadcast_address)This patch make this in bit-operation manner. Perfomace test:import ipaddressimport timeitclass IPv6Network2(ipaddress.IPv6Network):    def __contains__(self, other):        # always false if one is v4 and the other is v6.        if self._version != other._version:            return False        # dealing with another network.        if isinstance(other, ipaddress._BaseNetwork):            return False        else:            # address            return other._ip & self.netmask._ip == self.network_address._ipclass IPv4Network2(ipaddress.IPv4Network):    def __contains__(self, other):        # always false if one is v4 and the other is v6.        if self._version != other._version:            return False        # dealing with another network.        if isinstance(other, ipaddress._BaseNetwork):            return False        # dealing with another address        else:            # address            return other._ip & self.netmask._ip == self.network_address._ipipv6_test_net = ipaddress.IPv6Network("::/0")ipv6_test_net2 = IPv6Network2("::/0")ipv4_test_net = ipaddress.IPv4Network("0.0.0.0/0")ipv4_test_net2 = IPv4Network2("0.0.0.0/0")dataipv6 = list()dataipv4 = list()for x in range(2000000):    dataipv6.append(ipaddress.IPv6Address(x))    dataipv4.append(ipaddress.IPv4Address(x))def test():    for d in dataipv6:        d in ipv6_test_netdef test2():    for d in dataipv6:        d in ipv6_test_net2def test3():    for d in dataipv4:        d in ipv4_test_netdef test4():    for d in dataipv4:        d in ipv4_test_net2t = timeit.Timer("test()", "from __main__ import test")print("ipv6 test origin __contains__", t.timeit(number=1))t = timeit.Timer("test2()", "from __main__ import test2")print("ipv6 test new __contains__", t.timeit(number=1))t = timeit.Timer("test3()", "from __main__ import test3")print("ipv4 test origin __contains__", t.timeit(number=1))t = timeit.Timer("test4()", "from __main__ import test4")print("ipv4 test new __contains__", t.timeit(number=1))Output:ipv6 test origin __contains__ 4.265904285013676ipv6 test new __contains__ 1.551749340025708ipv4 test origin __contains__ 3.689626455074176ipv4 test new __contains__ 2.0175559649942443
msg294242 -(view)Author: Aleksandr Balezin (gescheit)*Date: 2017-05-23 09:46
I think this patch can be easily applied. There are no any breaking changes. My colleagues and I assume that ipaddress module would work fast because it is easy to imagine tasks, operate with ton of networks.Also, current comparison implementation works as not so good pattern for other developers.
msg294252 -(view)Author: Antoine Pitrou (pitrou)*(Python committer)Date: 2017-05-23 14:58
Hi Aleksandr,well, sorry for the detail.  We now use GitHub for patch submission, would you like to submit a PR athttps://github.com/python/cpython/ (seedevguide athttps://cpython-devguide.readthedocs.io/ for more information).If you don't want to do so, that's fine, too. A core developer may take care of it for you.
msg294253 -(view)Author: Antoine Pitrou (pitrou)*(Python committer)Date: 2017-05-23 14:58
s/sorry for the detail/sorry for the delay/
msg294442 -(view)Author: Serhiy Storchaka (serhiy.storchaka)*(Python committer)Date: 2017-05-25 07:20
I confirm that the proposed code is equivalent to the existing code and is faster. The largest part (~70%) of the speed up is caused by replacing int() calls with direct attribute access to _ip. I'm wondering whether it is worth to get rid of int() calls in the rest of the code.The overhead of using int():* Looking up the "int" name in globals (failed).* Looking up the "int" name in builtins.* Calling the C function.* Calling the Python function.
msg331931 -(view)Author: Karthikeyan Singaravelan (xtreak)*(Python committer)Date: 2018-12-16 18:00
Though this is out of the scope of the issue I tried converting num_addresses, __hash__, __getitem__ and __eq__ as per Serhiy's idea for IPv6Network replacing the stdlib implementation's int calls with _ip in a custom class. I can see up to 50% speedups as below and no test case failures converting rest of the call sites to use _ip instead of int in stdlib. But some of the methods may not be used as frequently as in this benchmark like thus being not worthy enough of change. Shall I open a new issue for further discussion? $ python3.7 bpo25430_1.pyipv6 test num_addresses with int 1.54065761ipv6 test num_addresses without int 0.8266360819999998ipv6 test hash with int 1.320016881ipv6 test hash without int 0.6266323200000001ipv6 test equality with int 1.6104001990000008ipv6 test equality without int 1.0374885390000008ipv6 test get item with int 2.092343390000001ipv6 test get item without int 1.5606673410000003$ bpo25430_1.pyimport ipaddressimport timeitclass IPv6Network2(ipaddress.IPv6Network):    @property    def num_addresses(self):        return self.broadcast_address._ip - self.network_address._ip + 1    def __hash__(self):        return hash(self.network_address._ip ^ self.netmask._ip)    def __eq__(self, other):        try:            return (self._version == other._version and                    self.network_address == other.network_address and                    self.netmask._ip == other.netmask._ip)        except AttributeError:            return NotImplemented    def __getitem__(self, n):        network = self.network_address._ip        broadcast = self.broadcast_address._ip        if n >= 0:            if network + n > broadcast:                raise IndexError('address out of range')            return self._address_class(network + n)        else:            n += 1            if broadcast + n < network:                raise IndexError('address out of range')            return self._address_class(broadcast + n)ipv6_test_net = ipaddress.IPv6Network("::/0")ipv6_test_net2 = IPv6Network2("::/0")def test1_num_address():    return ipv6_test_net.num_addressesdef test2_num_address():    return ipv6_test_net2.num_addressesdef test1_hash_address():    return hash(ipv6_test_net)def test2_hash_address():    return hash(ipv6_test_net2)if __name__ == "__main__":    t = timeit.Timer("test1_num_address()", "from __main__ import test1_num_address")    print("ipv6 test num_addresses with int", t.timeit(number=1000000))    t = timeit.Timer("test2_num_address()", "from __main__ import test2_num_address")    print("ipv6 test num_addresses without int", t.timeit(number=1000000))    t = timeit.Timer("test1_hash_address()", "from __main__ import test1_hash_address")    print("ipv6 test hash with int", t.timeit(number=1000000))    t = timeit.Timer("test2_hash_address()", "from __main__ import test2_hash_address")    print("ipv6 test hash without int", t.timeit(number=1000000))    t = timeit.Timer("ipv6_test_net == ipv6_test_net", "from __main__ import ipv6_test_net")    print("ipv6 test equality with int", t.timeit(number=1000000))    t = timeit.Timer("ipv6_test_net2 == ipv6_test_net2", "from __main__ import ipv6_test_net2")    print("ipv6 test equality without int", t.timeit(number=1000000))    t = timeit.Timer("ipv6_test_net[10000]", "from __main__ import ipv6_test_net")    print("ipv6 test get item with int", t.timeit(number=1000000))    t = timeit.Timer("ipv6_test_net2[10000]", "from __main__ import ipv6_test_net2")    print("ipv6 test get item without int", t.timeit(number=1000000))    assert test1_num_address() == test2_num_address()    assert hash(ipv6_test_net2) == hash(ipv6_test_net)    assert ipv6_test_net2 == ipv6_test_net    assert ipv6_test_net[10000] == ipv6_test_net2[10000]
msg341138 -(view)Author: Inada Naoki (methane)*(Python committer)Date: 2019-04-30 07:54
New changeset3bbcc92577f8e616bc94c679040043bacd00ebf1 by Inada Naoki (gescheit) in branch 'master':bpo-25430: improve performance of IPNetwork.__contains__ (GH-1785)https://github.com/python/cpython/commit/3bbcc92577f8e616bc94c679040043bacd00ebf1
msg355303 -(view)Author: Karthikeyan Singaravelan (xtreak)*(Python committer)Date: 2019-10-24 08:45
Closing it as resolved since the optimization was merged in 3.8.0 . Thanks @gescheit for the report and patch.
History
DateUserActionArgs
2022-04-11 14:58:22adminsetgithub: 69616
2019-10-24 08:45:46xtreaksetstatus: open -> closed
resolution: fixed
messages: +msg355303

stage: patch review -> resolved
2019-04-30 07:54:50methanesetnosy: +methane
messages: +msg341138
2018-12-16 18:00:02xtreaksetnosy: +xtreak
messages: +msg331931
2018-12-12 20:54:14pitrousetversions: + Python 3.8, - Python 3.7
2017-08-03 06:02:06serhiy.storchakasetversions: + Python 3.7, - Python 3.6
2017-05-25 07:20:02serhiy.storchakasetnosy: +serhiy.storchaka
messages: +msg294442
2017-05-24 10:56:22python-devsetpull_requests: +pull_request1866
2017-05-23 14:58:31pitrousetmessages: +msg294253
2017-05-23 14:58:18pitrousetnosy: +pitrou
messages: +msg294252
2017-05-23 09:46:50gescheitsetmessages: +msg294242
2016-07-26 19:00:29berker.peksagunlinkissue25431 dependencies
2015-10-17 18:00:45serhiy.storchakasetnosy: +ncoghlan,pmoody
stage: patch review

versions: - Python 3.4, Python 3.5
2015-10-17 18:00:13serhiy.storchakalinkissue25431 dependencies
2015-10-17 16:44:35gescheitsetversions: + Python 3.4, Python 3.6
2015-10-17 11:57:18gescheitcreate
Supported byThe Python Software Foundation,
Powered byRoundup
Copyright © 1990-2022,Python Software Foundation
Legal Statements

[8]ページ先頭

©2009-2026 Movatter.jp