- Notifications
You must be signed in to change notification settings - Fork11
Open
Assignees
Description
Hi Cloudflare,
first please let me thank you for your code. It is really amazing.
We at Showmax are using it and have found an issue (a corner case, if you wish) which results in some prefixes not being stored into the generated file. The issue can be demonstrated on the following piece of code:
#!/usr/bin/env python3"""Geoexceptions file generator."""import argparseimport loggingimport logging.handlersimport sysimport geoip2.databaseimport mmdbencoderfrom geoip2.errors import AddressNotFoundErrordef exception_data(country_code): """ Format the data as needed by the MaxMind Country database format. Do not modify unless really sure what are you doing! :param country_code: ISO 2-character country code, e.g. ZA :return: MaxMind Country DB structure needed for an IP network record. """ return { 'continent': {}, 'registered_country': {}, 'represented_country': {}, 'traits': {}, 'country': { 'iso_code': country_code, }, }def tryip(reader, ip): """Try to retrieve country code from the MaxMind database.""" try: return reader.country(ip).country.iso_code except AddressNotFoundError: # Address is not present in the database return Nonedef main(): """Main.""" logging.basicConfig(stream=sys.stdout) logging.getLogger() logging.getLogger().setLevel(logging.DEBUG) logging.debug('Test case #1') test_pfxs = ['2001:db8:807:1::1', '2001:db8:807:102::1', '2001:db8:807:111::1'] fname = 'test.mmdb' cc_za = exception_data('ZA') cc_ke = exception_data('KE') enc = mmdbencoder.Encoder( 6, # IP version 32, # Size of the pointers 'Country', # Name of the table ['en'], # Languages {'en': 'Geoexceptions Country Database'}, # Description compat=True) data_za = enc.insert_data(cc_za) data_ke = enc.insert_data(cc_ke) enc.insert_network('2001:db8:807::/48', data_za, False) enc.insert_network('2001:db8:807:102::/64', data_ke, False) enc.write_file(fname) reader = geoip2.database.Reader(fname) for pfx in test_pfxs: logging.debug('prefix %s, present? %s', pfx, tryip(reader, pfx)) logging.debug('Test case #2') fname = 'test.mmdb' cc_za = exception_data('ZA') cc_ke = exception_data('KE') enc = mmdbencoder.Encoder( 6, # IP version 32, # Size of the pointers 'Country', # Name of the table ['en'], # Languages {'en': 'Geoexceptions Country Database'}, # Description compat=True) data_za = enc.insert_data(cc_za) data_ke = enc.insert_data(cc_ke) enc.insert_network('2001:db8:807:102::/64', data_ke, False) enc.insert_network('2001:db8:807::/48', data_za, False) enc.write_file(fname) reader = geoip2.database.Reader(fname) for pfx in test_pfxs: logging.debug('prefix %s, present? %s', pfx, tryip(reader, pfx)) logging.debug('Tests completed.')if __name__ == '__main__': main()The resulting output is:
$ python test_subnet.pyDEBUG:root:Test case #1DEBUG:root:prefix 2001:db8:807:1::1, present? ZADEBUG:root:prefix 2001:db8:807:102::1, present? KEDEBUG:root:prefix 2001:db8:807:111::1, present? ZADEBUG:root:Test case #2DEBUG:root:prefix 2001:db8:807:1::1, present? ZADEBUG:root:prefix 2001:db8:807:102::1, present? KEDEBUG:root:prefix 2001:db8:807:111::1, present? NoneDEBUG:root:Tests completed.If the more-specific prefix is inserted first, the less-specific data for2001:db8:807:100::/63,2001:db8:807:103::..2001:db8:807:1ff:ffff:ffff:ffff:ffff are missing from the resulting file.
I believe this should not be happening.
(If strict mode is disabled, both cases end up with errorException: Encoder: add_to_trie: try setting data on a non-final: (...128bit-int-number) already has child. Not updating in strict mode.)
Metadata
Metadata
Assignees
Labels
No labels