sign (sign)	original words of abbreviations	Administrative region meanings
			CO	country	State of the country
PR	province	Province of labor
			CI	city	city
AR	area	District of jurisdiction
			ST	street	Street with a light source
RO	road	road
			BU	building	construction of buildings
OT	other	Redundant information

as shown in fig. 2, taking the address information "mr. jun:

[ ('Tianjin', 'CI'), ('New harbor second road No. 18', 'RO'), ('coastal new zone', 'AR'), ('Limon', 'OT'), ('metabolic', 'OT'), ('Cooperation', 'OT') ]

Obviously, the binary array obtained as described above has several problems:

1. part of the geographical entity is missing. For example, street information is lacked between a new coastal area and a new harbor second road.

2. there is much redundant information. It should be noted that if the same geographic information appears in the address many times, only one is retained, and the rest of the duplicated geographic information should be attributed to redundant information.

In order to solve the above 2 problems, we edit the national administration level geographic information into a tree dictionary by using each administration level and each geographic entity of the level as a node according to the order of administration levels.

And sequencing the binary array predicted by the model to remove duplication, eliminating redundancy, and sequencing according to the administrative division level to obtain a new binary array, namely a standard address. Specifically, the classification coding is performed with reference to the administrative level standard CO > PR > CI > AR > ST > RO > BU, and the classification coding is arranged in ascending order of coding, and information of no corresponding administrative division level and repeated geographic information are eliminated as redundant information. After sorting and de-duplication as described above for the binary array, the following array is obtained as shown in FIG. 2:

[ ('Tianjin', 'CI'), ('coastal New zone', 'AR'), ('New Port No. 18', 'RO') ]

And then matching the sorted and de-duplicated binary array with the tree dictionary to determine whether geographic information is missing in the binary array. In particular, a recursive method can be adopted for defect-finding and completion. Such as the lack of street staffing between a new coastal region and a new harbour second road in the binary array described above.

And if the geographic information is missing, completing the geographic information of the binary array according to the tree dictionary. Geographic data is then obtained in standard form, as shown in fig. 2:

[ ('Tianjin', 'CI'), ('coastal New zone', 'AR'), ('Stadium street', 'ST'), ('New Port No. two way No. 18', 'RO') ]

After the geographic data in the standard form is obtained, the geographic data can be encoded by adopting the encoding technology to obtain a geographic encoding result.

The application mentioned above provides an administrative geographic entity relationship recognition model optimized by a feature selection algorithm, and the construction training process of the model is described as follows:

firstly, the natural language processing technology (NLP) is utilized to carry out feature extraction and selection on sample address information, and a sample feature vector is obtained through calculation. The method comprises the following specific steps:

1. a sample set of address information corpuses is constructed, which may be obtained from the original data system in fig. 1. In order to further improve the accuracy, the method and the device can divide the original address information corpus obtained from the original data system into data which cannot obtain coordinate codes by a coordinate analysis program, obtain data with incorrect coordinates, and obtain data with correct coordinates. Then each classified equal part is screened from the original address information corpus to be used as a basic corpus. And then, performing word segmentation on the screened linguistic data and marking a sample geographic entity of each word segmentation and an administrative division (administrative geographic identification) corresponding to the sample geographic entity. And randomly selecting a certain proportion of labeled data to perform model training, and reserving a certain proportion of labeled data to perform model verification.

2. Feature extraction and selection:

2.1 extracting the characteristics of the labeled address data used for model training, and then recalculating the extracted characteristics into characteristic frequencies FC, N for each geographic administrative division level_ikrepresenting the number of times a feature appears in the address information text, as in equation (1), N_iindicating the total number of features present in the address information.

2.2, calculating the relevance between each feature pw and each geographic administrative division level t, and obtaining a feature weight W as formula (2):

wherein, EX_ikThe number of texts present for the feature pw in the other levels than the geographic administrative division level t; UN (UN)_ikthe number of texts is the number of texts of which the features pw do not appear in the geographic administrative division level t; and S is the sum of the text numbers of the geographic entities in all the administrative entity classes.

2.3Calculating a weighted average W_avgand characteristic frequency average value FC_avgFN in (3) and (4) represents the total number of feature types, when the weight of the feature satisfies W>W_avgOr (W)<W_avgAnd FC>FC_avg) I.e. the selected target feature

3. Calculating a sample feature vector of the target feature:

3.1 there are x geographic administrative division levels, then each selected target feature will get x correlation degrees, and take the average of these x correlation degrees as the weight of each word. Obtaining a weighting matrix A according to the feature weight_rc:

A_rc＝(W_ija_ij)_r*c (5)

3.2 feature vector calculation, let Y ∈ R^n*nwith n independent eigenvectors, principal eigenvalues m₁satisfy | m₁|>|m₂|≥...≥|m_nif the vector is any administrative geographic entity feature vector v₀＝c₀Vector sequence { c) constructed as follows_k},{v_k}:

then there are:

the weighted normalized sample feature vector obtained by constructing the expression (2), (5), (6), (7) and (8) is expressed by the expression (9):

and then, taking the obtained sample feature vector v as vectorization input parameters of model training, and training oppositely quantized training corpora through a neural network and a conditional random field algorithm, such as an RNN (recurrent neural network) and a CRF (conditional random field) algorithm, so as to obtain an administrative geographic entity relationship recognition model. The final output of the model is a binary geographic entity relationship set as follows:

political relation＝[(e₁,t₁),(e₂,t₂),...(e_n,t_n)]

in the construction of the model, the selected target features have high correlation with the administrative division level, some messy features with low correlation with the administrative division level are abandoned, the adverse effect of the messy features on the result is reduced, and the data volume of the model input is reduced. The characteristic selection is utilized to carry out algorithm optimization, so that the parameters of the input model are not disordered address information, but are selected and optimized characteristic vectors, and the correlation degree of the input parameters with the geographic entities and the corresponding administrative divisions is improved, so that the calculation speed of the model is accelerated, and the accuracy of the identification result is improved.

The method comprises the steps of reading all standard geographic information and address rules into a memory to construct a dictionary tree, taking a server as an example, wherein all the regular dictionary trees need 4GB memory, replacing all the geographic information regular dictionary trees with administrative region geographic entity recognition models by using the scheme of the application, wherein the models only need 200MB of memory space, compared with the prior art, the method only needs 4.88% of the prior art when used for memory parties, and the use cost is reduced.

In addition, compared with the prior art, the method solves the problem of low geographic data quality, increases the effective analysis amount of address analysis, and provides more accurate data basis for upper-layer decision making:

The address analysis technical scheme combining the standard geographic dictionary database with the regular extraction has more limitations in processing address data, and for dirty data scenes with more address information due to human factors, correct geographic information cannot be obtained basically by using the common technical scheme. Here, the evaluation index is defined in conjunction with the address resolution scenario: accuracy, resolution, effective resolution.

as follows, R represents a record set in which the address resolution has obtained correct coordinates, G (wr)_iThe method comprises the steps of representing a certain type of analysis error result set i, wherein the main error type is that analysis coordinates have deviation, T represents the total number of addresses needing to be analyzed, S represents a record set that the addresses are successfully analyzed to obtain the coordinates, E represents a failure record set that the coordinates are not obtained after the addresses are analyzed, the accuracy of final address analysis is as shown in the formula (10), the resolution is as shown in the formula (11), and the effective resolution is as shown in the formula (12).

and analyzing a correct result set, wherein R analyzes an error result set:

total number of samples: and (3) a T-analysis success result set, namely an S-E analysis failure result set: e

10000 pieces of address data test results are used for comparative evaluation, the analysis accuracy based on the dictionary and the regular matching technology is 86.41%, wherein 13.59% of the incorrect analysis results are caused by data quality problems such as redundant information and word sequence disorder existing in address information, and meanwhile, the data quality problems also cause that part of data analysis fails to obtain coordinates, and the analysis rate using the technology is only 81%. Under the same condition, the resolution rate of the scheme reaches 98%, is improved by 17% compared with the prior art, and the effective resolution rate is improved from 70% to 93%, as shown in table 2.

TABLE 2 technical index boost

and the administrative geographic entity relationship recognition model is optimized by utilizing a feature selection algorithm, the accuracy of extracting geographic information is higher than that of traditional rule matching, and the extracted geographic data is more accurate.

the following is a specific implementation of the first embodiment of the present application:

And constructing a bottom layer data synchronization task, and storing the originally input address information in the original data system into the HDFS of the analysis task cluster. The analytic task cluster is based on spark technology, data processing tasks are developed by java, and task scheduling distribution is achieved. And deploying a pre-trained administrative geographic entity relationship recognition model in the analysis task cluster, recognizing administrative division levels and geographic entity relationships of low-quality address information, and extracting effective information. The method comprises the steps of performing model training based on an RNN (neural network) and a CRF (conditional random field) algorithm, embedding an administrative geographic entity feature optimization algorithm, and reducing noise of artificial interference information. And then, sequencing and recombining the administrative geographic entities by adopting an administrative hierarchical sequencing algorithm, and checking and mending the data by using the constructed tree dictionary to obtain standard geographic data and provide quality-improved address information for subsequent coding.

and the geocoding function can perform concurrent scheduling on the spark task cluster, and adopts a java developed http analysis address batch analysis interface based on RESTful style to perform coding analysis on address information completed after the model is extracted, so as to obtain standard geocoding information. In order to improve the analysis efficiency, the data can be analyzed and coded in batches by using a single batch submission mode while task concurrent scheduling is adopted, and the analysis and coding throughput is improved under the condition that the cluster pressure is not increased.

Because the independent batch coding analysis service is adopted, resources cannot be occupied with extraction and calculation, the analysis time is obviously shortened, the original 1 million of data needs to be analyzed in 15 days after the administrative geographic entity relationship model is embedded into the spark calculation engine, and the speed is improved by 36 times after the patent scheme is adopted and only 10 hours are needed.

example two

Based on the above description, a second embodiment of the present application provides an address information parsing method, as shown in fig. 3, where the method includes:

s31, acquiring address information to be analyzed in the original data;

s32, extracting and selecting the characteristics of the address information to be analyzed by using a natural language processing technology, and vectorizing the selected characteristics to obtain a characteristic vector to be identified; the specific mode can refer to the steps of feature extraction selection and vectorization in model training.

s33, inputting the feature vector to be recognized into a preset model to obtain an initial array comprising a geographic entity and an administrative division level corresponding to the geographic entity;

S34, sorting and de-duplicating the geographic entities in the initial array according to the administrative division level to obtain a standard array;

S35, the standard array is coded to obtain a geocoding result. Specifically, an encoding interface of an external server may be called to encode the standard array to obtain a geocoding result.

If the address information is not analyzed, extracting the characteristics of the address information to be analyzed by using a natural language processing technology.

To avoid incomplete information in the array, before encoding the standard array to obtain the geocoding result, the method further includes:

The method of the present application further comprises the step of pre-constructing the preset model:

The more specific steps of constructing the preset model in advance can be referred to the process of training the model.

the geocoding result can be combined with other data to provide a data basis for subsequent application decision, and for this reason, the geocoding result and the original data corresponding to the result can be stored in an associated manner in the application.

Taking the original data as the sales data as an example, after the address information of the original data is analyzed to obtain an accurate geocoding result, the geocoding result and the corresponding original data can be stored in a correlation manner, and then the commodity sales condition of a certain geographic position can be obtained. This associated information may be stored in the elasticsearch engine for ease of subsequent retrieval.

EXAMPLE III

Based on the above-mentioned association storage, taking an example of requesting to obtain related data in a certain region range, a third embodiment of the present application provides a data obtaining method, including:

Receiving candidate address information;

analyzing the candidate address information according to the address analysis method to obtain analyzed candidate geocoded data;

By the method, the original data in a certain geographic range can be obtained by utilizing the geocoding result, and a data basis is provided for subsequent decisions such as sale, popularization and the like.

Example four

Corresponding to the method of the second embodiment, a fourth embodiment of the present invention provides an address information analyzing apparatus, as shown in fig. 4, the apparatus including:

An address information to be resolved obtaining unit 41, configured to obtain address information to be resolved in original data;

a first feature vectorization unit 42, configured to perform feature extraction, selection and vectorization on the address information to be analyzed by using a natural language processing technology to obtain a feature vector;

a model prediction unit 43, configured to input the feature vector into a preset model to obtain an initial array including a geographic entity and an administrative division level corresponding to the geographic entity; the preset model is obtained by training based on the combination of a cyclic neural network and a conditional random field algorithm;

the sorting unit 44 is configured to sort and deduplicate the geographic entities in the initial array according to the administrative division level to obtain a standard array;

And the geocoding unit 45 is used for coding the standard array to obtain a geocoding result.

Preferably, the apparatus further comprises:

The analysis record judging unit 46 is connected with the to-be-analyzed address information acquiring unit 41 and is used for judging whether the to-be-analyzed address information is analyzed or not according to the pre-stored historical address information analysis record; the historical address information analysis record comprises historical address information and corresponding historical geographic coding data;

and the analysis record obtaining unit 47 is connected to the analysis record judging unit 46, and is used for obtaining the corresponding historical geocoding data as a geocoding result when the address information to be analyzed is judged to be analyzed.

the first feature vectorization unit 42 is specifically configured to, when it is determined that the address information to be analyzed is not analyzed, perform feature extraction on the address information to be analyzed by using a natural language processing technology.

to avoid incomplete information in the array, the apparatus further comprises, before encoding the standard array to obtain the geocoding result, the method further comprising:

A completion unit 48, configured to match the standard array obtained by sorting by the sorting unit 44 with a pre-stored geographic position tree dictionary, determine whether the standard array is missing, and complete the standard array according to the geographic position tree dictionary when the standard array is missing; the geographic position tree dictionary is formed by dividing step by step according to administrative regions;

the geocoding unit 45 is specifically configured to code the complemented standard array to obtain a geocoding result.

the device also comprises a unit for constructing the preset model in advance, and specifically comprises

The second characteristic vectorization unit is used for extracting characteristics of the address data in the sample set by using a natural language processing technology, selecting the characteristics, and vectorizing the selected characteristics to obtain sample characteristic vectors; the specific process of this step can be seen in the related description in the first embodiment. Wherein the second feature vectoring unit may be the same as or different from the first feature vectoring unit.

the sample administrative entity relation unit is used for performing corpus labeling on the address data in the sample set to obtain a sample array consisting of sample geographic entities and sample administrative division levels corresponding to the sample geographic entities;

And the model training unit is used for inputting the sample feature vector, outputting the sample array, training through an RNN (recurrent neural network) and a CRF (conditional random field) algorithm, and constructing the preset model.

the geocoding result can be combined with other data to provide a data basis for subsequent application decision, and for this reason, the device further comprises an associated storage unit for storing the geocoding result and original data corresponding to the result in an associated manner.

EXAMPLE five

corresponding to the above method and apparatus, a fifth embodiment of the present application provides a computer system, including:

One or more processors; and

Acquiring address information to be analyzed in original data;

Extracting and selecting the characteristics of the address information to be analyzed by using a natural language processing technology, and vectorizing the selected characteristics to obtain characteristic vectors;

inputting the feature vector into a preset model to obtain an initial array comprising a geographic entity and an administrative division level corresponding to the geographic entity;

and coding the standard array to obtain a geocoding result.

Fig. 5 illustrates an architecture of a computer system, which may include, in particular, a processor 1510, a video display adapter 1511, a disk drive 1512, an input/output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, video display adapter 1511, disk drive 1512, input/output interface 1513, network interface 1514, and memory 1520 may be communicatively coupled via a communication bus 1530.

The processor 1510 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solution provided by the present Application.

the Memory 1520 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1520 may store an operating system 1521 for controlling the operation of the computer system 1500, a Basic Input Output System (BIOS) for controlling low-level operations of the computer system 1500. In addition, a web browser 1523, a data storage management system 1524, an icon font processing system 1525, and the like can also be stored. The icon font processing system 1525 may be an application program that implements the operations of the foregoing steps in this embodiment of the application. In summary, when the technical solution provided by the present application is implemented by software or firmware, the relevant program codes are stored in the memory 1520 and called for execution by the processor 1510.

the input/output interface 1513 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.

the network interface 1514 is used to connect a communication module (not shown) to enable the device to communicatively interact with other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).

The bus 1530 includes a path to transfer information between the various components of the device, such as the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520.

in addition, the computer system 1500 may also obtain information of specific extraction conditions from the virtual resource object extraction condition information database 1541 for performing condition judgment, and the like.

It should be noted that although the above devices only show the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, the memory 1520, the bus 1530, etc., in a specific implementation, the devices may also include other components necessary for proper operation. Furthermore, it will be understood by those skilled in the art that the apparatus described above may also include only the components necessary to implement the solution of the present application, and not necessarily all of the components shown in the figures.

from the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a cloud server, or a network device) to execute the method according to the embodiments or some parts of the embodiments of the present application.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

The data processing method, device and apparatus provided by the present application are introduced in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, the specific embodiments and the application range may be changed. In view of the above, the description should not be taken as limiting the application.