Detailed Description
As used in the specification and in the claims, the terms "a," "an," "the," and/or "the" are not specific to a singular, but may include a plurality, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.
Although the present application makes various references to certain modules in a system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and/or server. The modules are merely illustrative, and different aspects of the systems and methods may use different modules.
A flowchart is used in the present application to describe the operations performed by a system according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in order precisely. Rather, the various steps may be processed in reverse order or simultaneously, as desired. Also, other operations may be added to or removed from these processes.
Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
It should be noted in advance that the acquisition and processing of all information or data in the present application are performed on the premise of conforming to the policy of the corresponding data protection rule and obtaining the authorization given by the corresponding rights manager.
Fig. 1 is a flowchart of an intelligent place name address translation method based on multilingual syllable segmentation according to an embodiment of the present application. Fig. 2 is a data flow diagram of an intelligent place name address translation method based on multilingual syllable segmentation according to an embodiment of the present application. The intelligent place name address translation method based on multi-language syllable segmentation comprises the steps of S1, obtaining source language place name address texts and target languages appointed by users, S2, conducting translation strategy decision on each word unit in the source language place name address texts to obtain a sequence of source language place name address text words containing translation strategy labels, wherein the translation strategy comprises transliteration and dictionary translation, S3, calling a target language universal vocabulary translation dictionary to conduct standard translation on source language place name address text words marked as dictionary translation in the sequence of source language place name address text words containing the translation strategy labels based on the target languages to obtain a sequence of place name address universal vocabulary translation results, S4, conducting syllable segmentation mapping on the source language place name address text words marked as transliteration in the sequence of the source language place name address text words containing the translation strategy labels to obtain a sequence of place name address special translation results based on the target languages, S5, and conducting transliteration structure reconstruction on the sequence of the place name universal vocabulary translation results and the place name translation structure special address translation results.
In the above-mentioned intelligent place name address translation method based on multi-language syllable segmentation, in step S1, a source language place name address text and a target language specified by a user are obtained. It should be understood that the translation requirement of the place name address in practical application has obvious diversified characteristics, and the user needs to perform targeted conversion aiming at different source language place name addresses in the scenes of cross-border electronic commerce, international logistics, map service and the like and display the place name addresses in a target language required by a target user or a service scene. Therefore, in order to ensure the pertinence and the suitability of the translation service, the application is based on a customized input mechanism, firstly, the source language place name address text is collected, and simultaneously, the user is allowed to specify the target language of expected translation, so that basic input is provided for the subsequent translation process. In the implementation process, the required original place name address information and target language instructions can be obtained through a user input interface, a one-stop interface or a batch import mode, so that the standardized starting of the translation process and the flexibility of multi-language coverage are realized.
In the specific implementation process, the collection of the source language place name address text and the target language instruction can be realized in various modes. For example, in a cross-border e-commerce platform, the system may embed a multilingual address entry component that allows a user to fill in a ship-to or ship-to address in natural language form and automatically recognize the language hierarchy used for the address at the time of form submission as source language place name address text for extraction. At the same time, when the user selects the distribution destination area, the system automatically matches the official language or the common language of the area and records the official language or the common language as a target language instruction. The intelligent guiding mechanism based on the user behavior not only improves the convenience of data acquisition, but also enhances the language attribute recognition capability of the address information.
In the international logistics scene, the address information is often sourced from clients, suppliers or transportation nodes in different countries, and the formats and language styles of the address information are greatly different. In order to uniformly manage and efficiently process the heterogeneous data, the system can provide a batch import interface, allow a user to upload a file (such as an Excel table or a CSV file) containing a plurality of place name addresses, and perform language detection on the address content of each row in a file analysis stage, so as to automatically mark the source language type of the address content. Meanwhile, the user can manually designate the target language into which the imported data needs to be translated in the uploading interface, or automatically recommend the target language options according to preset rules (such as according to the country of the receiving part). The method is particularly suitable for large-scale address translation tasks, can remarkably improve data processing efficiency and reduces errors caused by manual intervention.
A map service application is another typical application environment. In such systems, a user may need to translate a particular place name address into another language version when navigating, searching for places, or sharing locations. At this time, the system can receive the spoken address information of the user through the voice recognition module, and judge the language type used by the text after the voice is converted into the text by utilizing the language recognition model, and the language type is used as the source language place name address text. The user can then switch language display modes on the map interface, whereupon the system determines the target language and incorporates address information for the current voice entry into the translation queue. The real-time interactive address acquisition mode not only improves user experience, but also provides instant and accurate language input for subsequent translation processing.
In addition, in some highly customized business systems, automatic acquisition of source language place name address text and target language instructions can also be realized by adopting an API interface calling mode. For example, if the information management system inside the enterprise needs to interface with the external multi-language address translation service, the information management system can automatically send a request to the translation server through configuring a standard RESTful API interface when the address data is newly added or modified each time. The request body should contain the original address text and the corresponding source language identification, and the target language parameter set by the user or the system. The method is suitable for an informatization system with higher integration level, and is beneficial to realizing cross-platform and cross-system address translation automatic flow.
At the technical implementation level, the retrieval of source language place name address text relies on language recognition technology in natural language processing. Language classification models, such as FastText, BERT, based on deep learning are generally used to perform language discrimination on the input text. After the model is trained by large-scale multilingual language materials, language attribution judgment of any text can be completed in millisecond time, and the model has high accuracy and is particularly suitable for complex scenes of mixed language or multilingual mixed input. The designation of the target language is more dependent on the design of the user interaction logic, including the modes of drop-down menu selection, automatic positioning matching, default language setting and the like, and can be used as an effective input channel.
It should be noted that in the actual deployment process, the integrity and normalization of the address text should also be considered. For example, some non-standardized addresses may lack the necessary administrative division level information, or may be misspelled, and have a confusing grammar structure, which may increase the difficulty of language identification. Therefore, the original text can be subjected to preliminary cleaning and structuring processing, such as redundant character removal, missing field complementation, unified unit expression and the like, by combining an address standardization module while the source language place name address text is acquired, so that the stability of the subsequent translation process is improved.
In the above-mentioned intelligent place name address translation method based on multi-language syllable segmentation, step S2 is to make a translation policy decision for each vocabulary unit in the source language place name address text to obtain a sequence of source language place name address text words including a translation policy label, where the translation policy includes transliteration and dictionary translation. That is, considering that the place name address often contains both a special vocabulary (such as a landmark, a road name, and a community name) that needs to retain voice characteristics, and a general vocabulary (such as "street", "head", "district") that needs to strictly reflect semantics and conform to local expression habits, the single translation strategy cannot fully satisfy the requirements of information fidelity and usability. Therefore, in order to flexibly adopt different processing strategies on the granularity of each vocabulary, the application introduces a translation strategy self-adaptive decision mechanism based on a deep neural network model, and the translation strategy which is adopted by each vocabulary unit is automatically identified and marked by carrying out deep semantic analysis and context awareness on the text of the address of the place name of the implemented source language. Fig. 3 is a flowchart of a sub-step S2 of the intelligent place name address translation method based on multilingual syllable segmentation according to an embodiment of the present application. As shown in FIG. 3, the step S2 includes the steps of S21, performing word segmentation processing on the source language place name address text to obtain a sequence of source language place name address text words, S22, performing semantic embedding coding on each source language place name address text word in the sequence of source language place name address text words based on mBERT models to obtain a sequence of source language place name address word granularity semantic embedding coding vectors, and S23, inputting the sequence of source language place name address word granularity semantic embedding coding vectors into a translation strategy classifier based on a transducer architecture to obtain the sequence of source language place name address text words containing translation strategy labels.
Specifically, in the step S21, word segmentation is performed on the source language place name address text to obtain a sequence of words of the source language place name address text. It should be understood that the place name address text is generally composed of a plurality of vocabulary units, including both universal vocabulary requiring dictionary translation and proper nouns or landmarks requiring transliteration processing, and in order to achieve granularity subdivision of the place name address text in subsequent translation policy decisions, the present application forms a sequence of place name address text words in a source language by performing word segmentation preprocessing on the input place name address text in a source language based on a word segmentation algorithm to segment continuous character strings into vocabulary units with independent semantic meaning. In the specific implementation process, aiming at the structural characteristics of different languages, the system selects an adaptive word segmentation tool, for example, a word segmentation method based on word frequency statistics and dictionary matching is adopted in a Chinese environment, and space and punctuation segmentation is adopted in English or other Latin system languages. By the method, the sequence of the text words of the source language place name address can be established on the minimum semantic unit level of the address, a standardized data input format is provided for the subsequent translation strategy labeling process, the problems of fuzzy word boundaries and mixed special words and universal words are fundamentally eliminated, and the fineness and the intelligent level of the whole translation processing flow are greatly improved.
Specifically, in the step S22, semantic embedded encoding based on mBERT models is performed on each source language place name address text word in the sequence of source language place name address text words, so as to obtain a sequence of source language place name address word granularity semantic embedded encoding vectors. Specifically, since each word unit in the place name address text has not only spelling information of a surface layer, but also has a context semantic dependency relationship with other word units. If classification labeling is performed only based on vocabulary information, specific category attributes of words, especially special category words such as landmarks and place names, are difficult to accurately identify, and functions of the special category words can be effectively distinguished only in specific contexts. Therefore, in order to capture deep semantic information of each source language place name address text word in the context, the application introduces a multi-language BERT (mBERT) deep pre-training semantic model, and performs semantic embedding encoding on each source language place name address text word so as to obtain semantic embedding representation of each text word. It should be appreciated that mBERT model has powerful cross-language understanding capability, can achieve efficient semantic coding of multi-language text, and accumulates rich contextual semantic knowledge through pre-training mechanisms. In the application, based on the strong cross-language semantic representation capability of mBERT model, each source language place name address text word can be mapped to a high-dimensional semantic space, and deep semantic association and context dependence among vocabularies are captured, so that a sequence of source language place name address word granularity semantic embedded coding vectors is obtained, and accurate and comprehensive semantic information support is provided for the self-adaptive decision of a subsequent translation strategy.
Specifically, in the step S23, the sequence of the source language place name address word granularity semantic embedded coding vector is input into a translation policy classifier based on a transducer architecture, so as to obtain the sequence of the source language place name address text word containing the translation policy label. It should be understood that, when performing a translation policy classification decision on text words of each source language place name address, the conventional fully-connected network often has over-fitting problems caused by fixed structure and excessive parameters, and long-distance semantic dependency relations in the sequence are difficult to capture. In order to realize fine-granularity context sensitive classification, the application adopts a transducer model to construct a translation strategy classifier, utilizes a self-attention mechanism and position coding of the transducer architecture to process long-distance dependency relations in a sequence, simultaneously maintains the sensitivity and parallel computing capacity of the model to each position in an input sequence, realizes deep context understanding of semantic information of text words of each source language place name address, and finally outputs binary labels (transliteration/dictionary translation) of text words of each source language place name address through a Softmax layer, thereby adaptively determining the most suitable translation strategy of each text word and providing powerful technical support for multilingual translation services of place name addresses.
In the above-mentioned intelligent place name address translation method based on multi-language syllable segmentation, step S3, based on the target language, calls a target language universal vocabulary translation dictionary to perform standard translation on the source language place name address text words marked as dictionary translation in the sequence of the source language place name address text words including translation policy labels, so as to obtain a sequence of place name address universal vocabulary translation results. Specifically, because the universal vocabulary often has clear and stable equivalent expression among different languages, the accurate transmission of the semantics can be ensured by directly utilizing the translation of the authoritative target language dictionary. Therefore, in order to ensure the normalization, the definition and the local audience universality of public descriptive words (such as azimuth, ordinal numbers, types and the like) in the address, the application is based on a preset multilingual dictionary, and the corresponding target language expression is directly searched by searching and matching each text word marked as dictionary translation in the source language place name address text by calling the target language universal word translation dictionary, so that the corresponding universal word translation result is generated. Through the standardized flow, possible misinterpretations or ambiguities of the existing neural translation model can be effectively avoided, and high-quality matching of universal vocabulary is achieved.
In the above-mentioned intelligent place name address translation method based on multi-language syllable segmentation, step S4 is to perform syllable segmentation mapping on the source language place name address text words marked as transliterations in the sequence of source language place name address text words including translation policy labels based on the target language, so as to obtain a sequence of place name address special vocabulary transliteration results. Specifically, due to the fact that the proprietary place names have high regional and phonological characteristics, the transliteration mode is adopted to keep the pronunciation information in the original language, and therefore recognition and scene restoration of cross-language users are achieved. Meanwhile, it is considered that direct whole-word transliteration may cause pronunciation distortion of a target language (such as transliteration of Chinese ' xi ' an ' to ' Xian ' is easy to be misread as ' Xian '). Therefore, in order to ensure the reliability and fluency of the transliteration process, the application further breaks each source language place name address text word marked as transliteration into the minimum pronunciation unit through fine granularity segmentation of syllable level, and maps each syllable one by one according to syllable structural features of the target language, so that each syllable can find the closest pronunciation expression in the target language, and natural fluency and easy understanding of the transliteration result are ensured. Fig. 4 is a flowchart of a substep S4 of the intelligent place name address translation method based on multilingual syllable segmentation according to an embodiment of the present application. As shown in FIG. 4, the step S4 includes the steps of S41, performing phoneme conversion on the source language place name address text words marked as transliterations by using a G2P model to obtain a source language place name address exclusive vocabulary phoneme sequence, S42, performing syllable segmentation on the source language place name address exclusive vocabulary phoneme sequence to obtain a source language place name address exclusive vocabulary syllable sequence, S43, performing syllable mapping based on a target language syllable library on each source language place name address exclusive vocabulary syllable in the source language place name address exclusive vocabulary syllable sequence to obtain a target language place name address exclusive vocabulary syllable sequence, and S44, performing transliteration conversion on the target language place name address exclusive vocabulary syllable sequence into text by using a P2G model to obtain a place name address exclusive vocabulary syllable result.
Specifically, in step S41, the G2P model is used to perform phoneme conversion on the text word of the source language place name address marked as transliteration, so as to obtain a phoneme sequence of the source language place name address proprietary vocabulary. It should be appreciated that the present application contemplates proper nouns in place name addresses (e.g., street name'"Or City name"(Seoul) ") the written form (glyph) varies greatly among different languages, transliteration based directly on characters is susceptible to spelling rule interference (e.g., english" Leicester "pronounciation +.Instead of the surface pronunciation of the letter combination), and the multi-language phonetic system is obviously different (such as the Chinese phonetic alphabet 'x' corresponds to the international phonetic symbol +.V, spanish "j" send/x/sound). Thus, to accurately capture the actual pronunciation characteristics of the source language place name, the present application converts spelled characters into international phonetic symbols (IPA) by training a multitasking G2P (Grapheme-to-Phoneme) model based on the phonetic phoneme conversion principle. In a specific implementation process, a transducer architecture G2P model (such as a Pinyin-to-IPA model based on initial and final decomposition is adopted in Chinese, and a mapping rule from Cyrillic letters to X-SAMPA symbols is adopted in Russian) pre-trained for a source language is loaded, phoneme-level conversion is carried out on a source language place name address text word marked as transliteration (such as 'Rivoli' in French 'Rue de Rivoli') to generate a standardized special vocabulary phoneme sequence (such as 'Rivoli'. Fwdarw. & gt)V) provides an accurate pronunciation reference for subsequent syllable segmentation mappings.
Specifically, in step S42, syllable segmentation is performed on the source language place name address specific vocabulary phoneme sequence to obtain a source language place name address specific vocabulary syllable sequence. Specifically, because of the intrinsic difference of syllable dividing rules of different languages (such as English allows complex consonant clusters and Japanese strictly follows CV structure), and mechanical division of phonemes is liable to cause pronunciation distortion (such as "western" in Chinese)Error cut into-/). Therefore, in order to realize syllable boundary positioning conforming to voice perception, the application further carries out context dependency analysis on each phoneme in the phoneme sequence of the source language place name address special vocabulary so as to accurately identify syllable boundaries, thereby dividing each syllable unit in the source language place name address special vocabulary. Fig. 5 is a flowchart of a substep S42 of the intelligent place name address translation method based on multilingual syllable segmentation according to the embodiment of the present application. As shown in FIG. 5, the step S42 includes the steps of S421 inputting the source language place name address exclusive vocabulary phoneme sequence into a Bi-LSTM model containing an embedding layer to obtain a sequence of source language place name address exclusive vocabulary phoneme embedding association coding vectors, S422 extracting the source language place name address exclusive vocabulary phoneme embedding association coding vectors to be analyzed from the sequence of source language place name address exclusive vocabulary phoneme embedding association coding vectors, and performing syllable boundary prediction on the source language place name address exclusive vocabulary phoneme embedding association coding vectors to be analyzed to obtain syllable boundary marking prediction results, S423 performing syllable segmentation on the source language place name address exclusive vocabulary phoneme sequence according to the syllable boundary prediction results to obtain the source language place name address exclusive vocabulary syllable sequence.
More specifically, in step S421, the sequence of source language place name address specific vocabulary phonemes is input into a Bi-LSTM model including an embedding layer to obtain a sequence of source language place name address specific vocabulary phonemes embedding associated encoding vectors. It should be appreciated that due to the original phoneme sequence derived from the G2P model (e.g., as "/b/,"The discrete phoneme notation of/,/r/,/k/,/i/") itself lacks direct, rich context-related information, whereas the boundary partitioning of syllables is often highly dependent on the combined environment before and after the phoneme and the phoneme arrangement rules of the language (phonotactics). Therefore, in order to fully understand the specific roles and contextual dependencies of each phoneme in its sequence and thereby accurately predict syllable boundaries, the present application is based on the principles of deep learning sequence modeling by first mapping each discrete phoneme symbol into a low-dimensional dense real vector through an embedding layer (Embedding Layer) and preserving phoneme-to-phoneme similarity in vector space. And modeling the context dependency relationship among the phonemes through a Bi-directional long-short-term memory network (Bi-LSTM), and capturing long-distance dependency characteristics among the phonemes to obtain deep-level association information among the phonemes. Specifically, the Bi-LSTM layer separately captures forward and backward dependency relationships of phonemes by traversing the phoneme sequences from the forward direction and the backward direction using two separate LSTM networks, and generates a context-aware representation of each phoneme by concatenating the forward hidden state and the backward hidden state of each phoneme embedding feature to obtain a sequence of source language place name address-specific vocabulary phoneme embedding associated encoding vectors. By the method, complex interaction among phonemes can be more accurately understood, and a powerful basis is provided for subsequent syllable boundary prediction.
More specifically, in step S422, the relevant encoding vector for embedding the source language place name address specific vocabulary phonemes is extracted from the sequence of relevant encoding vectors for embedding the source language place name address specific vocabulary phonemes, and syllable boundary prediction is performed on the relevant encoding vector for embedding the source language place name address specific vocabulary phonemes to be analyzed, so as to obtain a syllable boundary labeling prediction result. Specifically, since syllable segmentation requires consideration of both local phoneme combination legitimacy (e.g., japanese disallowed "and" following "consonants) and global prosody structure (e.g., chinese bissyllabization tendency), feature extraction at a single scale tends to result in over-segmentation or under-segmentation (e.g., french" aujourd' hui ")Error segmentation into trisyllabic and not correct tetrasyllabic). In order to realize accurate syllable boundary prediction, the application extracts the source language place name address special vocabulary phoneme embedding association coding vector to be analyzed from the sequence of the source language place name address special vocabulary phoneme embedding association coding vector as an analysis target, and measures syllable boundary adaptation degree under different granularities by mining local neighborhood structures and global distribution characteristics of the source language place name address special vocabulary phoneme embedding association coding vector in a phoneme embedding space, thereby realizing intelligent prediction of syllable boundaries. Fig. 6 is a flowchart of sub-step S422 of the intelligent place name address translation method based on multilingual syllable segmentation according to the embodiment of the present application. As shown in FIG. 6, the step S422 includes the step S4221 of performing syllable boundary prediction based on global context correlation sensing on the to-be-analyzed source language place name address private vocabulary phoneme embedding association code vector to obtain a to-be-analyzed phoneme feature syllable boundary global adaptation degree, the step S4222 of performing syllable boundary prediction based on local neighborhood correlation sensing on the to-be-analyzed source language place name address private vocabulary phoneme embedding association code vector to obtain a to-be-analyzed phoneme feature syllable boundary local adaptation degree, and the step S4223 of determining whether a phoneme corresponding to the to-be-analyzed source language place name address private vocabulary phoneme embedding association code vector is a syllable boundary based on the to-be-analyzed phoneme feature syllable boundary global adaptation degree and the to-be-analyzed phoneme feature syllable boundary local adaptation degree.
In a specific example of the present application, the step S4221 includes first inputting the sequence of the source language place name address specific vocabulary phoneme embedding associated code vector to be analyzed and the source language place name address specific vocabulary phoneme embedding associated code vector into a global context adaptation degree learning module based on a converter structure to obtain a phoneme feature syllable boundary global context adaptation associated code vector to be analyzed, which is expressed by a formula:
;
;
wherein, theA sequence representing the source language place name address specific vocabulary phoneme embedding an associated encoding vector,、、AndThe 1 st, 2 nd and 2 nd of the sequence of embedded associated coding vectors respectively representing the phonemes of the source language place name address private vocabularyAnd (b)The phonemes of the private vocabulary of the individual source language place name are embedded with associated encoding vectors,Embedding associated coding vectors into the phonemes of the special vocabulary for representing the place name and the address of the source language to be analyzed,Representing the transform encoder and the data of the transform,The global context adaptation associated coding vector representing the syllable boundary of the phoneme feature to be analyzed.
That is, a transducer architecture captures complex dependency between a proprietary vocabulary phoneme of a place name address of a source language to be analyzed and an overall sequence of the proprietary vocabulary phoneme in the source language, and generates a global context adaptation associated coding vector of a syllable boundary of a feature syllable of the phoneme to be analyzed with high context awareness. The global context adaptation associated coding vector of the syllable boundary of the phoneme feature to be analyzed not only contains the attribute information of the phoneme to be analyzed, but also encapsulates the structural roles of the phoneme feature to be analyzed in the whole proprietary vocabulary phoneme distribution, the interaction relation of the phoneme feature to other phonemes and the position of the phoneme feature in the global phoneme ecosystem, and provides an information basis for accurately dividing syllable boundaries and realizing cross-language syllable mapping.
And then, performing explicit decoding on the global context adaptation associated coding vector of the syllable boundary of the phoneme feature to be analyzed to obtain the global adaptation degree of the syllable boundary of the phoneme feature to be analyzed, wherein the global adaptation degree is expressed as follows by a formula:
;
wherein, theRepresenting the sigmoid activation function,The transpose is represented by the number,In order for the weight parameters to be learnable,The term of the bias is indicated,And representing the global adaptation degree of syllable boundaries of the phoneme characteristic to be analyzed.
That is, through an explicit decoding process, the syllable boundary global context adaptation associated coding vector of the phoneme feature to be analyzed is projected from a high-dimensional space to a low-dimensional scalar value space, and a signal most relevant to the syllable boundary global adaptation is extracted and explicit from the high-dimensional space to quantify the degree of adaptation of the phoneme feature to be analyzed to a macroscopic pattern or distribution defined by the whole source language place name address private vocabulary phoneme set. Through the information compression and abstract extraction, the global adaptation degree of the syllable boundary of the phoneme feature to be analyzed, which directly reflects the rationality of syllable boundary division, can be obtained, the subsequent syllable segmentation and mapping decision are effectively guided, and the pronunciation restoration precision and the cross-language voice mapping accuracy in the proper noun transliteration process are improved.
Fig. 7 is a flowchart of substep S4222 of the intelligent place name address translation method based on multilingual syllable segmentation according to an embodiment of the present application. As shown in FIG. 7, the step S4222 comprises the steps of S42221 of extracting local neighborhood context information of the source language place name address exclusive vocabulary phoneme embedding association code vector to be analyzed in the sequence of the source language place name address exclusive vocabulary phoneme embedding association code vector to obtain a local neighborhood set of the source language place name address exclusive vocabulary phoneme embedding association code vector, S42222 of inputting the local neighborhood set of the source language place name address exclusive vocabulary phoneme embedding association code vector to be analyzed and the source language place name address exclusive vocabulary phoneme embedding association code vector into a local context adaption degree learning module to obtain the local adaption degree of the syllable boundary of the phoneme feature to be analyzed.
In a specific example of the present application, the step S42221 is expressed as:
;
wherein, theRepresenting the local neighborhood feature extraction radius,Expressed in terms ofEmbedding a sequence of associated encoding vectors into the source language place name address private vocabulary phonemesThe position of (3) is taken as the center, and the radius is extractedThe phonemes of the source language place name address specific vocabulary in the interior are embedded with associated encoding vectors,A local neighborhood set representing source language place name address specific vocabulary phonemes embedding associated encoding vectors,Embedding the dimensions of vectors in the local neighborhood set of associated encoding vectors for the source language place name address specific vocabulary phonemes,And embedding the number of vectors in the local neighborhood set of the associated coding vector into the special vocabulary phonemes of the source language place name address.
That is, the micro-environment or small range most directly related to the source language place name address proprietary vocabulary phonemes to be analyzed is defined and isolated, and a data basis is provided for the subsequent evaluation of the local adaptation degree between the micro-environment or small range and the peripheral phonemes, so that the phoneme characteristics are evaluated in different local environments, and the robustness of the evaluation is enhanced. By the method, local context information around the phonemes to be analyzed can be focused, short-distance dependency relations and local voice modes among the phonemes are captured, support is provided for follow-up more accurate syllable boundary recognition and cross-language syllable mapping realization, and accuracy and naturalness of transliteration results are improved.
In a specific example of the present application, the step S42222 is expressed as:
;
;
wherein, theEmbedding the special vocabulary phonemes of the source language place name address into the local neighborhood set of the associated encoding vectorThe phonemes of the private vocabulary of the individual source language place name are embedded with associated encoding vectors,Representation ofRelative toIs a syllable boundary monomer phoneme association adaptation degree,In order for the weight parameters to be learnable,、AndFor the local neighborhood weight matrix,In order to activate the function,Is thatAndIs used for the feature scale of (a),Representing the hyperbolic tangent function,And representing the local adaptation degree of the syllable boundary of the phoneme characteristic to be analyzed.
Namely, the adaptation degree of the phoneme features and syllable boundaries is evaluated from the dimension of the local context, the syllable boundary monomer phoneme association adaptation degree between the phoneme features to be analyzed and each phoneme feature in each local neighborhood is mined through quantization, and the context association adaptation information of all local neighborhood phonemes is aggregated in a mean value, so that the phonetic feature association mode of the phonemes in the local range is captured, the syllable boundary local adaptation degree of the phoneme features to be analyzed is obtained, the accurate recognition of the syllable boundaries conforming to the pronunciation rules of the source language is facilitated, and the accuracy of proper nouns in the syllable segmentation stage is effectively improved. Meanwhile, by combining local context information, the subsequent syllable mapping process can better keep the pronunciation characteristics of the source language, and finally the recognizability and the voice reduction degree of the proprietary name translation in the multilingual environment are enhanced.
In a preferred example of the present application, the step S42221 includes centering on the source language place name address exclusive vocabulary phoneme embedding association encoding vector to be analyzed, extracting a local neighborhood set of the source language place name address exclusive vocabulary phoneme embedding association encoding vector based on an initial local neighborhood feature extraction radius, iteratively optimizing the initial local neighborhood feature extraction radius based on a mixed state information entanglement strength between the local neighborhood set of the source language place name address exclusive vocabulary phoneme embedding association encoding vector and the source language place name address exclusive vocabulary phoneme embedding association encoding vector to be analyzed, and extracting a local neighborhood set of the source language place name address exclusive vocabulary phoneme embedding association encoding vector based on the optimized local neighborhood feature extraction radius based on the source language place name address exclusive vocabulary phoneme embedding association encoding vector to be analyzed.
Considering that when the local neighborhood set corresponding to the source language place name address special vocabulary phoneme embedding association coding vector to be analyzed is extracted, although the local neighborhood set of the source language place name address special vocabulary phoneme embedding association coding vector defines a related micro-environment or a small range, the source language place name address special vocabulary phoneme embedding association coding vector to be analyzed and the local neighborhood set thereof still transition from a geometric eigenstate to an association coupling phase state, so that measurement association coupling is caused when the characteristic local adaptation degree between the source language place name address special vocabulary phoneme embedding association coding vector and the local neighborhood set thereof is calculated. Therefore, the application expects to improve the calculation accuracy of the local adaptation degree of syllable boundaries of the phoneme feature to be analyzed by iteratively optimizing the initial local neighborhood feature extraction radius.
Specifically, first, let theTaking the immediate manifold architecture based on microscopic behavior patterns as probability density reduced distribution, thereby obtaining the mixed functional index value of the pure-state adaptive functional in the technical form of the global delocalized entropy characteristic:
;
wherein, theThe logarithmic value representing the base of the natural constant,Representation ofRelative toIs used for the pure state adaptation degree of the (a),Representing the global mixed state functional index value.
That is, each is toAs a case of homomorphic fitness distributions, global non-local uncertainties or degrees of mixing are formally measured based on a statistical mix of each homomorphic fitness distribution in order to quantify the mixture characterization at different degrees.
At the same time, the geometric regularization factor is introducedIn the case of (a), a manifold measure expansion value is calculated:
;
wherein, theThe logarithmic value based on 2 is shown,Represents the entanglement strength of the mixed-state information,Represent the firstAnd a geometric regularization factor.
That is, the global non-negative eigen decomposition of the homography adaptation degree distribution is utilized to expand the paradigm based on entropy correlation, thereby expanding the hybrid representation to correlate information entropy with degree of entanglement. Thus, the initial local neighborhood feature extraction radius can be made by modulating the neighborhood parametersWhich is minimized to reduce the measure associative coupling. And finally, based on the modulated and optimized local neighborhood feature extraction radius, taking the source language place name address special vocabulary phoneme embedding association coding vector to be analyzed as a center, re-extracting a local neighborhood set of the source language place name address special vocabulary phoneme embedding association coding vector, and inputting the local neighborhood set of the source language place name address special vocabulary phoneme embedding association coding vector to be analyzed and the local neighborhood set of the source language place name address special vocabulary phoneme embedding association coding vector into the local context adaption learning module to calculate the local adaption degree of the syllable boundary of the phoneme feature to be analyzed, thereby improving the calculation accuracy of the local adaption degree of the syllable boundary of the phoneme feature to be analyzed.
In a specific example of the present application, the step S4223 includes determining a global fit of the syllable boundary of the phoneme feature to be analyzed, based on the global fit of the syllable boundary of the phoneme feature to be analyzed and the local fit of the syllable boundary of the phoneme feature to be analyzed, and the global fit of the syllable boundary of the phoneme feature to be analyzed is expressed as:
;
;
wherein, theAndRespectively representing the global adaptation degree of the syllable boundary of the phoneme characteristic to be analyzed and the weighted fusion weight of the local adaptation degree of the syllable boundary of the phoneme characteristic to be analyzed,Representing the normalized exponential function of the sample,Representing a comprehensive fit weight parameter matrix,Representing the proper configuration offset of the heddle,And representing the fit degree of syllable boundary of the phoneme characteristic to be analyzed.
That is, by integrating global adaptation of syllable boundaries of the phoneme feature to be analyzed and local adaptation of syllable boundaries of the phoneme feature to be analyzed, a comprehensive evaluation standard which is more comprehensive and more robust is constructed so as to solve the limitation of single-scale evaluation in syllable segmentation of proper nouns of cross-language place name addresses. The global adaptation degree can capture the overall association mode of the phoneme features and syllable boundaries from the macroscopic level, the local adaptation degree can mine the fine voice structure of the phonemes in the local context, and the combination of the global adaptation degree and the syllable boundaries can fully exert a synergistic effect, accurately position the syllable boundaries which accord with the pronunciation rules of the source language and give consideration to the voice habits of the target language. The generated comprehensive fit degree of the syllable boundary of the phoneme feature to be analyzed can effectively solve the problems that the global analysis ignores local voice details and the local analysis lacks overall voice association, and the accuracy and the stability of syllable boundary recognition are remarkably improved.
And then, based on the comparison between the comprehensive fit degree of the syllable boundary of the phoneme feature to be analyzed and a preset threshold value, determining whether the phoneme corresponding to the phoneme embedding association coding vector of the special vocabulary of the place name address of the source language to be analyzed is a syllable boundary or not. That is, by comparing the overall fit of syllable boundaries of the phoneme feature to be analyzed with a preset threshold, an explicit decision basis is provided for whether each phoneme constitutes a syllable boundary or not on the basis of comprehensively considering global speech association and local pronunciation details. Through the binary result generated by the threshold decision mechanism, redundant phoneme boundaries with weak relevance of voice features can be accurately filtered, effective boundaries conforming to the multi-language pronunciation rules are reserved, and the accuracy and the efficiency of syllable segmentation of proper nouns are remarkably improved. When facing to complex scenes such as emerging landmarks or cold place names, transliteration deviation caused by syllable segmentation errors can be effectively reduced, and semantic recognizability and information transfer accuracy in a cross-language environment are enhanced.
More specifically, step S423 performs syllable segmentation on the syllable sequence of the source language place name address specific vocabulary according to the syllable boundary prediction result to obtain the syllable sequence of the source language place name address specific vocabulary. It should be appreciated that the syllable boundary prediction results reveal the ending location of each syllable unit in the sequence of source language place name address specific vocabulary phones. Based on the syllable boundary information, the syllable sequence of the source language place name address exclusive vocabulary is segmented, so that a series of separated syllable units, namely the syllable sequence of the source language place name address exclusive vocabulary, can be obtained. For example, if the original source language place name address specific vocabulary phoneme sequence is +.If/, and the predicted boundary label indicates that the syllable unit is ending at/k/and/i/, then the segmentation operation will produce two syllables, the first syllable being +.The second syllable is/li/. In this way, the original continuous stream of phonemes is efficiently and accurately converted into an ordered list or sequence of syllables, e.g. [ -A ]/,/li/]。
Specifically, in the step S43, syllable mapping based on the target language syllable library is performed on each source language place name address special vocabulary syllable in the source language place name address special vocabulary syllable sequence, so as to obtain the target language place name address special vocabulary syllable sequence. It should be appreciated that due to transliteration taking into account pronunciation approximations and target language orthography specifications (e.g., japanese katakana cannot directly represent french nasal vowels), simple phoneme transliteration is prone to illegal syllables (e.g., english "Smith")Mapping to Korean language is to avoid occurrence ""Etc. inactive consonants). Therefore, in order to realize the balance between the pronunciation fidelity and the target language acceptability, the application searches the optimal mapping path by constructing the target language syllable library and adopting a multi-level syllable mapping method based on the optimal syllable alignment principle. Specifically, a target language syllable list (such as Japanese JIS X4063 katakana syllable list and Arabic ISO 233 transliteration rule) is loaded first, and then multiple candidate matching is performed on each source syllable, for example, chinese syllable "zhang" is given toWhen mapping into Russian, calculate its and "”(//)、“”(//(V)) etc. and in combination with historical translation frequencies (e.g. "Zhang →And (3) selecting an optimal mapping by comprehensively considering the phoneme editing distance and the historical translated name frequency, and finally generating the syllable sequence of the target language place name address exclusive vocabulary.
Specifically, in the step S44, the P2G model is used to convert the syllable sequence of the target language place name address specific vocabulary into text, so as to obtain the transliteration result of the place name address specific vocabulary. That is, in order to convert the target language place name address specific vocabulary syllable sequence into a text form conforming to the target language writing specification, the present application converts the target language syllable sequence into a final text output by using a P2G model trained for the target language based on the phoneme-word position conversion (Phoneme-to-Grapheme, P2G) principle. Specifically, the P2G model is also a deep learning based sequence-to-sequence model, and can select the most appropriate character or letter combination according to syllable sequence, context information and common vocabulary and writing habit of the target language, so that accurate pronunciation of the transliteration result is ensured, a text form which accords with the writing specification of the target language and is easy to read and understand is generated, complete conversion from proper names of the source language to high-quality transliteration text of the target language is completed, and the final place name address translation result can accurately convey geographic information and can meet reading and using habits of a target language user.
In the above-mentioned intelligent place name address translation method based on multi-language syllable segmentation, in step S5, the sequence of the place name address universal vocabulary translation result and the sequence of the place name address special vocabulary transliteration result are structurally reconstructed to obtain the target language place name address translation result. It should be understood that, in the present application, the structural requirement of the place name address text is high, and although the single vocabulary has been correctly translated or transliterated, if the single vocabulary is not recombined according to grammar or local habit, the problems of confusion of language sequence, unclear expression and the like easily occur. Therefore, in order to finally output normalized and standardized destination language place name address on the premise of comprehensively guaranteeing ideographic accuracy and readability, the application is based on a structural recombination engine to automatically splice and format universal translation words and transliteration proprietary word results according to the word sequence, punctuation and writing standards of the destination language. In the implementation process, firstly, a target language template library (comprising English "[ block ] [ type ], [ city ] [ suffix ]", japanese "[ Du Fu Yuan ] [ City district village ] [ Ding mu ]", and the like) is constructed, transliteration results and dictionary translation results are ordered according to target language specifications (English, necessary prepositions and format symbols (such as English commas and Japanese black dots) are inserted, and in this way, the translation results not only keep source semantic information, but also accord with the cognition habit of target users.
In summary, the intelligent place name address translation method based on multi-language syllable segmentation according to the embodiment of the application is explained, which carries out self-adaptive translation strategy decision on each vocabulary unit in the source language place name address text by introducing a deep learning algorithm so as to automatically distinguish proper nouns which should adopt transliteration from universal vocabulary which should adopt dictionary translation. And then, carrying out fine granularity segmentation of syllable level and target language syllable mapping on the proper nouns marked as transliteration to realize accurate transliteration of the proper nouns, and simultaneously, carrying out accurate translation on the universal vocabulary marked as dictionary translation by adopting a dictionary matching algorithm. Furthermore, the transliteration result of the proper noun and the translation result of the universal vocabulary are normalized and recombined to obtain the place name address translation result. The method can effectively improve the translation effect of the special place name when the special place name and the general word are mixed and output, and enhance the translation accuracy and flexibility of the place name address in a multi-language environment.
The basic principles of the present invention have been described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present invention are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be construed as necessarily possessed by the various embodiments of the invention. Furthermore, the particular details of the above-described embodiments are for purposes of illustration and understanding only, and are not intended to limit the invention to the particular details described above, but are not necessarily employed.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods may be implemented in other ways. For example, the system embodiments described above are merely illustrative, e.g., the unit division is merely a logical function division, and other manners of division may be implemented in practice. The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. The units recited in the system claims may also be implemented by means of software or hardware.
Finally, it should be noted that the foregoing description has been presented for the purposes of illustration and description. Furthermore, the foregoing embodiments are merely for illustrating the technical scheme of the present invention, and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the technical scheme of the present invention.