CN114492426B - Sub-word segmentation method, model training method, device and electronic equipment - Google Patents
Sub-word segmentation method, model training method, device and electronic equipmentDownload PDFInfo
- Publication number
- CN114492426B CN114492426BCN202111656289.9ACN202111656289ACN114492426BCN 114492426 BCN114492426 BCN 114492426BCN 202111656289 ACN202111656289 ACN 202111656289ACN 114492426 BCN114492426 BCN 114492426B
- Authority
- CN
- China
- Prior art keywords
- state
- probability
- candidate
- target
- text sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
- G06F18/295—Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
- G10L15/063—Training
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/14—Speech classification or search using statistical models, e.g. Hidden Markov Models [HMMs]
- G10L15/142—Hidden Markov Models [HMMs]
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L2015/088—Word spotting
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Acoustics & Sound (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本公开涉及计算机技术领域，尤其涉及一种子词切分方法、模型训练方法、装置、电子设备、存储介质和计算机程序产品。The present disclosure relates to the field of computer technology, and in particular to a subword segmentation method, a model training method, a device, electronic equipment, a storage medium and a computer program product.
背景技术Background technique
目前，随着人工智能技术的发展，语言模型在语音识别、语音交互、语音合成等领域得到了广泛应用，比如，同声传译场景中，可通过语言模型将演讲者发出的语音翻译成设定语言类别的文本，以及合成设定语言类别的语音，并通过设备播放合成的语音。语言模型需要词表进行训练，词表大多基于子词切分方法得到，然而，相关技术中的子词切分方法存在泛化性差的问题。At present, with the development of artificial intelligence technology, language models have been widely used in speech recognition, speech interaction, speech synthesis and other fields. The text of the language category, and synthesize the voice of the set language category, and play the synthesized voice through the device. The language model needs a vocabulary for training, and the vocabulary is mostly obtained based on subword segmentation methods. However, the subword segmentation methods in the related art have the problem of poor generalization.
发明内容Contents of the invention
本公开提供了一种子词切分方法、模型训练方法、装置、电子设备、存储介质和计算机程序产品。The disclosure provides a subword segmentation method, a model training method, a device, an electronic device, a storage medium and a computer program product.
根据本公开的一方面，提供了一种子词切分方法，包括：获取待切分的文本序列，其中，所述文本序列包括多个元素；获取每个候选状态下的初始状态概率、每个所述候选状态下的每个所述元素的观测概率、任意相邻两个所述元素的所述候选状态之间的状态转移概率，其中，所述候选状态用于表征所述元素是否为切分边界；根据所述初始状态概率、所述观测概率和所述状态转移概率，从所述候选状态中确定所述元素的目标状态；根据所述元素的所述目标状态，对所述文本序列进行切分，得到多个子词，其中，所述子词包括至少一个所述元素。According to an aspect of the present disclosure, a subword segmentation method is provided, including: obtaining a text sequence to be segmented, wherein the text sequence includes a plurality of elements; obtaining the initial state probability of each candidate state, each The observation probability of each element in the candidate state, the state transition probability between the candidate states of any two adjacent elements, wherein the candidate state is used to represent whether the element is cut According to the initial state probability, the observation probability and the state transition probability, determine the target state of the element from the candidate state; according to the target state of the element, the text sequence Segmentation is performed to obtain a plurality of subwords, wherein the subwords include at least one of the elements.
根据本公开的另一方面，提供了一种模型训练方法，包括：获取样本文本序列，其中，所述样本文本序列包括多个样本元素；根据所述样本文本序列训练概率图模型，对所述概率图模型的模型参数进行更新，其中，所述概率图模型用于输出每个候选状态下的训练初始状态概率、每个所述候选状态下的每个所述样本元素的训练观测概率、任意相邻两个所述样本元素的所述候选状态之间的训练状态转移概率，其中，所述候选状态用于表征所述样本元素是否为切分边界；在未满足模型训练结束条件的情况下，返回采用下一个样本文本序列继续对更新后的所述概率图模型进行训练，直至满足所述模型训练结束条件，生成训练好的所述概率图模型。According to another aspect of the present disclosure, a model training method is provided, including: acquiring a sample text sequence, wherein the sample text sequence includes a plurality of sample elements; training a probability graphical model according to the sample text sequence, and The model parameters of the probability graph model are updated, wherein the probability graph model is used to output the training initial state probability in each candidate state, the training observation probability of each sample element in each candidate state, any The training state transition probability between the candidate states of two adjacent sample elements, wherein the candidate state is used to characterize whether the sample element is a segmentation boundary; in the case that the model training end condition is not met , return to continue training the updated probability graphical model by using the next sample text sequence until the model training end condition is met, and generate the trained probability graphical model.
根据本公开的另一方面，提供了一种子词切分装置，包括：第一获取模块，用于获取待切分的文本序列，其中，所述文本序列包括多个元素；第二获取模块，用于获取每个候选状态下的初始状态概率、每个所述候选状态下的每个所述元素的观测概率、任意相邻两个所述元素的所述候选状态之间的状态转移概率，其中，所述候选状态用于表征所述元素是否为切分边界；确定模块，用于根据所述初始状态概率、所述观测概率和所述状态转移概率，从所述候选状态中确定所述元素的目标状态；切分模块，用于根据所述元素的所述目标状态，对所述文本序列进行切分，得到多个子词，其中，所述子词包括至少一个所述元素。According to another aspect of the present disclosure, a subword segmentation device is provided, including: a first acquisition module, configured to acquire a text sequence to be segmented, wherein the text sequence includes a plurality of elements; a second acquisition module, It is used to obtain the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the state transition probability between the candidate states of any two adjacent elements, Wherein, the candidate state is used to represent whether the element is a segmentation boundary; a determining module is used to determine the A target state of an element; a segmentation module configured to segment the text sequence according to the target state of the element to obtain a plurality of subwords, wherein the subwords include at least one of the elements.
根据本公开的另一方面，提供了一种模型训练装置，包括：获取模块，用于获取样本文本序列，其中，所述样本文本序列包括多个样本元素；训练模块，用于根据所述样本文本序列训练概率图模型，对所述概率图模型的模型参数进行更新，其中，所述概率图模型用于输出每个候选状态下的训练初始状态概率、每个所述候选状态下的每个所述样本元素的训练观测概率、任意相邻两个所述样本元素的所述候选状态之间的训练状态转移概率，其中，所述候选状态用于表征所述样本元素是否为切分边界；所述训练模块，还用于在未满足模型训练结束条件的情况下，返回采用下一个样本文本序列继续对更新后的所述概率图模型进行训练，直至满足所述模型训练结束条件，生成训练好的所述概率图模型。According to another aspect of the present disclosure, a model training device is provided, including: an acquisition module, configured to acquire a sample text sequence, wherein the sample text sequence includes a plurality of sample elements; a training module, configured to The text sequence trains the probability graph model, and updates the model parameters of the probability graph model, wherein the probability graph model is used to output the training initial state probability in each candidate state, each of the candidate states in each The training observation probability of the sample element, the training state transition probability between the candidate states of any two adjacent sample elements, wherein the candidate state is used to represent whether the sample element is a segmentation boundary; The training module is also used to return to the next sample text sequence to continue training the updated probability graph model until the end condition of the model training is met, and generate a training Well said probabilistic graphical model.
根据本公开的另一方面，提供了一种电子设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行子词切分方法，或者执行模型训练方法。According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores Instructions to be executed, the instructions are executed by the at least one processor, so that the at least one processor can execute the subword segmentation method, or execute the model training method.
根据本公开的另一方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，所述计算机指令用于使所述计算机执行子词切分方法，或者执行模型训练方法。According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to make the computer execute the subword segmentation method, or execute the model training method.
根据本公开的另一方面，提供了一种计算机程序产品，包括计算机程序，其中，所述计算机程序被处理器执行时实现子词切分方法的步骤，或者实现模型训练方法的步骤。According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein, when the computer program is executed by a processor, the steps of the subword segmentation method are realized, or the steps of the model training method are realized.
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.
附图说明Description of drawings
附图用于更好地理解本方案，不构成对本公开的限定。其中：The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:
图1是根据本公开第一实施例的子词切分方法的流程示意图；FIG. 1 is a schematic flow diagram of a subword segmentation method according to a first embodiment of the present disclosure;
图2是根据本公开第二实施例的子词切分方法中的状态路径的示意图；2 is a schematic diagram of a state path in a subword segmentation method according to a second embodiment of the present disclosure;
图3是根据本公开第三实施例的子词切分方法的流程示意图；FIG. 3 is a schematic flow diagram of a subword segmentation method according to a third embodiment of the present disclosure;
图4是根据本公开第四实施例的子词切分方法的流程示意图；FIG. 4 is a schematic flow diagram of a subword segmentation method according to a fourth embodiment of the present disclosure;
图5是根据本公开第五实施例的子词切分方法的流程示意图；Fig. 5 is a schematic flow chart of a subword segmentation method according to a fifth embodiment of the present disclosure;
图6是根据本公开第六实施例的子词切分方法中的状态路径的示意图；FIG. 6 is a schematic diagram of a state path in a subword segmentation method according to a sixth embodiment of the present disclosure;
图7是根据本公开第一实施例的模型训练方法的流程示意图；FIG. 7 is a schematic flowchart of a model training method according to the first embodiment of the present disclosure;
图8是根据本公开第一实施例的子词切分装置的框图；Fig. 8 is a block diagram of a subword segmentation device according to the first embodiment of the present disclosure;
图9是根据本公开第二实施例的模型训练装置的框图；9 is a block diagram of a model training device according to a second embodiment of the present disclosure;
图10是用来实现本公开实施例的子词切分方法和/或模型训练方法的电子设备的框图。Fig. 10 is a block diagram of an electronic device for implementing the subword segmentation method and/or the model training method of the embodiment of the present disclosure.
具体实施方式Detailed ways
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本公开的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
AI(Artificial Intelligence，人工智能)是研究、开发用于模拟、延伸和扩展人的智能的理论、方法、技术及应用系统的一门技术科学。目前，AI技术具有自动化程度高、精确度高、成本低的优点，得到了广泛的应用。AI (Artificial Intelligence) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. At present, AI technology has the advantages of high automation, high precision, and low cost, and has been widely used.
NLU(Natural Language Processing，自然语言处理)是研究能有效地实现自然语言通信的计算机系统，特别是其中的软件系统的一门科学，是计算机科学领域与人工智能领域中的一个重要方向。NLU (Natural Language Processing, Natural Language Processing) is a science that studies computer systems that can effectively realize natural language communication, especially the software system. It is an important direction in the field of computer science and artificial intelligence.
DL(Deep Learning，深度学习)是ML(Machine Learning，机器学习)领域中一个新的研究方向，是学习样本数据的内在规律和表示层次，使得机器能够像人一样具有分析学习能力，能够识别文字、图像和声音等数据的一门科学，广泛应用于语音和图像识别。DL (Deep Learning, deep learning) is a new research direction in the field of ML (Machine Learning, machine learning), which is to learn the internal laws and representation levels of sample data, so that machines can analyze and learn like humans, and can recognize text A science of data such as , images, and sounds, widely used in speech and image recognition.
图1是根据本公开第一实施例的子词切分方法的流程示意图。Fig. 1 is a schematic flowchart of a subword segmentation method according to a first embodiment of the present disclosure.
如图1所示，本公开第一实施例的子词切分方法，包括：As shown in Figure 1, the subword segmentation method of the first embodiment of the present disclosure includes:
S101，获取待切分的文本序列，其中，文本序列包括多个元素。S101. Acquire a text sequence to be segmented, where the text sequence includes multiple elements.
需要说明的是，本公开实施例的子词切分方法的执行主体可为具有数据信息处理能力的硬件设备和/或驱动该硬件设备工作所需必要的软件。可选地，执行主体可包括工作站、服务器，计算机、用户终端及其他智能设备。其中，用户终端包括但不限于手机、电脑、智能语音交互设备、智能家电、车载终端等。It should be noted that the execution subject of the subword segmentation method in the embodiment of the present disclosure may be a hardware device capable of processing data information and/or necessary software required to drive the hardware device to work. Optionally, the execution subject may include workstations, servers, computers, user terminals and other intelligent devices. Wherein, the user terminal includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, and the like.
本公开的实施例中，可获取待切分的文本序列，其中，文本序列包括多个元素。应说明的是，对文本序列的获取方式不做过多限定，比如，可对文本进行编码得到文本序列。对文本的语言类型、是否需要空格进行分隔等均不做过多限定，比如，文本包括但不限于中文、日文、英文、韩文等。对文本序列包括的元素数量不做过多限定，比如，文本序列可包括10个元素。对元素的数据量不做过多限定，比如，元素的数据量可为一个字节。In the embodiments of the present disclosure, a text sequence to be segmented may be obtained, where the text sequence includes multiple elements. It should be noted that the way of obtaining the text sequence is not too limited, for example, the text sequence can be obtained by encoding the text. There are no too many restrictions on the language type of the text, whether spaces need to be separated, etc. For example, the text includes but is not limited to Chinese, Japanese, English, Korean, etc. The number of elements included in the text sequence is not limited too much, for example, the text sequence may include 10 elements. The amount of data of an element is not limited too much, for example, the amount of data of an element may be one byte.
在一种实施方式中，元素可由至少一个数字组成，比如，元素可由0和/或1的数字组成。比如，元素的数据量为一个字节时，元素可为10100101。In one embodiment, an element may consist of at least one number, for example, an element may consist of 0 and/or 1 numbers. For example, when the data amount of the element is one byte, the element may be 10100101.
例如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，其中，n为正整数，其中，元素x1＝11100100，x2＝10111000，x3＝10111000，其余元素的取值这里不再赘述。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, respectively x1 , x2 , x3 to xn , wherein, n is a positive integer, Among them, the elements x1 =11100100, x2 =10111000, x3 =10111000, and the values of the other elements will not be repeated here.
S102，获取每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，其中，候选状态用于表征元素是否为切分边界。S102, obtain the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the state transition probability between the candidate states of any two adjacent elements, where the candidate state is used to represent the element Whether it is a segmentation boundary.
本公开的实施例中，候选状态可自行设定，候选状态用于表征元素是否为切分边界。应说明的是，对候选状态的类别、数量等均不做过多限定。In the embodiments of the present disclosure, the candidate state can be set by itself, and the candidate state is used to indicate whether an element is a segmentation boundary. It should be noted that the category and quantity of the candidate states are not too limited.
本公开的实施例中，可获取每个候选状态下的初始状态概率。比如，初始状态概率可为虚拟起始元素和文本序列中的第一个元素之间的状态转移概率，虚拟起始元素位于所述第一个元素之前。可以理解的是，不同候选状态下的初始状态概率可能不同。In the embodiments of the present disclosure, the initial state probability of each candidate state may be obtained. For example, the initial state probability may be the state transition probability between the virtual start element and the first element in the text sequence, and the virtual start element is located before the first element. It is understandable that the initial state probabilities in different candidate states may be different.
本公开的实施例中，可获取每个候选状态下的每个元素的观测概率。可以理解的是，每个候选状态下的不同元素的观测概率可能不同，不同候选状态下的同一元素的观测概率可能不同。In the embodiments of the present disclosure, the observation probability of each element in each candidate state may be obtained. It can be understood that the observation probabilities of different elements in each candidate state may be different, and the observation probabilities of the same element in different candidate states may be different.
比如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，候选状态包括B、I时，可获取候选状态B下的x1、x2、x3至xn的观测概率，以及候选状态I下的x1、x2、x3至xn的观测概率。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, which are x1 , x2 , x3 to xn , and candidate states include B, I , the observation probabilities of x1 , x2 , x3 to xn in the candidate state B, and the observation probabilities of x1 , x2 , x3 to xn in the candidate state I can be obtained.
本公开的实施例中，可获取任意相邻两个元素的候选状态之间的状态转移概率。比如，状态转移概率可为相邻两个元素中位置靠前的第一元素的候选状态转移至位置靠后的第二元素的候选状态之间的状态转移概率。In the embodiments of the present disclosure, the state transition probabilities between the candidate states of any two adjacent elements may be obtained. For example, the state transition probability may be the state transition probability between the candidate state of the first element at the front among the two adjacent elements transitioning to the candidate state of the second element at the rear.
本公开的实施例中，状态转移概率包括任意两个候选状态之间的状态转移概率，比如，候选状态的数量为2时，状态转移概率的数量为4，例如，候选状态包括B、I时，状态转移概率包括候选状态B转移到候选状态B的状态转移概率、候选状态B转移到候选状态I的状态转移概率、候选状态I转移到候选状态B的状态转移概率、候选状态I转移到候选状态I的状态转移概率。以此类推，候选状态的数量为3时，状态转移概率的数量为9，这里不再赘述。In the embodiments of the present disclosure, the state transition probability includes the state transition probability between any two candidate states. For example, when the number of candidate states is 2, the number of state transition probabilities is 4. For example, when the candidate states include B and I , the state transition probability includes the state transition probability from candidate state B to candidate state B, the state transition probability from candidate state B to candidate state I, the state transition probability from candidate state I to candidate state B, and the state transition probability from candidate state I to candidate state B. The state transition probability of state I. By analogy, when the number of candidate states is 3, the number of state transition probabilities is 9, which will not be repeated here.
例如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，可获取x1的候选状态转移至x2的候选状态的状态转移概率，可获取x2的候选状态转移至x3的候选状态的状态转移概率，这里不再赘述。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, which are x1 , x2 , x3 to xn , and the candidate states of x1 can be obtained The state transition probability of transitioning to the candidate state of x2 can obtain the state transition probability of the candidate state of x2 transitioning to the candidate state of x3 , which will not be repeated here.
S103，根据初始状态概率、观测概率和状态转移概率，从候选状态中确定元素的目标状态。S103. Determine the target state of the element from the candidate states according to the initial state probability, observation probability and state transition probability.
在一种实施方式中，根据初始状态概率、观测概率和状态转移概率，从候选状态中确定元素的目标状态，可包括将初始状态概率、观测概率和状态转移概率输入至Viterbi(维特比)算法，由维特比算法输出元素的目标状态。In one embodiment, according to the initial state probability, observation probability and state transition probability, determining the target state of the element from the candidate states may include inputting the initial state probability, observation probability and state transition probability into the Viterbi (Viterbi) algorithm , the target state of the element output by the Viterbi algorithm.
在一种实施方式中，根据初始状态概率、观测概率和状态转移概率，从候选状态中确定元素的目标状态，可包括根据文本序列中的每个元素的每个候选状态，生成每个第一节点至每个第二节点之间的状态路径，其中，状态路径包括多个节点，节点用于表示候选状态，第一节点用于表示文本序列中的第一个元素的候选状态，第二节点用于表示文本序列中的最后一个元素的候选状态，根据初始状态概率、观测概率和状态转移概率，确定每条状态路径的路径概率，将路径概率最大的状态路径中的候选状态确定为目标状态。In one embodiment, determining the target state of an element from candidate states according to the initial state probability, observation probability and state transition probability may include generating each first A state path between the node and each second node, wherein the state path includes multiple nodes, the nodes are used to represent candidate states, the first node is used to represent the candidate state of the first element in the text sequence, and the second node It is used to represent the candidate state of the last element in the text sequence. According to the initial state probability, observation probability and state transition probability, the path probability of each state path is determined, and the candidate state in the state path with the highest path probability is determined as the target state .
比如，如图2所示，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，每个元素对应两个候选状态B、I，则文本序列A可对应2n个节点，例如，x1对应的节点分别为C11、C12，x2对应的节点分别为C21、C22，x3对应的节点分别为C31、C32，以此类推，xn对应的节点分别为Cn1、Cn2。其中，C11、C21至Cn1分别用于表示候选状态B，C12、C22至Cn2分别用于表示候选状态I，第一节点包括C11、C12，第二节点包括Cn1、Cn2。For example, as shown in Figure 2, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, which are x1 , x2 , x3 to xn , each elements correspond to two candidate states B and I, then the text sequence A can correspond to 2n nodes, for example, the nodes corresponding to x1 are C11 , C12 , and the nodes corresponding to x2 are C21 , C The nodes corresponding to22 and x3 are C31 and C32 respectively, and so on, the nodes corresponding to xn are Cn1 and Cn2 respectively. Among them, C11 , C21 to Cn1 are respectively used to represent candidate state B, C12 , C22 to Cn2 are respectively used to represent candidate state I, and the first node includes C11 , C12 , the second node includes Cn1 and Cn2 .
继续以图2为例，生成每个第一节点(C11、C12)至每个第二节点(Cn1、Cn2)之间的状态路径共有2n条，可根据初始状态概率、观测概率和状态转移概率，确定每条状态路径的路径概率，将路径概率最大的状态路径中的候选状态确定为目标状态。Continuing to take Figure 2 as an example, a total of 2n state paths between each first node (C11 , C12 ) and each second node (Cn1 , Cn2 ) are generated, which can be calculated according to the initial State probability, observation probability and state transition probability determine the path probability of each state path, and determine the candidate state in the state path with the highest path probability as the target state.
S104，根据元素的目标状态，对文本序列进行切分，得到多个子词，其中，子词包括至少一个元素。S104. Segment the text sequence according to the target state of the element to obtain multiple subwords, where the subwords include at least one element.
需要说明的是，对文本序列切分后得到的子词数量不做过多限定，对子词包括的元素数量不做过多限定。比如，子词可包括1个、2个、5个等元素。It should be noted that the number of subwords obtained after the text sequence is segmented is not too limited, and the number of elements included in the subwords is not too limited. For example, a subword may include 1, 2, 5, etc. elements.
在一种实施方式中，根据元素的目标状态，对文本序列进行切分，得到多个子词，可包括根据元素的目标状态确定切分位置，按照切分位置对文本序列进行切分，得到多个子词。可以理解的是，可在每个切分位置对文本序列进行一次切分。In one embodiment, the text sequence is segmented according to the target state of the element to obtain multiple subwords, which may include determining the segmentation position according to the target state of the element, and segmenting the text sequence according to the segmentation position to obtain multiple subwords. a subword. It can be understood that the text sequence can be segmented once at each segment position.
在一种实施方式中，根据元素的目标状态，对文本序列进行切分，得到多个子词，可包括获取文本序列的切分结果，将切分结果中位置连续的至少一个元素进行拼接，得到一个子词。In one embodiment, the text sequence is segmented according to the target state of the element to obtain a plurality of subwords, which may include obtaining the segmentation result of the text sequence, and splicing at least one element with consecutive positions in the segmentation result to obtain a subword.
比如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，其中，元素x1＝11100100，x2＝10111000，若根据x1、x2、x3至xn的目标状态确定的切分位置包括x3，则可将切分结果中位置连续的x1、x2进行拼接，得到子词d＝1110010010111000。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, respectively x1 , x2 , x3 to xn , where element x1 =11100100 , x2 =10111000, if the segmentation position determined according to the target states of x1 , x2 , x3 to xn includes x3 , then the consecutive positions of x1 and x2 in the segmentation results can be spliced to obtain Subword d=1110010010111000.
综上，根据本公开实施例的子词切分方法，可根据每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，从候选状态中确定元素的目标状态，并根据元素的目标状态，对文本序列进行切分，得到多个子词，其中，候选状态用于表征元素是否为切分边界。由此，可考虑到元素的上下文和相邻元素之间的转移关系实现子词切分，可消除相关子词切分技术中相邻元素之间的独立性假设，适用于任意语言或领域的文本序列的子词切分，泛化性较好。In summary, according to the subword segmentation method of the embodiment of the present disclosure, the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the The target state of the element is determined from the candidate state, and the text sequence is segmented according to the target state of the element to obtain multiple subwords. The candidate state is used to represent whether the element is a segmentation boundary. As a result, subword segmentation can be realized considering the context of elements and the transfer relationship between adjacent elements, which can eliminate the independence assumption between adjacent elements in related subword segmentation technology, and is applicable to any language or field. Subword segmentation of text sequences has good generalization.
图3是根据本公开第三实施例的子词切分方法的流程示意图。Fig. 3 is a schematic flowchart of a subword segmentation method according to a third embodiment of the present disclosure.
如图3所示，本公开第三实施例的子词切分方法，包括：As shown in Figure 3, the subword segmentation method of the third embodiment of the present disclosure includes:
S301，获取待切分的文本序列，其中，文本序列包括多个元素。S301. Acquire a text sequence to be segmented, where the text sequence includes multiple elements.
在一种实施方式中，获取待切分的文本序列，可包括获取文本，按照通用编码策略对文本进行编码，得到编码文本，根据元素的数据量对编码文本进行切分，得到多个元素，根据多个元素，生成文本序列。由此，该方法可采用通用编码策略生成文本序列，适用于任意语言类型、是否需要空格进行分隔等情况，泛化性较好。In one embodiment, obtaining the text sequence to be segmented may include obtaining the text, encoding the text according to a general encoding strategy to obtain the encoded text, and segmenting the encoded text according to the data amount of the elements to obtain multiple elements, From multiple elements, generate a sequence of text. Therefore, this method can generate text sequences with a general coding strategy, which is suitable for any language type, whether spaces are required for separation, etc., and has good generalization.
需要说明的是，通用编码策略指的是适用于任意语言类型、是否需要空格进行分隔等情况。对通用编码策略的类别不做过多限定，比如，通用编码策略可为字符级别的编码策略，有助于避免产生未登录词的问题。比如，通用编码策略可为UTF-8(8-bit UnicodeTransformation Format，8比特通用编码转化格式)，此时字符集仅包括256个码点，可有效避免词表的数据量过大的问题，可保证词表的数据量适中。It should be noted that the general coding strategy refers to situations such as being applicable to any language type and whether spaces are required for separation. The category of the general encoding strategy is not limited too much. For example, the general encoding strategy can be a character-level encoding strategy, which helps to avoid the problem of unregistered words. For example, the general encoding strategy can be UTF-8 (8-bit Unicode Transformation Format, 8-bit universal encoding transformation format). At this time, the character set only includes 256 code points, which can effectively avoid the problem of excessive data volume in the vocabulary. Ensure that the amount of data in the vocabulary is moderate.
需要说明的是，对元素的数据量不做过多限定，比如，元素的数据量可为一个字节，此时文本序列为字节序列。It should be noted that the data volume of an element is not too limited. For example, the data volume of an element may be one byte, and the text sequence is a byte sequence at this time.
S302，将文本序列输入至训练好的概率图模型，由概率图模型对文本序列进行状态预测，以输出每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，其中，候选状态用于表征元素是否为切分边界。S302, input the text sequence into the trained probability graph model, and use the probability graph model to predict the state of the text sequence to output the initial state probability in each candidate state, the observation probability of each element in each candidate state, The state transition probability between the candidate states of any two adjacent elements, where the candidate state is used to represent whether the element is a segmentation boundary.
本公开的实施例中，概率图模型可自行设定，概率图模型的输入为文本序列，输出为每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率。应说明的是，对概率图模型的类别不做过多限定，比如，概率图模型包括但不限于HMM(Hidden Markov Model，隐马尔可夫模型)、CRF(Conditional Random Fields，条件随机场)等。In the embodiments of the present disclosure, the probabilistic graphical model can be set by itself. The input of the probabilistic graphical model is a text sequence, and the output is the initial state probability in each candidate state, the observation probability of each element in each candidate state, and any The state transition probability between the candidate states of two adjacent elements. It should be noted that the category of probabilistic graphical models is not limited too much. For example, probabilistic graphical models include but are not limited to HMM (Hidden Markov Model, hidden Markov model), CRF (Conditional Random Fields, conditional random field), etc. .
S303，根据初始状态概率、观测概率和状态转移概率，从候选状态中确定元素的目标状态。S303. Determine the target state of the element from the candidate states according to the initial state probability, observation probability and state transition probability.
步骤S303的相关内容可参见上述实施例，这里不再赘述。Relevant content of step S303 may refer to the foregoing embodiments, and details are not repeated here.
S304，将目标状态为第一候选状态的元素确定为切分边界。S304. Determine an element whose target state is the first candidate state as a segmentation boundary.
S305，按照切分边界对文本序列进行切分，得到多个子词。S305. Segment the text sequence according to the segmentation boundary to obtain multiple subwords.
本公开的实施例中，候选状态包括用于表征元素为切分边界的第一候选状态，以及用于表征元素为非切分边界的第二候选状态。In the embodiments of the present disclosure, the candidate states include a first candidate state for representing an element as a segmentation boundary, and a second candidate state for representing an element as a non-segmentation boundary.
应说明的是，对切分边界的数量不做过多限定。It should be noted that the number of segmentation boundaries is not limited too much.
在一种实施方式中，将目标状态为第一候选状态的元素确定为切分边界，可包括将目标状态为第一候选状态的元素的前端确定为切分边界。In one embodiment, determining the element whose target state is the first candidate state as a segmentation boundary may include determining the front end of the element whose target state is the first candidate state as a segmentation boundary.
比如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，其中，元素x1＝11100100，x2＝10111000，x3＝10111000，x1、x2、x3对应的目标状态分别为第一候选状态B、第二候选状态I、第一候选状态B，则可将x1、x3的前端确定为切分边界，并按照上述切分边界对文本序列A进行切分，得到多个子词。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, respectively x1 , x2 , x3 to xn , where element x1 =11100100 , x2 =10111000, x3 =10111000, the target states corresponding to x1 , x2 , and x3 are the first candidate state B, the second candidate state I, and the first candidate state B, then x1 , x The front end of3 is determined as the segmentation boundary, and the text sequence A is segmented according to the above segmentation boundary to obtain multiple subwords.
在一种实施方式中，得到多个子词之后，还包括将得到的多个子词添加到词表中，对添加后的词表进行剪枝，得到目标词表。应说明的是，对剪枝方式不做过多限定，比如，可采用Unigram方法对词表进行剪枝。由此，该方法中可将得到的多个子词添加到词表中，可基于文本序列的子词切分结果实现词表的扩充，还可对添加后的词表进行剪枝，得到目标词表，可避免目标词表的数据量过大。In one embodiment, after obtaining the plurality of subwords, adding the obtained plurality of subwords to a vocabulary, and pruning the added vocabulary to obtain a target vocabulary. It should be noted that the pruning method is not too limited, for example, the Unigram method can be used to prune the vocabulary. Therefore, in this method, the obtained multiple subwords can be added to the vocabulary, and the vocabulary can be expanded based on the subword segmentation results of the text sequence, and the added vocabulary can also be pruned to obtain the target word table, which can avoid the data volume of the target vocabulary from being too large.
综上，根据本公开实施例的子词切分方法，可基于概率图模型实现初始状态概率、观测概率和状态转移概率的自动获取，还可将目标状态为第一候选状态的元素确定为切分边界，按照切分边界对文本序列进行切分，得到多个子词。In summary, according to the subword segmentation method of the embodiment of the present disclosure, the automatic acquisition of the initial state probability, observation probability and state transition probability can be realized based on the probability graph model, and the element whose target state is the first candidate state can also be determined as the segmentation The segmentation boundary is used to segment the text sequence according to the segmentation boundary to obtain multiple subwords.
图4是根据本公开第四实施例的子词切分方法的流程示意图。Fig. 4 is a schematic flowchart of a subword segmentation method according to a fourth embodiment of the present disclosure.
如图4所示，本公开第四实施例的子词切分方法，包括：As shown in Figure 4, the subword segmentation method of the fourth embodiment of the present disclosure includes:
S401，获取待切分的文本序列，其中，文本序列包括多个元素。S401. Acquire a text sequence to be segmented, where the text sequence includes multiple elements.
S402，获取每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，其中，候选状态用于表征元素是否为切分边界。S402, obtain the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the state transition probability between the candidate states of any two adjacent elements, where the candidate state is used to represent the element Whether it is a segmentation boundary.
步骤S401-S402的相关内容可参见上述实施例，这里不再赘述。Relevant content of steps S401-S402 may refer to the foregoing embodiments, and details are not repeated here.
S403，将文本序列中的第一个元素的目标状态确定为第一候选状态。S403. Determine the target state of the first element in the text sequence as the first candidate state.
S404，从文本序列中的第二个元素开始，将第二个元素确定为待检测元素，并根据初始状态概率、文本序列中位于待检测元素之前的元素以及待检测元素对应的观测概率和状态转移概率，从候选状态中确定待检测元素的目标状态。S404, starting from the second element in the text sequence, determining the second element as the element to be detected, and according to the initial state probability, the element before the element to be detected in the text sequence, and the observation probability and state corresponding to the element to be detected Transition probability, which determines the target state of the element to be detected from the candidate states.
S405，将待检测元素的下一个元素更新为待检测元素，直至遍历到文本序列中的最后一个元素。S405. Update the next element of the element to be detected to be the element to be detected until the last element in the text sequence is traversed.
本公开的实施例中，候选状态包括用于表征元素为切分边界的第一候选状态，以及用于表征元素为非切分边界的第二候选状态。In the embodiments of the present disclosure, the candidate states include a first candidate state for representing an element as a segmentation boundary, and a second candidate state for representing an element as a non-segmentation boundary.
本公开的实施例中，可从文本序列中的第二个元素开始，依次向后遍历文本序列中的元素，每遍历一次可将当前遍历到的元素确定为待检测元素，并确定待检测元素的目标状态，直至遍历到文本序列中的最后一个元素。In the embodiment of the present disclosure, starting from the second element in the text sequence, the elements in the text sequence can be traversed backwards one by one, and the currently traversed element can be determined as the element to be detected each time, and the element to be detected can be determined The target state of , until the last element in the text sequence is traversed.
例如，文本序列A＝{x1,x2,x3……xn}，文本序列A包括n个元素，分别为x1、x2、x3至xn，候选状态包括用于表征元素为切分边界的第一候选状态B，以及用于表征元素为非切分边界的第二候选状态I。For example, text sequence A={x1 , x2 , x3 ... xn }, text sequence A includes n elements, respectively x1 , x2 , x3 to xn , and the candidate states include elements used to represent The first candidate state B is a segmentation boundary, and the second candidate state I is a characterizing element that is a non-segmentation boundary.
可将x1的目标状态确定为第一候选状态B，并从x2开始，将x2确定为待检测元素，并根据初始状态概率，以及x1、x2对应的观测概率和状态转移概率，从候选状态中确定x2的目标状态。The target state of x1 can be determined as the first candidate state B, and starting from x2 , determine x2 as the element to be detected, and according to the initial state probability, and the corresponding observation probability and state transition probability of x1 and x2 , to determine the target state ofx2 from the candidate states.
进一步地，可将x3更新为待检测元素，根据初始状态概率，以及x1、x2、x3对应的观测概率和状态转移概率，从候选状态中确定x3的目标状态。Further, x3 can be updated as the element to be detected, and the target state of x3 can be determined from the candidate states according to the initial state probability, and the corresponding observation probability and state transition probability of x1 , x2 , and x3 .
以此类推，可遍历到xn，将xn更新为待检测元素，根据初始状态概率，以及x1、x2至xn-1对应的观测概率和状态转移概率，从候选状态中确定xn的目标状态。By analogy, it is possible to traverse to xn , update xn as the element to be detected, and determine x from the candidate states according to the initial state probability, and the observation probability and state transition probability corresponding to x1 , x2 to xn-1 The target state ofn .
S406，根据元素的目标状态，对文本序列进行切分，得到多个子词，其中，子词包括至少一个元素。S406. Segment the text sequence according to the target state of the element to obtain multiple subwords, where the subwords include at least one element.
步骤S406的相关内容可参见上述实施例，这里不再赘述。Relevant content of step S406 may refer to the foregoing embodiments, and details are not repeated here.
综上，根据本公开实施例的子词切分方法，可将文本序列中的第一个元素的目标状态确定为第一候选状态，从文本序列中的第二个元素开始，将第二个元素确定为待检测元素，并根据初始状态概率、文本序列中位于待检测元素之前的元素以及待检测元素对应的观测概率和状态转移概率，从候选状态中确定待检测元素的目标状态，将待检测元素的下一个元素更新为待检测元素，直至遍历到文本序列中的最后一个元素。由此，可考虑到元素的上文和相邻元素之间的转移关系，以确定元素的目标状态。In summary, according to the subword segmentation method of the embodiment of the present disclosure, the target state of the first element in the text sequence can be determined as the first candidate state, starting from the second element in the text sequence, the second The element is determined as the element to be detected, and the target state of the element to be detected is determined from the candidate states according to the initial state probability, the element before the element to be detected in the text sequence, and the observation probability and state transition probability corresponding to the element to be detected. The next element of the detected element is updated as the element to be detected until the last element in the text sequence is traversed. Thus, the element's context and the transition relationship between adjacent elements can be considered to determine the target state of the element.
图5是根据本公开第五实施例的子词切分方法的流程示意图。Fig. 5 is a schematic flowchart of a subword segmentation method according to a fifth embodiment of the present disclosure.
如图5所示，本公开第五实施例的子词切分方法，包括：As shown in Figure 5, the subword segmentation method of the fifth embodiment of the present disclosure includes:
S501，获取待切分的文本序列，其中，文本序列包括多个元素。S501. Acquire a text sequence to be segmented, where the text sequence includes multiple elements.
S502，获取每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，其中，候选状态用于表征元素是否为切分边界。S502, obtain the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the state transition probability between the candidate states of any two adjacent elements, where the candidate state is used to represent the element Whether it is a segmentation boundary.
S503，将文本序列中的第一个元素的目标状态确定为第一候选状态。S503. Determine the target state of the first element in the text sequence as the first candidate state.
S504，从文本序列中的第二个元素开始，将第二个元素确定为待检测元素。S504. Starting from the second element in the text sequence, determine the second element as the element to be detected.
步骤S501-S504的相关内容可参见上述实施例，这里不再赘述。Relevant content of steps S501-S504 may refer to the above-mentioned embodiments, and details are not repeated here.
S505，获取包括位置连续的多个目标元素的文本序列单元，其中，文本序列单元的第一个目标元素的目标状态为第一候选状态，第二个目标元素至倒数第二个目标元素的目标状态为第二候选状态，最后一个目标元素为待检测元素。S505. Obtain a text sequence unit including multiple target elements with continuous positions, wherein the target state of the first target element of the text sequence unit is the first candidate state, and the targets of the second target element to the penultimate target element The state is the second candidate state, and the last target element is the element to be detected.
本公开的实施例中，文本序列单元包括位置连续的多个目标元素，其中，第一个目标元素的目标状态为第一候选状态，第二个目标元素至倒数第二个目标元素的目标状态为第二候选状态，即第一个目标元素为位于待检测元素之前，距离待检测元素最近且目标状态为第一候选状态的元素，文本序列单元中第一个目标元素以外，且位于待检测元素之前的目标元素的目标状态均为第二候选状态。In the embodiment of the present disclosure, the text sequence unit includes multiple target elements with consecutive positions, wherein the target state of the first target element is the first candidate state, and the target states of the second target element to the penultimate target element is the second candidate state, that is, the first target element is located before the element to be detected, the element closest to the element to be detected, and the target state is the first candidate state, outside the first target element in the text sequence unit, and located in the element to be detected The target states of the target elements preceding the element are all second candidate states.
需要说明的是，对文本序列单元包括的目标元素的数量不做过多限定。在一种实施方式中，文本序列单元至少包括两个目标元素，比如，文本序列单元包括第一个目标元素和待检测元素，其中，第一个目标元素的目标状态为第一候选状态。It should be noted that there is no excessive limitation on the number of target elements included in the text sequence unit. In one embodiment, the text sequence unit includes at least two target elements. For example, the text sequence unit includes a first target element and an element to be detected, wherein the target state of the first target element is the first candidate state.
比如，文本序列A＝{x1,x2,x3……xn}，x1、x2的目标状态分别为第一候选状态B、第二候选状态I，待检测元素为x3，则可获取文本序列单元E＝{x1,x2,x3}；或者，x1、x2的目标状态均为第一候选状态B，待检测元素为x3，则可获取文本序列单元E＝{x2,x3}。For example, text sequence A={x1 , x2 , x3 ... xn }, the target states of x1 and x2 are the first candidate state B and the second candidate state I respectively, and the element to be detected is x3 , Then the text sequence unit E={x1 ,x2 ,x3 } can be obtained; or, the target states of x1 and x2 are both the first candidate state B, and the element to be detected is x3 , then the text sequence unit can be obtained E={x2 ,x3 }.
S506，根据文本序列单元中的第一个目标元素至倒数第二个目标元素的目标状态，以及待检测元素的每个候选状态，生成状态路径。S506. Generate a state path according to the target states of the first target element to the penultimate target element in the text sequence unit and each candidate state of the element to be detected.
本公开的实施例中，生成的状态路径的数量为2。In the embodiment of the present disclosure, the number of generated state paths is two.
比如，如图6所示，文本序列单元E＝{x1,x2,x3,x4}，文本序列单元E包括4个元素，分别为x1、x2、x3、x4，其中，x1、x2、x3的目标状态分别为第一候选状态B、第二候选状态I、第二候选状态I，x4对应两个候选状态B、I，则文本序列单元E可对应5个节点，例如，x1对应的节点为C11，x2对应的节点为C22，x3对应的节点为C32，x4对应的节点分别为C41、C42。其中，C11、C41分别用于表示候选状态B，C22、C32、C42分别用于表示候选状态I。For example, as shown in Figure 6, the text sequence unit E={x1 , x2 , x3 , x4 }, the text sequence unit E includes 4 elements, namely x1 , x2 , x3 , x4 , Among them, the target states of x1 , x2 , and x3 are respectively the first candidate state B, the second candidate state I, and the second candidate state I, and x4 corresponds to two candidate states B, I, then the text sequence unit E can be Corresponding to 5 nodes, for example, the node corresponding to x1 is C11 , the node corresponding to x2 is C22 , the node corresponding to x3 is C32 , and the nodes corresponding to x4 are C41 and C42 . Among them, C11 and C41 are respectively used to represent the candidate state B, and C22 , C32 , and C42 are respectively used to represent the candidate state I.
继续以图6为例，生成的状态路径包括L1、L2(图中未示出)，其中，状态路径L1包括节点C11、C22、C32、C41，状态路径L2包括节点C11、C22、C32、C42。Continuing to take Figure 6 as an example, the generated state path includes L1 and L2 (not shown in the figure), wherein, the state path L1 includes nodes C11 , C22 , C32 , and C41 , and the state Path L2 includes nodes C11 , C22 , C32 , C42 .
S507，根据初始状态概率、目标元素对应的观测概率和状态转移概率，确定状态路径的路径概率。S507. Determine the path probability of the state path according to the initial state probability, the observation probability corresponding to the target element, and the state transition probability.
步骤S507的相关内容可参见上述实施例，这里不再赘述。Relevant content of step S507 may refer to the foregoing embodiments, and details are not repeated here.
S508，获取路径概率最大的目标状态路径，并将目标状态路径中的待检测元素的候选状态确定为待检测元素的目标状态。S508. Acquire a target state path with the highest path probability, and determine a candidate state of the element to be detected in the target state path as the target state of the element to be detected.
继续以图6为例，可确定状态路径L1、L2的路径概率，若路径概率最大的目标状态路径为L1，则可将目标状态路径L1中的x4的候选状态(即为第一候选状态B)确定为x4的目标状态，即可将x4的目标状态确定为第一候选状态B。Continuing to take Fig. 6 as an example, the path probabilities of the state paths L1 and L2 can be determined. If the target state path with the highest path probability is L1 , then the candidate state of x4 in the target state path L1 (that is, The first candidate state B) is determined as the target state of x4 , that is, the target state of x4 can be determined as the first candidate state B.
或者，若路径概率最大的目标状态路径为L2，则可将目标状态路径L2中的x4的候选状态(即为第二候选状态I)确定为x4的目标状态，即可将x4的目标状态确定为第二候选状态I。Alternatively, if the target state path with the highest path probability is L2 , then the candidate state of x4 in the target state path L2 (that is, the second candidate state I) can be determined as the target state of x4 , that is, x The target state of4 is determined as the second candidate state I.
在一种实施方式中，在目标状态路径的路径概率小于设定阈值的情况下，表明此时目标状态路径的路径概率较小，可将待检测元素的目标状态确定为第一候选状态，即在目标状态路径的路径概率小于设定阈值的情况下，可直接将待检测元素的目标状态确定为第一候选状态，可避免子词的数据量过大的情况。In one embodiment, when the path probability of the target state path is less than the set threshold, it indicates that the path probability of the target state path is relatively small at this time, and the target state of the element to be detected can be determined as the first candidate state, namely When the path probability of the target state path is less than the set threshold, the target state of the element to be detected can be directly determined as the first candidate state, which can avoid the situation that the data amount of the subword is too large.
S509，将待检测元素的下一个元素更新为待检测元素，直至遍历到文本序列中的最后一个元素。S509. Update the next element of the element to be detected to be the element to be detected until the last element in the text sequence is traversed.
S510，根据元素的目标状态，对文本序列进行切分，得到多个子词，其中，子词包括至少一个元素。S510. Segment the text sequence according to the target state of the element to obtain multiple subwords, where the subwords include at least one element.
步骤S509-S510的相关内容可参见上述实施例，这里不再赘述。Relevant content of steps S509-S510 may refer to the foregoing embodiments, and details are not repeated here.
综上，根据本公开实施例的子词切分方法，可获取文本序列单元，文本序列单元的第一个目标元素的目标状态为第一候选状态，第二个目标元素至倒数第二个目标元素的目标状态为第二候选状态，最后一个目标元素为待检测元素，基于文本序列单元中的目标元素的目标状态和候选状态，生成状态路径，进而确定待检测元素的目标状态。To sum up, according to the subword segmentation method of the embodiment of the present disclosure, the text sequence unit can be obtained, the target state of the first target element of the text sequence unit is the first candidate state, and the second target element to the penultimate target The target state of the element is the second candidate state, and the last target element is the element to be detected. Based on the target state and the candidate state of the target element in the text sequence unit, a state path is generated to determine the target state of the element to be detected.
图7是根据本公开第一实施例的模型训练方法的流程示意图。Fig. 7 is a schematic flowchart of a model training method according to the first embodiment of the present disclosure.
如图7所示，本公开第一实施例的模型训练方法，包括：As shown in Figure 7, the model training method of the first embodiment of the present disclosure includes:
S701，获取样本文本序列，其中，样本文本序列包括多个样本元素。S701. Acquire a sample text sequence, where the sample text sequence includes a plurality of sample elements.
需要说明的是，本公开实施例的子词切分方法的执行主体可为具有数据信息处理能力的硬件设备和/或驱动该硬件设备工作所需必要的软件。可选地，执行主体可包括工作站、服务器，计算机、用户终端及其他智能设备。其中，用户终端包括但不限于手机、电脑、智能语音交互设备、智能家电、车载终端等。It should be noted that the execution subject of the subword segmentation method in the embodiment of the present disclosure may be a hardware device capable of processing data information and/or necessary software required to drive the hardware device to work. Optionally, the execution subject may include workstations, servers, computers, user terminals and other intelligent devices. Wherein, the user terminal includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, and the like.
本公开的实施例中，可获取大量的样本文本序列。应说明的是，对样本文本序列的获取方式不做过多限定，比如，可对样本文本进行编码得到样本文本序列。对样本文本的语言类型、是否需要空格进行分隔等均不做过多限定，比如，样本文本包括但不限于中文、日文、英文、韩文等。In the embodiments of the present disclosure, a large number of sample text sequences can be obtained. It should be noted that the manner of obtaining the sample text sequence is not too limited, for example, the sample text sequence may be obtained by encoding the sample text. There are no restrictions on the language type of the sample text, whether spaces are required for separation, etc. For example, the sample text includes but is not limited to Chinese, Japanese, English, Korean, etc.
在一种实施方式中，获取样本文本序列，可包括获取样本文本，按照通用编码策略对样本文本进行编码，得到编码样本文本，根据元素的数据量对编码样本文本进行切分，得到多个样本元素，根据多个样本元素，生成样本文本序列。由此，该方法可采用通用编码策略生成样本文本序列，适用于任意语言类型、是否需要空格进行分隔等情况，泛化性较好。In one embodiment, obtaining the sample text sequence may include obtaining the sample text, encoding the sample text according to a general coding strategy to obtain the coded sample text, and segmenting the coded sample text according to the data volume of the elements to obtain multiple samples element to generate a sample text sequence based on multiple sample elements. Therefore, this method can use a general coding strategy to generate a sample text sequence, which is suitable for any language type, whether spaces are required for separation, etc., and has good generalization.
步骤S701的相关内容可参见上述实施例，这里不再赘述。Relevant content of step S701 may refer to the foregoing embodiments, and details are not repeated here.
S702，根据样本文本序列训练概率图模型，对概率图模型的模型参数进行更新，其中，概率图模型用于输出每个候选状态下的训练初始状态概率、每个候选状态下的每个样本元素的训练观测概率、任意相邻两个样本元素的候选状态之间的训练状态转移概率，其中，候选状态用于表征样本元素是否为切分边界。S702, train the probability graphical model according to the sample text sequence, and update the model parameters of the probability graphical model, wherein the probability graphical model is used to output the training initial state probability in each candidate state, each sample element in each candidate state The training observation probability of , the training state transition probability between the candidate states of any two adjacent sample elements, where the candidate states are used to represent whether the sample elements are segmentation boundaries.
需要说明的是，对概率图模型的训练策略不做过多限定，比如，训练策略可为EM(Expectation-maximization，期望最大化)算法。It should be noted that the training strategy of the probability graphical model is not limited too much, for example, the training strategy may be an EM (Expectation-maximization, expectation maximization) algorithm.
本公开的实施例中，概率图模型可自行设定，训练过程中概率图模型的输入为样本文本序列，输出为每个候选状态下的训练初始状态概率、每个候选状态下的每个样本元素的训练观测概率、任意相邻两个样本元素的候选状态之间的训练状态转移概率。应说明的是，对概率图模型的类别不做过多限定，比如，概率图模型包括但不限于HMM(HiddenMarkov Model，隐马尔可夫模型)、CRF(Conditional Random Fields，条件随机场)等。In the embodiments of the present disclosure, the probability graph model can be set by itself. During the training process, the input of the probability graph model is a sample text sequence, and the output is the training initial state probability in each candidate state, each sample in each candidate state The training observation probability of the element, the training state transition probability between the candidate states of any two adjacent sample elements. It should be noted that the category of probabilistic graphical models is not limited too much. For example, probabilistic graphical models include but are not limited to HMM (Hidden Markov Model, hidden Markov model), CRF (Conditional Random Fields, conditional random field), etc.
在一种实施方式中，在概率图模型为HMM的情况下，模型参数包括初始状态概率π、状态转移概率矩阵A、观测概率矩阵B。In one embodiment, when the probability graphical model is HMM, the model parameters include initial state probability π, state transition probability matrix A, and observation probability matrix B.
在一种实施方式中，在概率图模型为CRF的情况下，可根据样本文本序列，以及样本文本序列中每个样本元素的样本目标状态，训练概率图模型，对概率图模型的模型参数进行更新。In one embodiment, when the probability graphical model is CRF, the probability graphical model can be trained according to the sample text sequence and the sample target state of each sample element in the sample text sequence, and the model parameters of the probability graphical model can be renew.
S703，在未满足模型训练结束条件的情况下，返回采用下一个样本文本序列继续对更新后的概率图模型进行训练，直至满足模型训练结束条件，生成训练好的概率图模型。S703. If the end condition of the model training is not satisfied, return to the next sample text sequence to continue training the updated probability graphical model until the end condition of the model training is satisfied, and generate a trained probability graphical model.
本公开的实施例中，模型训练结束条件可自行设定，这里不做过多限定，比如，模型训练结束条件包括但不限于模型训练次数达到设定次数，模型精度达到设定精度等。应说明的是，对设定次数、设定精度均不做过多限定。In the embodiments of the present disclosure, the model training end condition can be set by itself, and there are no too many limitations here. For example, the model training end condition includes but is not limited to the number of model training times reaching the set number, and the model accuracy reaching the set accuracy. It should be noted that neither the setting times nor the setting accuracy are too limited.
综上，根据本公开实施例的模型训练方法，可基于样本文本序列对概率图模型进行训练，概率图模型用于输出每个候选状态下的训练初始状态概率、每个候选状态下的每个样本元素的训练观测概率、任意相邻两个样本元素的候选状态之间的训练状态转移概率，其中，候选状态用于表征样本元素是否为切分边界，可将概率图模型应用于子词切分场景。To sum up, according to the model training method of the embodiment of the present disclosure, the probability graph model can be trained based on the sample text sequence, and the probability graph model is used to output the training initial state probability in each candidate state, each candidate state in each The training observation probability of the sample element, the training state transition probability between the candidate states of any two adjacent sample elements, where the candidate state is used to represent whether the sample element is a segmentation boundary, and the probability graph model can be applied to the subword segmentation sub-scene.
本公开的技术方案中，所涉及的用户个人信息的收集、存储、使用、加工、传输、提供和公开等处理，均符合相关法律法规的规定，且不违背公序良俗。In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.
根据本公开的实施例，本公开还提供了一种子词切分装置，用于实现上述的子词切分方法。According to an embodiment of the present disclosure, the present disclosure also provides a subword segmentation device for implementing the above subword segmentation method.
图8是根据本公开第一实施例的子词切分装置的框图。Fig. 8 is a block diagram of a subword segmentation device according to the first embodiment of the present disclosure.
如图8所示，本公开实施例的子词切分装置800，包括：第一获取模块801、第二获取模块802、确定模块803和切分模块804。As shown in FIG. 8 , the
第一获取模块801用于获取待切分的文本序列，其中，所述文本序列包括多个元素；The first acquiring
第二获取模块802用于获取每个候选状态下的初始状态概率、每个所述候选状态下的每个所述元素的观测概率、任意相邻两个所述元素的所述候选状态之间的状态转移概率，其中，所述候选状态用于表征所述元素是否为切分边界；The
确定模块803用于根据所述初始状态概率、所述观测概率和所述状态转移概率，从所述候选状态中确定所述元素的目标状态；The determining
切分模块804用于根据所述元素的所述目标状态，对所述文本序列进行切分，得到多个子词，其中，所述子词包括至少一个所述元素。The
在本公开的一个实施例中，所述第二获取模块802还用于：将所述文本序列输入至训练好的概率图模型，由所述概率图模型对所述文本序列进行状态预测，以输出所述初始状态概率、所述观测概率和所述状态转移概率。In an embodiment of the present disclosure, the
在本公开的一个实施例中，所述候选状态包括用于表征所述元素为切分边界的第一候选状态，以及用于表征所述元素为非切分边界的第二候选状态。In an embodiment of the present disclosure, the candidate states include a first candidate state for characterizing the element as a segmentation boundary, and a second candidate state for characterizing the element as a non-segmentation boundary.
在本公开的一个实施例中，所述切分模块804还用于：将所述目标状态为所述第一候选状态的所述元素确定为切分边界；按照所述切分边界对所述文本序列进行切分，得到多个所述子词。In an embodiment of the present disclosure, the
在本公开的一个实施例中，所述确定模块803还用于：将所述文本序列中的第一个元素的所述目标状态确定为所述第一候选状态；从所述文本序列中的第二个元素开始，将所述第二个元素确定为待检测元素，并根据所述初始状态概率、所述文本序列中位于所述待检测元素之前的所述元素以及所述待检测元素对应的所述观测概率和所述状态转移概率，从所述候选状态中确定所述待检测元素的所述目标状态；将所述待检测元素的下一个元素更新为所述待检测元素，直至遍历到所述文本序列中的最后一个元素。In an embodiment of the present disclosure, the
在本公开的一个实施例中，所述确定模块803还用于：获取包括位置连续的多个目标元素的文本序列单元，其中，所述文本序列单元的第一个所述目标元素的所述目标状态为所述第一候选状态，第二个所述目标元素至倒数第二个所述目标元素的所述目标状态为所述第二候选状态，最后一个所述目标元素为所述待检测元素；根据所述文本序列单元中的第一个所述目标元素至倒数第二个所述目标元素的所述目标状态，以及所述待检测元素的每个所述候选状态，生成状态路径；根据所述初始状态概率、所述目标元素对应的所述观测概率和所述状态转移概率，确定所述状态路径的路径概率；获取所述路径概率最大的目标状态路径，并将所述目标状态路径中的所述待检测元素的所述候选状态确定为所述待检测元素的所述目标状态。In an embodiment of the present disclosure, the determining
在本公开的一个实施例中，所述确定模块803还用于：在所述目标状态路径的所述路径概率小于设定阈值的情况下，将所述待检测元素的所述目标状态确定为所述第一候选状态。In an embodiment of the present disclosure, the
在本公开的一个实施例中，所述第一获取模块801还用于：获取文本；按照通用编码策略对所述文本进行编码，得到编码文本；根据所述元素的数据量对所述编码文本进行切分，得到多个所述元素；根据多个所述元素，生成所述文本序列。In an embodiment of the present disclosure, the first acquiring
在本公开的一个实施例中，所述数据量为一个字节。In an embodiment of the present disclosure, the data volume is one byte.
在本公开的一个实施例中，所述切分模块804还用于：将得到的多个所述子词添加到词表中；对添加后的所述词表进行剪枝，得到目标词表。In an embodiment of the present disclosure, the
综上，本公开实施例的子词切分装置，可根据每个候选状态下的初始状态概率、每个候选状态下的每个元素的观测概率、任意相邻两个元素的候选状态之间的状态转移概率，从候选状态中确定元素的目标状态，并根据元素的目标状态，对文本序列进行切分，得到多个子词，其中，候选状态用于表征元素是否为切分边界。由此，可考虑到元素的上下文和相邻元素之间的转移关系实现子词切分，可消除相关子词切分技术中相邻元素之间的独立性假设，适用于任意语言或领域的文本序列的子词切分，泛化性较好。To sum up, the subword segmentation device in the embodiment of the present disclosure can be based on the initial state probability in each candidate state, the observation probability of each element in each candidate state, and the distance between the candidate states of any two adjacent elements. Determine the target state of the element from the candidate state, and segment the text sequence according to the target state of the element to obtain multiple subwords, where the candidate state is used to represent whether the element is a segmentation boundary. As a result, subword segmentation can be realized considering the context of elements and the transfer relationship between adjacent elements, which can eliminate the independence assumption between adjacent elements in related subword segmentation technology, and is applicable to any language or field. Subword segmentation of text sequences has good generalization.
根据本公开的实施例，本公开还提供了一种模型训练装置，用于实现上述的模型训练方法。According to an embodiment of the present disclosure, the present disclosure also provides a model training device, which is used to implement the above-mentioned model training method.
图9是根据本公开第一实施例的模型训练装置的框图。FIG. 9 is a block diagram of a model training device according to the first embodiment of the present disclosure.
如图9所示，本公开实施例的模型训练装置900，包括：获取模块901和训练模块902。As shown in FIG. 9 , the
获取模块901用于获取样本文本序列，其中，所述样本文本序列包括多个样本元素；The obtaining
训练模块902用于根据所述样本文本序列训练概率图模型，对所述概率图模型的模型参数进行更新，其中，所述概率图模型用于输出每个候选状态下的训练初始状态概率、每个所述候选状态下的每个所述样本元素的训练观测概率、任意相邻两个所述样本元素的所述候选状态之间的训练状态转移概率，其中，所述候选状态用于表征所述样本元素是否为切分边界；The
所述训练模块902还用于在未满足模型训练结束条件的情况下，返回采用下一个样本文本序列继续对更新后的所述概率图模型进行训练，直至满足所述模型训练结束条件，生成训练好的所述概率图模型。The
综上，本公开实施例的模型训练装置，可基于样本文本序列对概率图模型进行训练，概率图模型用于输出每个候选状态下的训练初始状态概率、每个候选状态下的每个样本元素的训练观测概率、任意相邻两个样本元素的候选状态之间的训练状态转移概率，其中，候选状态用于表征样本元素是否为切分边界，可将概率图模型应用于子词切分场景。To sum up, the model training device in the embodiment of the present disclosure can train the probability graph model based on the sample text sequence, and the probability graph model is used to output the training initial state probability in each candidate state, each sample in each candidate state The training observation probability of the element, the training state transition probability between the candidate states of any two adjacent sample elements, where the candidate state is used to represent whether the sample element is a segmentation boundary, and the probability graph model can be applied to subword segmentation Scenes.
根据本公开的实施例，本公开还提供了一种电子设备、一种可读存储介质和一种计算机程序产品。According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
图10示出了可以用来实施本公开的实施例的示例电子设备1000的示意性框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。FIG. 10 shows a schematic block diagram of an example
如图10所示，电子设备1000包括计算单元1001，其可以根据存储在只读存储器(ROM)1002中的计算机程序或者从存储单元1008加载到随机访问存储器(RAM)1003中的计算机程序，来执行各种适当的动作和处理。在RAM 1003中，还可存储电子设备1000操作所需的各种程序和数据。计算单元1001、ROM 1002以及RAM1003通过总线1004彼此相连。输入/输出(I/O)接口1005也连接至总线1004。As shown in FIG. 10 , an
电子设备1000中的多个部件连接至I/O接口1005，包括：输入单元1006，例如键盘、鼠标等；输出单元1007，例如各种类型的显示器、扬声器等；存储单元1008，例如磁盘、光盘等；以及通信单元1009，例如网卡、调制解调器、无线通信收发机等。通信单元1009允许电子设备1000通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。Multiple components in the
计算单元1001可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元1001的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元1001执行上文所描述的各个方法和处理，例如图1至图6所述的子词切分方法，和/或图7所述的模型训练方法。例如，在一些实施例中，子词切分方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元1008。在一些实施例中，计算机程序的部分或者全部可以经由ROM 1002和/或通信单元1009而被载入和/或安装到电子设备1000上。当计算机程序加载到RAM 1003并由计算单元1001执行时，可以执行上文描述的子词切分方法的一个或多个步骤。备选地，在其他实施例中，计算单元1001可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行子词切分方法。The
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、负载可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS服务("Virtual Private Server"，或简称"VPS")中，存在的管理难度大，业务扩展性弱的缺陷。服务器也可以为分布式系统的服务器，或者是结合了区块链的服务器。A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the problem of traditional physical host and VPS service ("Virtual Private Server", or "VPS") Among them, there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.
根据本公开的实施例，本公开还提供了一种计算机程序产品，包括计算机程序，其中，所述计算机程序被处理器执行时实现本公开上述实施例所述的子词切分方法的步骤，或者实现本公开上述实施例所述的模型训练方法的步骤。According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, wherein, when the computer program is executed by a processor, the steps of the subword segmentation method described in the above-mentioned embodiments of the present disclosure are implemented, Or implement the steps of the model training method described in the above-mentioned embodiments of the present disclosure.
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发公开中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本公开公开的技术方案所期望的结果，本文在此不进行限制。It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.
上述具体实施方式，并不构成对本公开保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本公开的精神和原则之内所作的修改、等同替换和改进等，均应包含在本公开保护范围之内。The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.
Claims (14)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111656289.9ACN114492426B (en) | 2021-12-30 | 2021-12-30 | Sub-word segmentation method, model training method, device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111656289.9ACN114492426B (en) | 2021-12-30 | 2021-12-30 | Sub-word segmentation method, model training method, device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114492426A CN114492426A (en) | 2022-05-13 |
| CN114492426Btrue CN114492426B (en) | 2023-04-07 |
Family
ID=81508530
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111656289.9AActiveCN114492426B (en) | 2021-12-30 | 2021-12-30 | Sub-word segmentation method, model training method, device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114492426B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114861667B (en)* | 2022-05-16 | 2023-04-28 | 中电金信软件有限公司 | Named entity tag identification method and device |
| CN115130472B (en)* | 2022-08-31 | 2023-02-21 | 北京澜舟科技有限公司 | Method, system and readable storage medium for segmenting subwords based on BPE |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103441806A (en)* | 2013-07-30 | 2013-12-11 | 长春理工大学 | Pure discontinuous Markov process spectrum sensing method for cognitive radio |
| CN104900059A (en)* | 2015-05-26 | 2015-09-09 | 大连理工大学 | Method for enhancing cell phone base station positioning precision by using Hidden Markov map-matching algorithm |
| CN106569997A (en)* | 2016-10-19 | 2017-04-19 | 中国科学院信息工程研究所 | Scientific and technological compound phrase identification method based on hidden Markov model |
| CN108073570A (en)* | 2018-01-04 | 2018-05-25 | 焦点科技股份有限公司 | A kind of Word sense disambiguation method based on hidden Markov model |
| CN113724698A (en)* | 2021-09-01 | 2021-11-30 | 马上消费金融股份有限公司 | Training method, device and equipment of speech recognition model and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9875742B2 (en)* | 2015-01-26 | 2018-01-23 | Verint Systems Ltd. | Word-level blind diarization of recorded calls with arbitrary number of speakers |
| CN108509423A (en)* | 2018-04-04 | 2018-09-07 | 福州大学 | A kind of acceptance of the bid webpage name entity abstracting method based on second order HMM |
| CN108647208A (en)* | 2018-05-09 | 2018-10-12 | 上海应用技术大学 | A kind of novel segmenting method based on Chinese |
| CN108959262B (en)* | 2018-07-09 | 2022-07-26 | 鼎富智能科技有限公司 | Named entity identification method and device |
| CN109710759B (en)* | 2018-12-17 | 2021-06-08 | 北京百度网讯科技有限公司 | Text segmentation method and device, computer equipment and readable storage medium |
| CN112528645A (en)* | 2019-09-02 | 2021-03-19 | 株式会社Ntt都科摩 | Text processing method and device, electronic equipment and computer readable storage medium |
| CN111177402B (en)* | 2019-12-13 | 2023-09-22 | 中移（杭州）信息技术有限公司 | Evaluation method, device, computer equipment and storage medium based on word segmentation processing |
| CN112380855B (en)* | 2020-11-20 | 2024-03-08 | 北京百度网讯科技有限公司 | Method for determining sentence fluency, method and device for determining probabilistic prediction model |
- 2021
- 2021-12-30CNCN202111656289.9Apatent/CN114492426B/enactiveActive
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103441806A (en)* | 2013-07-30 | 2013-12-11 | 长春理工大学 | Pure discontinuous Markov process spectrum sensing method for cognitive radio |
| CN104900059A (en)* | 2015-05-26 | 2015-09-09 | 大连理工大学 | Method for enhancing cell phone base station positioning precision by using Hidden Markov map-matching algorithm |
| CN106569997A (en)* | 2016-10-19 | 2017-04-19 | 中国科学院信息工程研究所 | Scientific and technological compound phrase identification method based on hidden Markov model |
| CN108073570A (en)* | 2018-01-04 | 2018-05-25 | 焦点科技股份有限公司 | A kind of Word sense disambiguation method based on hidden Markov model |
| CN113724698A (en)* | 2021-09-01 | 2021-11-30 | 马上消费金融股份有限公司 | Training method, device and equipment of speech recognition model and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| Fei Wang."Statistic Chinese New Word Recognition by Combing Supervised and Unsupervised Learning".《2019 IEEE Intl Conf on Parallel &Distributed Processing with Applications, Big Data &Cloud Computing, Sustainable Computing &Communications, Social Computing &Networking 》.2020,第1239-1243页.* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114492426A (en) | 2022-05-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111709248B (en) | Training method and device for text generation model and electronic equipment | |
| CN116051668B (en) | Training method of Vincent graph diffusion model and text-based image generation method | |
| CN115309877B (en) | Dialogue generation method, dialogue model training method and device | |
| CN112466288B (en) | Voice recognition method and device, electronic equipment and storage medium | |
| CN112507706B (en) | Training method and device for knowledge pre-training model and electronic equipment | |
| CN116127045A (en) | Generative large language model training method, model-based human-computer voice interaction method | |
| CN111666759B (en) | Method, device, electronic equipment and storage medium for extracting key information of text | |
| CN112926306B (en) | Text error correction method, device, equipment and storage medium | |
| CN112786108B (en) | Training method, device, equipment and medium of molecular understanding model | |
| US20230004798A1 (en) | Intent recognition model training and intent recognition method and apparatus | |
| CN116244416A (en) | Training method for generating large language model and man-machine voice interaction method based on model | |
| CN114492426B (en) | Sub-word segmentation method, model training method, device and electronic equipment | |
| CN113889087B (en) | Speech recognition and model establishment method, device, equipment and storage medium | |
| CN115035890B (en) | Speech recognition model training methods, devices, electronic equipment and storage media | |
| CN114282551B (en) | Translation method, translation device, electronic equipment and storage medium | |
| CN114817476A (en) | Language model training method and device, electronic equipment and storage medium | |
| CN116166827B (en) | Training of semantic label extraction model and semantic label extraction method and device | |
| CN117033582A (en) | Training method and device for dialogue model, electronic equipment and storage medium | |
| CN114722841B (en) | Translation method, translation device and computer program product | |
| CN114417856B (en) | Text sparse coding method, device and electronic device | |
| CN114758649B (en) | A speech recognition method, device, equipment and medium | |
| CN114898742A (en) | Method, device, equipment and storage medium for training streaming voice recognition model | |
| CN113468857B (en) | Training method, device, electronic device and storage medium for style transfer model | |
| CN113553863A (en) | Text generation method and device, electronic equipment and storage medium | |
| CN116257611B (en) | Question-answering model training method, question-answering processing device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |