CN111563593A - Training method and device for neural network model - Google Patents
Training method and device for neural network modelDownload PDFInfo
- Publication number
- CN111563593A CN111563593ACN202010383383.0ACN202010383383ACN111563593ACN 111563593 ACN111563593 ACN 111563593ACN 202010383383 ACN202010383383 ACN 202010383383ACN 111563593 ACN111563593 ACN 111563593A
- Authority
- CN
- China
- Prior art keywords
- neural network
- network model
- target neural
- trained
- truncation strategy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本公开的实施例涉及计算机技术领域，具体涉及人工智能技术领域，尤其涉及神经网络模型的训练方法和装置。The embodiments of the present disclosure relate to the field of computer technology, in particular to the field of artificial intelligence technology, and in particular, to a method and apparatus for training a neural network model.
背景技术Background technique
神经网络模型的量化，是将高位宽的模型参数转换为低位宽的模型参数，以此提升模型的计算速度。通常在高位宽的神经网络模型训练完成之后进行量化。通常量化后得到的低位宽神经网络模型被直接用于执行相应的深度学习任务。然而，由于量化后参数的精度损失较大，可能导致量化后的模型的精度损失超过可接受的范围。The quantization of the neural network model is to convert high-bit-width model parameters into low-bit-width model parameters, so as to improve the calculation speed of the model. Quantization is usually performed after the training of the high-bit-width neural network model. Usually the low-bit-width neural network model obtained after quantization is directly used to perform the corresponding deep learning task. However, due to the large loss of accuracy of the parameters after quantization, it may cause the loss of accuracy of the quantized model to exceed the acceptable range.
发明内容SUMMARY OF THE INVENTION
本公开的实施例提供了神经网络模型的训练方法和装置、电子设备以及计算机可读存储介质。Embodiments of the present disclosure provide a training method and apparatus for a neural network model, an electronic device, and a computer-readable storage medium.
根据第一方面，提供了一种神经网络模型的训练方法，包括执行如下搜索操作：根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略，其中，截断策略包括神经网络模型的参数或中间输出数据的二进制表征中被截断的位数；基于当前的截断策略对待训练的目标神经网络模型进行迭代训练，其中，在训练过程中的每一次迭代，按照当前的截断策略对待训练的目标神经网络模型的参数或中间输出数据的二进制表征进行截断后生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数；获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息；响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。According to a first aspect, a method for training a neural network model is provided, including performing the following search operation: according to a preset truncation strategy controller, determining a current truncation strategy from the search space of the preset truncation strategy, wherein, The truncation strategy includes the number of truncated bits in the binary representation of the parameters of the neural network model or the intermediate output data; the target neural network model to be trained is iteratively trained based on the current truncation strategy, wherein, in each iteration in the training process, according to The current truncation strategy truncates the parameters of the target neural network model to be trained or the binary representation of the intermediate output data to generate the prediction result and loss function value of the target neural network model to be trained. parameters of the trained target neural network model; obtain the performance of the target neural network model trained based on the current truncation strategy and generate corresponding feedback information; in response to determining that the target neural network model trained based on the current truncation strategy does not reach the preset value The truncation policy controller is iteratively updated based on the feedback information to perform the next search operation based on the updated truncation policy controller.
根据第二方面，提供了一种神经网络模型的训练装置，包括搜索单元，被配置为执行搜索操作；搜索单元包括：确定单元，被配置为执行搜索操作中的如下步骤：根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略，其中，截断策略包括神经网络模型的参数或中间输出数据的二进制表征中被截断的位数；训练单元，被配置为执行搜索操作中的如下步骤：基于当前的截断策略对待训练的目标神经网络模型进行迭代训练，其中，在训练过程中的每一次迭代，按照当前的截断策略对待训练的目标神经网络模型的参数或中间输出数据的二进制表征进行截断后生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数；获取单元，被配置为执行搜索操作中的如下步骤：获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息；更新单元，被配置为执行搜索操作中的如下步骤：响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。According to a second aspect, a training device for a neural network model is provided, including a search unit configured to perform a search operation; the search unit includes: a determination unit configured to perform the following steps in the search operation: according to a preset truncation The strategy controller determines the current truncation strategy from the search space of the preset truncation strategy, wherein the truncation strategy includes the parameters of the neural network model or the number of bits truncated in the binary representation of the intermediate output data; the training unit is configured In order to perform the following steps in the search operation: iteratively train the target neural network model to be trained based on the current truncation strategy, wherein, in each iteration in the training process, the parameters of the target neural network model to be trained according to the current truncation strategy Or the binary representation of the intermediate output data is truncated to generate the prediction result and loss function value of the target neural network model to be trained, and the parameters of the target neural network model to be trained are updated by forward propagation of the loss function value; be configured to perform the following steps in the search operation: obtain the performance of the target neural network model trained based on the current truncation strategy and generate corresponding feedback information; the updating unit is configured to perform the following steps in the search operation: in response to determining based on The target neural network model trained by the current truncation strategy does not reach the preset convergence condition, and the truncation strategy controller is iteratively updated based on the feedback information to perform the next search operation based on the updated truncation strategy controller.
根据第三方面，提供了一种电子设备，包括：至少一个处理器；以及与至少一个处理器通信连接的存储器；其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够执行第一方面提供的神经网络模型的训练方法。According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are processed by the at least one processor The processor executes, so that at least one processor can execute the training method of the neural network model provided by the first aspect.
根据第四方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，计算机指令用于使计算机执行第一方面提供的神经网络模型的训练方法。According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method for training a neural network model provided in the first aspect.
根据本申请的技术通过在神经网络模型的训练过程中搜索最优的截断策略，使得训练得到的神经网络模型对量化不敏感，由此训练得到的神经网络模型在量化后的精度损失较小。According to the technology of the present application, by searching for an optimal truncation strategy in the training process of the neural network model, the neural network model obtained by training is insensitive to quantization, and thus the neural network model obtained by training suffers less precision loss after quantization.
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.
附图说明Description of drawings
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本公开的其它特征、目的和优点将会变得更明显：Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings:
图1是本公开的神经网络模型的训练方法的一个实施例的流程图；FIG. 1 is a flowchart of an embodiment of the training method of the neural network model of the present disclosure;
图2是本公开的神经网络模型的训练方法的另一个实施例的流程图；2 is a flowchart of another embodiment of the training method of the neural network model of the present disclosure;
图3是本公开的神经网络模型的训练方法的又一个实施例的流程图；FIG. 3 is a flowchart of another embodiment of the training method of the neural network model of the present disclosure;
图4是本公开的神经网络模型的训练装置的一个实施例的结构示意图；4 is a schematic structural diagram of an embodiment of a training device for a neural network model of the present disclosure;
图5是用来实现本公开的实施例的神经网络模型的训练方法的电子设备的框图。FIG. 5 is a block diagram of an electronic device used to implement the training method of the neural network model of the embodiment of the present disclosure.
具体实施方式Detailed ways
下面结合附图和实施例对本公开作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释相关发明，而非对该发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关发明相关的部分。The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.
需要说明的是，在不冲突的情况下，本公开中的实施例及实施例中的特征可以相互组合。下面将参考附图并结合实施例来详细说明本公开。It should be noted that the embodiments of the present disclosure and the features of the embodiments may be combined with each other under the condition of no conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
以下结合附图对本申请的示范性实施例做出说明，其中包括本申请实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本申请的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.
本公开的方法或装置可以应用于终端设备或服务器，或者可以应用于包括终端设备、网络和服务器的系统架构。其中，网络用以在终端设备和服务器之间提供通信链路的介质，可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。The method or apparatus of the present disclosure can be applied to a terminal device or a server, or can be applied to a system architecture including a terminal device, a network, and a server. The medium used by the network to provide a communication link between the terminal device and the server may include various connection types, such as wired, wireless communication links, or optical fiber cables.
终端设备可以是用户端设备，其上可以安装有各种客户端应用。例如，图像处理类应用、搜索应用、语音服务类应用等。终端设备可以是硬件，也可以是软件。当终端设备为硬件时，可以是各种电子设备，包括但不限于智能手机、平板电脑、电子书阅读器、膝上型便携计算机和台式计算机等等。当终端设备为软件时，可以安装在上述所列举的电子设备中。其可以实现成多个软件或软件模块，也可以实现成单个软件或软件模块。在此不做具体限定。The terminal device may be a client device on which various client applications may be installed. For example, image processing applications, search applications, voice service applications, etc. Terminal equipment can be hardware or software. When the terminal device is hardware, it can be various electronic devices, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, desktop computers, and the like. When the terminal device is software, it can be installed in the electronic devices listed above. It can be implemented as a plurality of software or software modules, and can also be implemented as a single software or software module. There is no specific limitation here.
服务器可以是运行各种服务的服务器，例如运行基于图像、视频、语音、文本、数字信号等数据的目标检测与识别、文本或语音识别、信号转换等服务的服务器。服务器可以获取深度学习任务数据来构建训练样本，对用于执行深度学习任务的神经网络模型进行训练。The server may be a server that runs various services, for example, a server that runs services such as object detection and recognition, text or speech recognition, and signal conversion based on data such as images, videos, voices, texts, and digital signals. The server can obtain deep learning task data to construct training samples, and train the neural network model for performing the deep learning task.
服务器可以是为终端设备上安装的应用提供后端支持的后端服务器。例如，服务器可以接收终端设备发送的待处理的数据，使用神经网络模型对数据进行处理，并将处理结果返回至终端设备。The server may be a back-end server that provides back-end support for applications installed on the terminal device. For example, the server may receive the data to be processed sent by the terminal device, process the data using the neural network model, and return the processing result to the terminal device.
需要说明的是，服务器可以是硬件，也可以是软件。当服务器为硬件时，可以实现成多个服务器组成的分布式服务器集群，也可以实现成单个服务器。当服务器为软件时，可以实现成多个软件或软件模块(例如用来提供分布式服务的多个软件或软件模块)，也可以实现成单个软件或软件模块。在此不做具体限定。It should be noted that the server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or can be implemented as a single software or software module. There is no specific limitation here.
需要说明的是，本公开的实施例所提供的神经网络模型的训练方法可以由终端设备或服务器执行，相应地，神经网络模型的训练装置可以设置于终端设备或服务器中。It should be noted that the training method of the neural network model provided by the embodiments of the present disclosure may be executed by a terminal device or a server, and correspondingly, the training apparatus of the neural network model may be set in the terminal device or the server.
请参考图1，其示出了根据本公开的神经网络模型的训练方法的一个实施例的流程100。该神经网络模型的训练方法，包括执行搜索操作，搜索操作具体包括如下步骤101、步骤102、步骤103和步骤104：Please refer to FIG. 1 , which shows a
步骤101，根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略。
截断策略控制器用于生成截断策略。其中，截断策略包括神经网络模型的参数或中间输出数据的二进制表征中被截断的位数。在参数或中间输出数据的二进制表征中，被截断的位数是二进制表征的最后若干位，也即二进制表征的最后一位向前的若干位，被截断的位数被置为0。例如，在一个8位整型的参数的二进制表征11001011，中，若被截断的位数为3，则将最后2位置为0，截断后二进制表征为11001000。被截断后的参数或中间输出数据的精度降低，例如当截断位数为3时，8位整型的数据11001011和11001110截断后的二进制表征都是11001000。The truncation policy controller is used to generate truncation policies. Among them, the truncation strategy includes the truncated number of bits in the parameters of the neural network model or the binary representation of the intermediate output data. In the binary representation of parameters or intermediate output data, the truncated number of bits is the last several bits of the binary representation, that is, several bits ahead of the last bit of the binary representation, and the truncated number of bits is set to 0. For example, in the binary representation of an 8-bit integer parameter 11001011, if the number of bits to be truncated is 3, the last 2 bits are set to 0, and the truncated binary representation is 11001000. The precision of the truncated parameters or intermediate output data is reduced. For example, when the number of truncated digits is 3, the truncated binary representations of 8-bit integer data 11001011 and 11001110 are both 11001000.
上述截断策略控制器用于控制神经网络模型的指定层的中间输出数据或参数的截断位数。在这里，中间输出数据可以是中间层输出的数据，例如中间的卷积层、全连接层、池化层等的输出的特征图或者向量数据。The above truncation strategy controller is used to control the truncation bits of the intermediate output data or parameters of the specified layer of the neural network model. Here, the intermediate output data may be the data output by the intermediate layer, such as the output feature map or vector data of the intermediate convolutional layer, fully connected layer, pooling layer, etc.
截断策略控制器可以实现为循环神经网络、卷积神经网络等神经网络模型，或者可以实现为概率模型等数学模型，还可以实现为强化学习算法、进化算法、模拟退火算法等，其可以在迭代执行搜索操作的过程中基于其搜索出的截断策略的评估结果进行自动更新，进而更新搜索出的当前截断策略。The truncation strategy controller can be implemented as a neural network model such as a cyclic neural network and a convolutional neural network, or as a mathematical model such as a probability model, or as a reinforcement learning algorithm, an evolutionary algorithm, a simulated annealing algorithm, etc., which can be implemented in an iterative manner. In the process of executing the search operation, automatic updating is performed based on the evaluation result of the searched truncation strategy, and then the current truncation strategy searched out is updated.
截断策略控制器可以生成截断策略序列，按照预先定义的截断策略的编解码规则，对截断策略序列进行解码得到神经网络模型对应层的参数或中间输出数据的截断策略。The truncation strategy controller can generate a truncation strategy sequence, and decode the truncation strategy sequence according to the pre-defined encoding and decoding rules of the truncation strategy to obtain the parameters of the corresponding layer of the neural network model or the truncation strategy of the intermediate output data.
预设的截断策略搜索空间可以包括神经网络模型的若干层的参数或中间输出数据的可选截断位数。在每一次搜索操作中，可以利用当前的截断策略控制器，从预设的截断策略搜索空间内搜索出当前的截断策略。The preset truncation strategy search space may include parameters of several layers of the neural network model or optional truncation bits of intermediate output data. In each search operation, the current truncation strategy controller can be used to search for the current truncation strategy from the preset truncation strategy search space.
需要说明的是，基于不同截断位数截断后的参数或中间输出数据的精度损失不同。一般地，截断位数越多，截断后的精度损失越大，但截断位数越少，训练完成的神经网络模型对量化的敏感度越高。本实施例的方法通过多次搜索操作可以搜索出目标神经网络模型的最优截断策略。It should be noted that the truncated parameters or intermediate output data have different precision losses based on different truncation digits. Generally, the more truncation bits, the greater the loss of precision after truncation, but the less truncation bits, the higher the sensitivity of the trained neural network model to quantization. The method of this embodiment can search for the optimal truncation strategy of the target neural network model through multiple search operations.
步骤102，基于当前的截断策略对待训练的目标神经网络模型进行迭代训练。Step 102: Perform iterative training on the target neural network model to be trained based on the current truncation strategy.
可以通过多次迭代训练目标神经网络模型。在训练过程中的每一次迭代，按照当前的截断策略对待训练的目标神经网络模型的参数或中间输出数据的二进制表征进行截断后生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数。The target neural network model can be trained through multiple iterations. At each iteration in the training process, the parameters of the target neural network model to be trained or the binary representation of the intermediate output data are truncated according to the current truncation strategy to generate the prediction results and loss function values of the target neural network model to be trained. The loss function value is forward propagated to update the parameters of the target neural network model to be trained.
具体地，在训练过程中的每一次迭代，将样本数据输入目标神经网络模型，根据当前的截断策略对目标神经网络模型的参数或中间输出数据的二进制表征进行截断，利用截断后的参数或中间输出数据得到待训练的目标神经网络模型对样本数据的预测结果，并利用截断后的参数或中间输出数据计算表征目标神经网络模型的预测误差的损失函数值，根据损失函数值，采用梯度下降法迭代更新待训练的目标神经网络模型的参数。当待训练的目标神经网络模型的参数收敛、或者上述损失函数值收敛时，可以停止训练目标神经网络模型，得到训练完成的目标神经网络模型。Specifically, at each iteration in the training process, the sample data is input into the target neural network model, the parameters of the target neural network model or the binary representation of the intermediate output data are truncated according to the current truncation strategy, and the truncated parameters or intermediate The output data obtains the prediction result of the target neural network model to be trained on the sample data, and uses the truncated parameters or intermediate output data to calculate the loss function value representing the prediction error of the target neural network model, and uses the gradient descent method according to the loss function value. Iteratively update the parameters of the target neural network model to be trained. When the parameters of the target neural network model to be trained converge or the value of the above-mentioned loss function converges, the training of the target neural network model can be stopped, and the trained target neural network model can be obtained.
可选地，上述截断策略可以包括神经网络模型的特征提取层输出的特征图的二进制表征中被截断的位数。在这里，特征图的二进制表征是特征图各像素值的二进制表征。这时，可以按照如下方式生成待训练的目标神经网络模型的预测结果和损失函数值：将样本图像数据输入待训练的目标神经网络模型进行特征提取，按照当前的截断策略，对待训练的目标神经网络模型的至少一个特征提取层输出的特征图的二进制表征截断对应的位数，并基于截断后的特征图的二进制表征生成待训练的目标神经网络模型的预测结果和损失函数值。Optionally, the above truncation strategy may include truncated bits in the binary representation of the feature map output by the feature extraction layer of the neural network model. Here, the binary representation of the feature map is the binary representation of each pixel value of the feature map. At this time, the prediction result and loss function value of the target neural network model to be trained can be generated as follows: Input the sample image data into the target neural network model to be trained for feature extraction, and according to the current truncation strategy, the target neural network to be trained The binary representation of the feature map output by at least one feature extraction layer of the network model is truncated by the corresponding number of bits, and the prediction result and the loss function value of the target neural network model to be trained are generated based on the binary representation of the truncated feature map.
在确定当前的截断策略，即确定当前的搜索操作中目标神经网络模型的至少一个特征提取层对应的截断位数后，可以将目标神经网络模型针对样本图像数据在对应的特征提取层输出的特征图的二进制表征进行相应位数的截断。并基于截断后的特征图输出目标神经网络模型对样本图像数据的预测结果以及用于监督目标数神经网络模型训练的损失函数的值。After determining the current truncation strategy, that is, determining the number of truncation bits corresponding to at least one feature extraction layer of the target neural network model in the current search operation, the target neural network model can target the sample image data in the corresponding feature extraction layer. The binary representation of the graph is truncated by the corresponding number of bits. And based on the truncated feature map, the prediction result of the target neural network model on the sample image data and the value of the loss function used to supervise the training of the target neural network model are output.
步骤103，获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息。Step 103: Obtain the performance of the target neural network model trained based on the current truncation strategy and generate corresponding feedback information.
可以利用测试数据对基于当前的截断策略训练完成的目标神经网络模型的性能进行测试。在这里，目标神经网络模型的性能可以包括：准确率、在指定的运行环境下运行的延时、召回率、或者内存占用率，等等。可以根据实践中需求确定测试的性能。例如在实时性需求较高的用户交互场景中，可以测试目标神经网络模型在指定硬件上运行的延时。在准确性需求较高的场景中，例如基于人脸的用户身份认证场景中，可以测试目标神经网络模型的准确率。The performance of the target neural network model trained based on the current truncation strategy can be tested by using the test data. Here, the performance of the target neural network model can include: accuracy, latency in a specified operating environment, recall, or memory usage, etc. The performance of the test can be determined based on practical requirements. For example, in a user interaction scenario with high real-time requirements, the running delay of the target neural network model on the specified hardware can be tested. In scenarios with high accuracy requirements, such as face-based user authentication scenarios, the accuracy of the target neural network model can be tested.
可以根据训练完成的目标神经网络模型的性能生成对应的反馈信息。该反馈信息可以用反馈值表示。反馈值的初始值可以设定为0。在每一次搜索操作中获取基于当前的截断策略训练完成的目标神经网络模型的性能之后，可以更新反馈值。该反馈值作为当前的截断策略的评估指标，可以被反馈至预设的截断策略控制器。Corresponding feedback information can be generated according to the performance of the trained target neural network model. The feedback information can be represented by a feedback value. The initial value of the feedback value can be set to 0. After obtaining the performance of the target neural network model trained based on the current truncation strategy in each search operation, the feedback value can be updated. The feedback value is used as the evaluation index of the current truncation strategy, and can be fed back to the preset truncation strategy controller.
步骤104，响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。
如果确定在当前的搜索操作中训练完成的目标神经网络模型未达到预设的收敛条件，则基于反馈信息迭代更新截断策略控制器。If it is determined that the target neural network model trained in the current search operation does not reach the preset convergence condition, the truncation policy controller is iteratively updated based on the feedback information.
上述预设的收敛条件可以包括以下至少一项：搜索操作的次数达到预设的次数阈值，训练完成的目标神经网络模型的性能达到预设的性能阈值，训练完成的目标神经网络模型在连续多次搜索操作中的性能的变化不超过预设的变化阈值，等等。The above-mentioned preset convergence conditions may include at least one of the following: the number of search operations reaches a preset number of times threshold, the performance of the trained target neural network model reaches a preset performance threshold, and the trained target neural network model is continuously The change in performance in the secondary search operation does not exceed a preset change threshold, and so on.
上述截断策略控制器可以在反馈值的作用下更新。当截断策略控制器实现为循环神经网络或卷积神经网络时，可以基于反馈值更新循环神经网络或卷积神经网络的参数。当截断策略控制器实现为进化算法时，可以将反馈值作为截断策略种群的适应度，对截断策略种群进行进化。当截断策略控制器实现为强化学习算法时，反馈值作为强化学习模型的奖励值(reward)，使得强化学习模型基于奖励值更新参数。The above truncation strategy controller can be updated under the action of the feedback value. When the truncation policy controller is implemented as a recurrent neural network or a convolutional neural network, the parameters of the recurrent neural network or the convolutional neural network can be updated based on the feedback values. When the truncation strategy controller is implemented as an evolutionary algorithm, the feedback value can be used as the fitness of the truncation strategy population to evolve the truncation strategy population. When the truncation policy controller is implemented as a reinforcement learning algorithm, the feedback value is used as the reward value of the reinforcement learning model, so that the reinforcement learning model updates the parameters based on the reward value.
在下一次搜索操作中，更新后的截断策略控制器可以生成新的当前的截断策略。通过多次执行搜索操作可以搜索出最优的截断策略。而由于在最优的截断策略中对目标神经网络模型的参数或中间输出数据进行了截断，使得神经网络模型对量化的敏感度降低，从而缩小了基于最优的截断策略截断后的目标神经网络模型的量化损失。In the next search operation, the updated truncation policy controller can generate a new current truncation policy. The optimal truncation strategy can be found by performing the search operation multiple times. However, because the parameters or intermediate output data of the target neural network model are truncated in the optimal truncation strategy, the sensitivity of the neural network model to quantization is reduced, thereby reducing the target neural network truncated based on the optimal truncation strategy. The quantization loss of the model.
请参考图2，其示出了本公开的神经网络模型的训练方法的另一个实施例的流程示意图。本实施例的神经网络模型的训练方法的流程200包括执行多次搜索操作，其中，搜索操作包括以下步骤201至步骤204：Please refer to FIG. 2 , which shows a schematic flowchart of another embodiment of the training method of the neural network model of the present disclosure. The
步骤201，根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略。
截断策略控制器可以实现为循环神经网络、卷积神经网络等神经网络模型，或者可以实现为概率模型等数学模型，还可以实现为强化学习算法、进化算法、模拟退火算法等，其可以根据基于搜索出的截断策略的评估结果进行自动更新。The truncation strategy controller can be implemented as a neural network model such as a cyclic neural network and a convolutional neural network, or as a mathematical model such as a probability model, or as a reinforcement learning algorithm, an evolutionary algorithm, a simulated annealing algorithm, etc. The evaluation results of the searched truncation strategies are automatically updated.
在本实施例中，截断策略包括神经网络模型的中间层输出的特征图的二进制表征中被截断的位数。可以预先设定目标神经网络模型的至少一个中间层为指定的进行截断操作的中间层，在每次搜索操作中从截断策略搜索空间中搜索出各个指定的进行截断操作的中间层对应的截断位数，作为当前的截断策略。In this embodiment, the truncation strategy includes the number of truncated bits in the binary representation of the feature map output by the middle layer of the neural network model. At least one intermediate layer of the target neural network model can be preset as a designated intermediate layer for truncation operation, and in each search operation, the truncation bit corresponding to each designated intermediate layer for truncation operation is searched from the truncation strategy search space. number as the current truncation strategy.
可选地，本实施例的神经网络模型的训练方法的流程还可以包括构建预设的截断策略的搜索空间的步骤。在这里，预设的截断策略搜索空间包括待训练的目标神经网络模型中的至少一个中间层输出的特征图对应的候选截断位数。各中间层输出的特征图对应的候选截断位数可以预先设定，例如设定为区间[1，32]中的每一个整数。则在每一次搜索操作中，截断策略控制器可以在该区间内搜索对应的特征图的截断位数，组合不同的而中间层输出的特征图对应的截断位数构成整个目标神经网络模型的当前的截断策略。Optionally, the flow of the neural network model training method in this embodiment may further include the step of constructing a search space of a preset truncation strategy. Here, the preset truncation strategy search space includes candidate truncation bits corresponding to the feature map output by at least one intermediate layer in the target neural network model to be trained. The number of candidate truncation bits corresponding to the feature maps output by each intermediate layer can be preset, for example, set to each integer in the interval [1, 32]. Then in each search operation, the truncation strategy controller can search for the truncation bits of the corresponding feature map in the interval, and combine different truncation bits corresponding to the feature maps output by the middle layer to form the current value of the entire target neural network model. truncation strategy.
步骤202，基于当前的截断策略对待训练的目标神经网络模型进行迭代训练。Step 202: Perform iterative training on the target neural network model to be trained based on the current truncation strategy.
步骤202包括执行多次迭代操作，其中每一次迭代操作包括以下步骤2021：Step 202 includes performing multiple iterations, wherein each iteration includes the following step 2021:
步骤2021，将样本图像数据输入待训练的目标神经网络模型进行特征提取，按照当前的截断策略，对待训练的目标神经网络模型的至少一个中间层输出的特征图的二进制表征截断对应的位数，并基于截断后的特征图的二进制表征生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数。
具体地，在上述步骤2021中，将样本图像数据输入待训练的目标神经网络模型之后，根据当前的截断策略，对目标神经网络模型的对应中间层输出的特征图的二进制表征进行截断，并将截断后的特征图替换原特征图，利用目标神经网络模型的得到最终的预测结果，并根据预测结果的误差计算损失函数值。Specifically, in the
可选地，上述迭代操作还包括：响应于确定待训练的目标神经网络模型的迭代操作次数未达到预设的阈值，且待训练的目标神经网络模型对应的损失函数值未收敛至预设的范围内，基于损失函数值更新目标神经网络模型的参数，并执行下一次迭代操作；以及响应于确定待训练的目标神经网络模型的迭代操作次数达到预设的阈值，或者待训练的目标神经网络模型对应的损失函数值收敛至预设的范围内，停止执行迭代操作，得到基于当前的截断策略训练完成的目标神经网络模型。Optionally, the above-mentioned iterative operation further includes: in response to determining that the number of iterative operations of the target neural network model to be trained does not reach a preset threshold, and the loss function value corresponding to the target neural network model to be trained does not converge to a preset value. Within the range, the parameters of the target neural network model are updated based on the loss function value, and the next iterative operation is performed; and in response to determining that the number of iterative operations of the target neural network model to be trained reaches a preset threshold, or the target neural network to be trained The loss function value corresponding to the model converges to a preset range, and the iterative operation is stopped to obtain the target neural network model trained based on the current truncation strategy.
步骤203，获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息。Step 203: Obtain the performance of the target neural network model trained based on the current truncation strategy and generate corresponding feedback information.
步骤204，响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。
本实施例的步骤203和步骤204分别与前述实施例的步骤103和步骤104一致，步骤203和步骤204的具体实现方式可以分别参考前述实施例对步骤103和步骤104的描述，此处不再赘述。
由于神经网络模型的特征图对神经网络模型的最终预测结果和损失函数值具有直接的影响，而不会直接影响神经网络模型参数的精度。本实施例通过对特征图截断来计算损失函数值，使得损失函数值对中间输出数据的精度不敏感，从而基于损失函数更新的神经网络模型对特征图的精度不敏感，从而在确保神经网络模型参数精度的情况下降低了神经网络模型对量化的敏感度。Since the feature map of the neural network model has a direct impact on the final prediction result and the loss function value of the neural network model, it will not directly affect the accuracy of the parameters of the neural network model. In this embodiment, the loss function value is calculated by truncating the feature map, so that the loss function value is not sensitive to the accuracy of the intermediate output data, so that the neural network model updated based on the loss function is not sensitive to the accuracy of the feature map, thus ensuring that the neural network model is not sensitive to the accuracy of the feature map. The sensitivity of the neural network model to quantization is reduced in the case of parameter accuracy.
继续参考图3，其示出了本公开的神经网络模型的训练方法的又一个实施例的流程图。如图3所示，本实施例的神经网络模型的训练方法的流程300，包括执行搜索操作。其中，搜索操作包括以下步骤301至步骤305：Continue to refer to FIG. 3 , which shows a flowchart of yet another embodiment of the training method of the neural network model of the present disclosure. As shown in FIG. 3 , the
步骤301，根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略，其中，截断策略包括神经网络模型的参数或中间输出数据的二进制表征中被截断的位数。
步骤302，基于当前的截断策略对待训练的目标神经网络模型进行迭代训练，其中，在训练过程中的每一次迭代，按照当前的截断策略对待训练的目标神经网络模型的参数或中间输出数据的二进制表征进行截断后生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数。
步骤303，获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息。Step 303: Obtain the performance of the target neural network model trained based on the current truncation strategy and generate corresponding feedback information.
步骤304，响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。
上述步骤301至步骤304分别与前述实施例的步骤101至步骤104一致，或者可以与前述实施例的步骤201至步骤204一致。步骤301至步骤304的具体实现方式可以分别参数前述实施例中对应步骤的描述，此处不再赘述。The foregoing steps 301 to 304 are respectively consistent with
步骤305，响应于确定基于当前的截断策略训练完成的目标神经网络模型达到预设的收敛条件，对基于当前的截断策略训练完成的目标神经网络模型进行量化，得到量化后的目标神经网络模型。
在搜索操作中确定基于当前的截断策略训练完成的目标神经网络模型达到预设的收敛条件时，可以停止执行搜索操作，这时训练完成的目标神经网络模型可以作为基于最优的截断策略训练完成的目标神经网络模型。可以对基于最优的截断策略训练完成的目标神经网络模型的参数进行量化，得到量化后的目标神经网络模型。In the search operation, when it is determined that the target neural network model trained based on the current truncation strategy reaches the preset convergence condition, the search operation can be stopped. At this time, the trained target neural network model can be trained as the optimal truncation strategy. target neural network model. The parameters of the target neural network model trained based on the optimal truncation strategy can be quantified to obtain the quantized target neural network model.
由于基于最优的截断策略训练完成的目标神经网络模型对模型参数或中间输出数据的二进制表征中被截断的位的数值不敏感，因此基于最优的截断策略训练完成的目标神经网络模型对参数量化造成的参数精度损失的敏感程度降低，量化得到的模型能够达到较高的精度。Since the target neural network model trained based on the optimal truncation strategy is not sensitive to the value of the model parameters or the truncated bits in the binary representation of the intermediate output data, the target neural network model trained based on the optimal truncation strategy is not sensitive to the parameter The sensitivity to the loss of parameter accuracy caused by quantization is reduced, and the model obtained by quantization can achieve higher accuracy.
可选地，上述神经网络模型的训练方法的流程300还可以包括：将量化后的目标神经网络模型发送至终端侧，以在终端侧部署量化后的目标神经网络模型并利用量化后的目标神经网络模型处理对应的任务数据。Optionally, the
通常终端侧对神经网络模型的实时性要求较高，利用量化后的模型可以提升模型的数据处理速度。并且上述量化后的目标神经网络模型能达到较高的精度，能够在终端侧高效地获得较准确的处理结果。Usually, the real-time performance of the neural network model on the terminal side is relatively high. Using the quantized model can improve the data processing speed of the model. In addition, the above-mentioned quantized target neural network model can achieve high precision, and can efficiently obtain more accurate processing results on the terminal side.
请参考图4，作为对上述神经网络模型的训练方法的实现，本公开提供了一种神经网络模型的训练装置的一个实施例，该装置实施例与上述各方法实施例相对应，该装置具体可以应用于各种电子设备中。Please refer to FIG. 4 , as an implementation of the training method for the above-mentioned neural network model, the present disclosure provides an embodiment of a training apparatus for a neural network model, and the apparatus embodiment corresponds to the above-mentioned method embodiments. Can be used in various electronic devices.
如图4所示，本实施例的神经网络模型的训练装置400包括搜索单元401。搜索单元401被配置为执行搜索操作。搜索单元401包括：确定单元4011，被配置为执行搜索操作中的如下步骤：根据预设的截断策略控制器，从预设的截断策略的搜索空间中确定出当前的截断策略，其中，截断策略包括神经网络模型的参数或中间输出数据的二进制表征中被截断的位数；训练单元4012，被配置为执行搜索操作中的如下步骤：基于当前的截断策略对待训练的目标神经网络模型进行迭代训练，其中，在训练过程中的每一次迭代，按照当前的截断策略对待训练的目标神经网络模型的参数或中间输出数据的二进制表征进行截断后生成待训练的目标神经网络模型的预测结果和损失函数值，通过将损失函数值前向传播以更新待训练的目标神经网络模型的参数；获取单元4013，被配置为执行搜索操作中的如下步骤：获取基于当前的截断策略训练完成的目标神经网络模型的性能并生成对应的反馈信息；更新单元4014，被配置为执行搜索操作中的如下步骤：响应于确定基于当前的截断策略训练完成的目标神经网络模型未达到预设的收敛条件，基于反馈信息迭代更新截断策略控制器，以基于更新后的截断策略控制器执行下一次搜索操作。As shown in FIG. 4 , the
在一些实施例中，上述截断策略包括神经网络模型的中间层输出的特征图的二进制表征中被截断的位数；以及上述训练单元4012被配置为按照如下方式生成待训练的目标神经网络模型的预测结果和损失函数值：将样本图像数据输入待训练的目标神经网络模型进行特征提取，按照当前的截断策略，对待训练的目标神经网络模型的至少一个中间层输出的特征图的二进制表征截断对应的位数，并基于截断后的特征图的二进制表征生成待训练的目标神经网络模型的预测结果和损失函数值。In some embodiments, the above-mentioned truncation strategy includes the number of bits truncated in the binary representation of the feature map output by the middle layer of the neural network model; and the above-mentioned
在一些实施例中，上述装置还包括：构建单元，被配置为构建预设的截断策略的搜索空间，预设的截断策略搜索空间包括待训练的目标神经网络模型中的至少一个中间层输出的特征图对应的候选截断位数。In some embodiments, the above-mentioned apparatus further includes: a construction unit, configured to construct a search space for a preset truncation strategy, the preset truncation strategy search space includes the output of at least one intermediate layer in the target neural network model to be trained. The number of candidate truncation bits corresponding to the feature map.
在一些实施例中，上述搜索单元401还包括：量化单元，被配置为执行搜索操作中的如下步骤：响应于确定基于当前的截断策略训练完成的目标神经网络模型达到预设的收敛条件，对基于当前的截断策略训练完成的目标神经网络模型进行量化，得到量化后的目标神经网络模型。In some embodiments, the above-mentioned
在一些实施例中，上述装置还包括：发送单元，被配置为将量化后的目标神经网络模型发送至终端侧，以在终端侧部署量化后的目标神经网络模型并利用量化后的目标神经网络模型处理对应的任务数据。In some embodiments, the above apparatus further includes: a sending unit configured to send the quantized target neural network model to the terminal side, so as to deploy the quantized target neural network model on the terminal side and utilize the quantized target neural network The model processes the corresponding task data.
上述装置400与前述方法实施例中的步骤相对应。由此，上文针对神经网络模型的训练方法描述的操作、特征及所能达到的技术效果同样适用于装置400及其中包含的单元，在此不再赘述。The foregoing
根据本申请的实施例，本申请还提供了一种电子设备和一种可读存储介质。According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.
如图5所示，是根据本申请实施例的神经网络模型的训练方法的电子设备的框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本申请的实现。As shown in FIG. 5 , it is a block diagram of an electronic device according to the training method of a neural network model according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.
如图5所示，该电子设备包括：一个或多个处理器501、存储器502，以及用于连接各部件的接口，包括高速接口和低速接口。各个部件利用不同的总线互相连接，并且可以被安装在公共主板上或者根据需要以其它方式安装。处理器可以对在电子设备内执行的指令进行处理，包括存储在存储器中或者存储器上以在外部输入/输出装置(诸如，耦合至接口的显示设备)上显示GUI的图形信息的指令。在其它实施方式中，若需要，可以将多个处理器和/或多条总线与多个存储器和多个存储器一起使用。同样，可以连接多个电子设备，各个设备提供部分必要的操作(例如，作为服务器阵列、一组刀片式服务器、或者多处理器系统)。图5中以一个处理器501为例。As shown in FIG. 5, the electronic device includes: one or
存储器502即为本申请所提供的非瞬时计算机可读存储介质。其中，存储器存储有可由至少一个处理器执行的指令，以使至少一个处理器执行本申请所提供的神经网络模型的训练方法。本申请的非瞬时计算机可读存储介质存储计算机指令，该计算机指令用于使计算机执行本申请所提供的神经网络模型的训练方法。The
存储器502作为一种非瞬时计算机可读存储介质，可用于存储非瞬时软件程序、非瞬时计算机可执行程序以及模块，如本申请实施例中的神经网络模型的训练方法对应的程序指令/单元/模块(例如，附图4所示的搜索单元401)。处理器501通过运行存储在存储器502中的非瞬时软件程序、指令以及模块，从而执行服务器的各种功能应用以及数据处理，即实现上述方法实施例中的神经网络模型的训练方法。As a non-transitory computer-readable storage medium, the
存储器502可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储根据用于生成神经网络的结构的电子设备的使用所创建的数据等。此外，存储器502可以包括高速随机存取存储器，还可以包括非瞬时存储器，例如至少一个磁盘存储器件、闪存器件、或其他非瞬时固态存储器件。在一些实施例中，存储器502可选包括相对于处理器501远程设置的存储器，这些远程存储器可以通过网络连接至用于生成神经网络的结构的电子设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。The
神经网络模型的训练方法的电子设备还可以包括：输入装置503和输出装置504。处理器501、存储器502、输入装置503和输出装置504可以通过总线505或者其他方式连接，图5中以通过总线505连接为例。The electronic device of the neural network model training method may further include: an
输入装置503可接收输入的数字或字符信息，以及产生与用于生成神经网络的结构的电子设备的用户设置以及功能控制有关的键信号输入，例如触摸屏、小键盘、鼠标、轨迹板、触摸板、指示杆、一个或者多个鼠标按钮、轨迹球、操纵杆等输入装置。输出装置Y04可以包括显示设备、辅助照明装置(例如，LED)和触觉反馈装置(例如，振动电机)等。该显示设备可以包括但不限于，液晶显示器(LCD)、发光二极管(LED)显示器和等离子体显示器。在一些实施方式中，显示设备可以是触摸屏。The
此处描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、专用ASIC(专用集成电路)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.
这些计算程序(也称作程序、软件、软件应用、或者代码)包括可编程处理器的机器指令，并且可以利用高级过程和/或面向对象的编程语言、和/或汇编/机器语言来实施这些计算程序。如本文使用的，术语“机器可读介质”和“计算机可读介质”指的是用于将机器指令和/或数据提供给可编程处理器的任何计算机程序产品、设备、和/或装置(例如，磁盘、光盘、存储器、可编程逻辑装置(PLD))，包括，接收作为机器可读信号的机器指令的机器可读介质。术语“机器可读信号”指的是用于将机器指令和/或数据提供给可编程处理器的任何信号。These computational programs (also referred to as programs, software, software applications, or codes) include machine instructions for programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or apparatus for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的发明范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述发明构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本申请中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned technical features or Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in this application (but not limited to) with similar functions.
Claims (12)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010383383.0ACN111563593B (en) | 2020-05-08 | 2020-05-08 | Training method and device for neural network model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010383383.0ACN111563593B (en) | 2020-05-08 | 2020-05-08 | Training method and device for neural network model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111563593Atrue CN111563593A (en) | 2020-08-21 |
| CN111563593B CN111563593B (en) | 2023-09-15 |
Family
ID=72071843
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010383383.0AActiveCN111563593B (en) | 2020-05-08 | 2020-05-08 | Training method and device for neural network model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563593B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112149266A (en)* | 2020-10-23 | 2020-12-29 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining network model quantization strategy |
| CN112231511A (en)* | 2020-10-20 | 2021-01-15 | 腾讯音乐娱乐科技（深圳）有限公司 | Neural network model training method and song mining method and device |
| CN113128680A (en)* | 2021-03-12 | 2021-07-16 | 山东英信计算机技术有限公司 | Neural network training method, system, device and medium |
| CN113159410A (en)* | 2021-04-14 | 2021-07-23 | 北京百度网讯科技有限公司 | Training method for automatic control model and fluid supply system control method |
| CN114064697A (en)* | 2021-11-30 | 2022-02-18 | 中国工商银行股份有限公司 | Automatic log searching method and system |
| CN114595373A (en)* | 2020-12-03 | 2022-06-07 | 北京搜狗科技发展有限公司 | A search method, device and electronic device |
| CN115494844A (en)* | 2022-09-26 | 2022-12-20 | 成都朴为科技有限公司 | Multi-robot searching method and system |
| CN116634553A (en)* | 2022-02-10 | 2023-08-22 | 维沃移动通信有限公司 | Information processing method and communication device |
| CN119851458A (en)* | 2023-10-17 | 2025-04-18 | 纬湃汽车电子(天津)有限公司 | Method and apparatus for determining a truncation policy for freshness value of secure vehicle communications |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030191728A1 (en)* | 2002-03-27 | 2003-10-09 | Kulkarni Bhaskar Dattatray | Performance of artificial neural network models in the presence of instrumental noise and measurement errors |
| US6990246B1 (en)* | 1999-08-21 | 2006-01-24 | Vics Limited | Image coding |
| US9536190B2 (en)* | 2013-10-17 | 2017-01-03 | Qualcomm Incorporated | Dynamically assigning and examining synaptic delay |
| US20170006129A1 (en)* | 2015-07-01 | 2017-01-05 | Oracle International Corporation | System and method for multi-version remote function execution control in a distributed computing environment |
| US20170061279A1 (en)* | 2015-01-14 | 2017-03-02 | Intel Corporation | Updating an artificial neural network using flexible fixed point representation |
| CN107646116A (en)* | 2015-05-08 | 2018-01-30 | 高通股份有限公司 | Bitwidth Selection for Fixed-Point Neural Networks |
| US20180032866A1 (en)* | 2016-07-28 | 2018-02-01 | Samsung Electronics Co., Ltd. | Neural network method and apparatus |
| CN108460760A (en)* | 2018-03-06 | 2018-08-28 | 陕西师范大学 | A kind of Bridge Crack image discriminating restorative procedure fighting network based on production |
| CN109146060A (en)* | 2018-08-09 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of method and device based on convolutional neural networks processing data |
| CN109800865A (en)* | 2019-01-24 | 2019-05-24 | 北京市商汤科技开发有限公司 | Neural network generation and image processing method and device, platform, electronic equipment |
| CN110210454A (en)* | 2019-06-17 | 2019-09-06 | 合肥工业大学 | A kind of human action pre-judging method based on data fusion |
| CN110210611A (en)* | 2019-05-13 | 2019-09-06 | 西安交通大学 | A kind of dynamic self-adapting data truncation method calculated for convolutional neural networks |
| CN110245753A (en)* | 2019-05-27 | 2019-09-17 | 东南大学 | A kind of neural network compression method based on power exponent quantization |
| CN110717585A (en)* | 2019-09-30 | 2020-01-21 | 上海寒武纪信息科技有限公司 | Training method of neural network model, data processing method and related product |
| CN110852421A (en)* | 2019-11-11 | 2020-02-28 | 北京百度网讯科技有限公司 | Model generation method and device |
| CN110852439A (en)* | 2019-11-20 | 2020-02-28 | 字节跳动有限公司 | Neural network model compression and acceleration method, data processing method and device |
- 2020
- 2020-05-08CNCN202010383383.0Apatent/CN111563593B/enactiveActive
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6990246B1 (en)* | 1999-08-21 | 2006-01-24 | Vics Limited | Image coding |
| US20030191728A1 (en)* | 2002-03-27 | 2003-10-09 | Kulkarni Bhaskar Dattatray | Performance of artificial neural network models in the presence of instrumental noise and measurement errors |
| US9536190B2 (en)* | 2013-10-17 | 2017-01-03 | Qualcomm Incorporated | Dynamically assigning and examining synaptic delay |
| US20170061279A1 (en)* | 2015-01-14 | 2017-03-02 | Intel Corporation | Updating an artificial neural network using flexible fixed point representation |
| CN107646116A (en)* | 2015-05-08 | 2018-01-30 | 高通股份有限公司 | Bitwidth Selection for Fixed-Point Neural Networks |
| US20170006129A1 (en)* | 2015-07-01 | 2017-01-05 | Oracle International Corporation | System and method for multi-version remote function execution control in a distributed computing environment |
| US20180032866A1 (en)* | 2016-07-28 | 2018-02-01 | Samsung Electronics Co., Ltd. | Neural network method and apparatus |
| CN107665364A (en)* | 2016-07-28 | 2018-02-06 | 三星电子株式会社 | Neural net method and equipment |
| CN108460760A (en)* | 2018-03-06 | 2018-08-28 | 陕西师范大学 | A kind of Bridge Crack image discriminating restorative procedure fighting network based on production |
| CN109146060A (en)* | 2018-08-09 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of method and device based on convolutional neural networks processing data |
| CN109800865A (en)* | 2019-01-24 | 2019-05-24 | 北京市商汤科技开发有限公司 | Neural network generation and image processing method and device, platform, electronic equipment |
| CN110210611A (en)* | 2019-05-13 | 2019-09-06 | 西安交通大学 | A kind of dynamic self-adapting data truncation method calculated for convolutional neural networks |
| CN110245753A (en)* | 2019-05-27 | 2019-09-17 | 东南大学 | A kind of neural network compression method based on power exponent quantization |
| CN110210454A (en)* | 2019-06-17 | 2019-09-06 | 合肥工业大学 | A kind of human action pre-judging method based on data fusion |
| CN110717585A (en)* | 2019-09-30 | 2020-01-21 | 上海寒武纪信息科技有限公司 | Training method of neural network model, data processing method and related product |
| CN110852421A (en)* | 2019-11-11 | 2020-02-28 | 北京百度网讯科技有限公司 | Model generation method and device |
| CN110852439A (en)* | 2019-11-20 | 2020-02-28 | 字节跳动有限公司 | Neural network model compression and acceleration method, data processing method and device |
Non-Patent Citations (4)
| Title |
|---|
| PABLO M. TOSTADO等: "Performance Trade-offs in Weight Quantization for Memory-Efficient Inference", 《2019 IEEE INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE CIRCUITS AND SYSTEMS》, pages 246 - 250* |
| ＲＩＣＡＲＤＯ ＣＡＳＴＲＯ ＥＴ ＡＬ: "ＳＶＤ ｔｒｕｎｃａｔｉｏｎ ｓｃｈｅｍｅｓ ｆｏｒ ｆｉｘｅｄ－ｓｉｚｅ ｋｅｒｎｅｌ ｍｏｄｅｌｓ", pages 3922 - 3929* |
| 王晨枫: "电力系统负荷预测的技术研究", 《中国优秀博硕士学位论文全文数据库（硕士）工程科技Ⅱ辑》, no. 2, pages 042 - 149* |
| 高琛: "基于ＦＰＧＡ的递归神经网络加速器的研究进展", vol. 5, pages 1 - 13* |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231511A (en)* | 2020-10-20 | 2021-01-15 | 腾讯音乐娱乐科技（深圳）有限公司 | Neural network model training method and song mining method and device |
| CN112149266A (en)* | 2020-10-23 | 2020-12-29 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining network model quantization strategy |
| CN112149266B (en)* | 2020-10-23 | 2024-09-24 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining network model quantization strategy |
| CN114595373A (en)* | 2020-12-03 | 2022-06-07 | 北京搜狗科技发展有限公司 | A search method, device and electronic device |
| CN113128680A (en)* | 2021-03-12 | 2021-07-16 | 山东英信计算机技术有限公司 | Neural network training method, system, device and medium |
| CN113128680B (en)* | 2021-03-12 | 2022-06-10 | 山东英信计算机技术有限公司 | A neural network training method, system, device and medium |
| CN113159410A (en)* | 2021-04-14 | 2021-07-23 | 北京百度网讯科技有限公司 | Training method for automatic control model and fluid supply system control method |
| CN113159410B (en)* | 2021-04-14 | 2024-02-27 | 北京百度网讯科技有限公司 | Training method of automatic control model and fluid supply system control method |
| CN114064697A (en)* | 2021-11-30 | 2022-02-18 | 中国工商银行股份有限公司 | Automatic log searching method and system |
| CN116634553A (en)* | 2022-02-10 | 2023-08-22 | 维沃移动通信有限公司 | Information processing method and communication device |
| CN115494844A (en)* | 2022-09-26 | 2022-12-20 | 成都朴为科技有限公司 | Multi-robot searching method and system |
| CN119851458A (en)* | 2023-10-17 | 2025-04-18 | 纬湃汽车电子(天津)有限公司 | Method and apparatus for determining a truncation policy for freshness value of secure vehicle communications |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111563593B (en) | 2023-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111563593B (en) | Training method and device for neural network model | |
| CN111428008B (en) | Method, apparatus, device and storage medium for training a model | |
| CN111667054B (en) | Method, device, electronic equipment and storage medium for generating neural network model | |
| KR102573637B1 (en) | Entity linking method and device, electronic equipment and storage medium | |
| CN111539479B (en) | Method and apparatus for generating sample data | |
| CN111582453A (en) | Method and apparatus for generating neural network model | |
| CN111582479B (en) | Distillation method and device for neural network model | |
| CN111539514A (en) | Method and apparatus for generating the structure of a neural network | |
| KR20210132578A (en) | Method, apparatus, device and storage medium for constructing knowledge graph | |
| CN111563592A (en) | Method and device for generating neural network model based on hypernetwork | |
| US20210200813A1 (en) | Human-machine interaction method, electronic device, and storage medium | |
| CN111667056B (en) | Method and apparatus for searching model structure | |
| JP7044839B2 (en) | End-to-end model training methods and equipment | |
| CN111738418A (en) | Super network training method and device | |
| CN111582477B (en) | Training method and device for neural network model | |
| CN110852379B (en) | Method and device for generating training samples for target object recognition | |
| CN111310449B (en) | Text generation method and device based on semantic representation | |
| CN111639753A (en) | Method, apparatus, device and storage medium for training a hyper-network | |
| CN111738419A (en) | Quantization method and device for neural network model | |
| CN111582454A (en) | Method and apparatus for generating neural network model | |
| CN111862987A (en) | Speech recognition method and device | |
| CN111340219A (en) | Neural network model searching method and device, image processing method and processor | |
| CN111695698A (en) | Method, device, electronic equipment and readable storage medium for model distillation | |
| CN111695699A (en) | Method, device, electronic equipment and readable storage medium for model distillation | |
| CN111274407A (en) | Method and device for calculating triple confidence in knowledge graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |