CN115546186A - Agricultural pest detection method and device based on YOLO v4 - Google Patents
Agricultural pest detection method and device based on YOLO v4Download PDFInfo
- Publication number
- CN115546186A CN115546186ACN202211338472.9ACN202211338472ACN115546186ACN 115546186 ACN115546186 ACN 115546186ACN 202211338472 ACN202211338472 ACN 202211338472ACN 115546186 ACN115546186 ACN 115546186A
- Authority
- CN
- China
- Prior art keywords
- agricultural
- frame
- yolo
- head
- loss function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
- G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Probability & Statistics with Applications (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Catching Or Destruction (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及农业智能化和目标检测软件技术领域，尤其涉及一种基于YOLO v4的农业病虫害检测方法及装置。The invention relates to the technical field of agricultural intelligence and target detection software, in particular to a method and device for detecting agricultural diseases and insect pests based on YOLO v4.
背景技术Background technique
农业病虫害种类繁多且形成环境因素复杂，如果不提前预防，一旦发生极易为农户带来巨大损失，严重威胁影响国家农业丰收和粮食安全。目前，单纯依靠人工观测和经验辨识很可能会导致疾病的误判，难以达到预期的防治效果。同时，由于目前的农业技术人员经验不足，各种现代化技术实际运用的能力较低，难以在作物生产中及时发现和预警病害。因此，基于深度学习算法的自动检测农业病虫害系统应用而生，通过深度学习完成目标检测的相关研究应用在农业病虫害识别时，不仅可以识别病虫害属于哪个种类，更重要的是可以识别出病害和虫害在图片中的具体位置，因此已被广泛使用。相关技术中，使用YOLOv4单阶段模型算法对农业病虫害进行检测，YOLO v4单阶段模型包括主干特征网络、Neck模块和引导头(Yolo head)模块，通过引导头对特征进行分类预测，由于引导头匹配的候选先验框数量有限，容易造成漏检，导致检测结果不准确，用户使用体验差。There are many kinds of agricultural diseases and insect pests and the environmental factors are complex. If they are not prevented in advance, once they occur, they will easily bring huge losses to farmers and seriously threaten the national agricultural harvest and food security. At present, relying solely on manual observation and empirical identification is likely to lead to misjudgment of diseases, and it is difficult to achieve the expected prevention and treatment effect. At the same time, due to the lack of experience of current agricultural technicians and the low ability to actually apply various modern technologies, it is difficult to detect and warn of diseases in crop production in time. Therefore, the application of the automatic detection system of agricultural pests and diseases based on deep learning algorithms was born. The relevant research on the completion of target detection through deep learning is applied to the identification of agricultural pests and diseases. The specific location in the picture has therefore been widely used. In related technologies, the YOLOv4 single-stage model algorithm is used to detect agricultural pests and diseases. The YOLO v4 single-stage model includes a backbone feature network, a Neck module, and a Yolo head module. The number of candidate prior frames is limited, which is easy to cause missed detection, resulting in inaccurate detection results and poor user experience.
发明内容Contents of the invention
本发明提供一种基于YOLO v4的农业病虫害检测方法及装置，用以解决传统YOLOv4模型进行农业病虫害检测时输出的预测结果不准确，导致用户使用体验差的缺陷。The invention provides a method and device for detecting agricultural diseases and insect pests based on YOLO v4, which is used to solve the defect that the traditional YOLOv4 model outputs inaccurate prediction results when detecting agricultural diseases and insect pests, resulting in poor user experience.
本发明提供一种农业病虫害检测方法，包括：The invention provides a method for detecting agricultural diseases and insect pests, comprising:
将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；Input the image of agricultural diseases and insect pests to be detected into the agricultural diseases and insect pests detection model based on YOLO v4;
根据所述基于YOLO v4的农业病虫害检测模型得到检测结果；Obtain detection results according to the agricultural pest detection model based on YOLO v4;
所述基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，所述检测结果根据所述引导头输出预测结果和辅助头输出预测结果加权求和得到。The YOLO v4-based agricultural pest detection model includes a guide head and an auxiliary head, and the detection result is obtained by weighted summation of the output prediction result of the guide head and the output prediction result of the auxiliary head.
根据本发明提供一种农业病虫害检测方法，还包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, which also includes:
根据所述引导头生成粗标签和所述细标签，所述粗标签用于辅助辅导头进行训练，所述细标签用于辅助引导头进行训练。A coarse label and the fine label are generated according to the guidance head, the coarse label is used to assist the guidance head in training, and the fine label is used to assist the guidance head in training.
根据本发明提供一种农业病虫害检测方法，所述基于YOLO v4的农业病虫害检测模型的训练过程包括：Provide a kind of agricultural disease and insect pest detection method according to the present invention, the training process of the described agricultural disease and insect pest detection model based on YOLO v4 comprises:
根据损失函数计算所述引导头输出预测结果与真实结果之间的第一损失值；calculating a first loss value between the guide head output prediction result and the real result according to a loss function;
根据所述第一损失值和所述细标签指导所述引导头输出预测结果；instructing the leader to output a prediction result according to the first loss value and the thin label;
根据损失函数计算所述辅助头输出预测结果与真实结果之间的第二损失值；calculating a second loss value between the auxiliary head output prediction result and the real result according to a loss function;
根据所述第二损失值和所述粗标签指导所述辅助头输出预测结果。Instructing the auxiliary head to output a prediction result according to the second loss value and the coarse label.
所述基于YOLO v4的农业病虫害检测模型的训练过程包括：The training process of the agricultural disease and insect pest detection model based on YOLO v4 includes:
根据损失函数计算所述引导头输出预测结果与真实结果之间的第一损失值；calculating a first loss value between the guide head output prediction result and the real result according to a loss function;
根据所述第一损失值指导所述引导头输出预测结果；instructing the seeker to output a prediction result according to the first loss value;
根据损失函数计算所述辅助头输出预测结果与真实结果之间的第二损失值；calculating a second loss value between the auxiliary head output prediction result and the real result according to a loss function;
根据所述第二损失值指导所述辅助头输出预测结果。instructing the auxiliary head to output a prediction result according to the second loss value.
根据本发明提供一种农业病虫害检测方法，所述损失函数公式为：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, and the loss function formula is:
其中，
根据本发明提供一种农业病虫害检测方法，所述子损失函数，包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, and the sub-loss function includes:
坐标损失函数、目标置信度损失函数和分类损失函数，其中目标置信度损失函数和分类损失函数采用带log的二值交叉熵损失函数，坐标损失函数采用CIoU损失函数。Coordinate loss function, target confidence loss function, and classification loss function, where the target confidence loss function and classification loss function use the binary cross-entropy loss function with log, and the coordinate loss function uses the CIoU loss function.
根据本发明提供一种农业病虫害检测方法，所述基于YOLO v4的农业病虫害检测模型根据训练样本集进行训练得到，所述训练样本集生成过程包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided. The agricultural diseases and insect pests detection model based on YOLO v4 is trained according to the training sample set, and the training sample set generation process includes:
将训练数据集中的多个GT框进行聚类操作得到多个先验框；Perform clustering operations on multiple GT boxes in the training data set to obtain multiple prior boxes;
将每个GT框与每个先验框进行匹配；Match each GT box with each prior box;
根据匹配结果筛选出正样本、负样本和背景；Screen out positive samples, negative samples, and background based on matching results;
根据所述正样本、负样本和背景生成训练样本集。A training sample set is generated according to the positive samples, negative samples and background.
根据本发明提供一种农业病虫害检测方法，所述训练样本集生成过程还包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, and the training sample set generation process also includes:
将离GT框的中心位置最近的预设个数邻域网格作为正样本。The preset number of neighborhood grids closest to the center of the GT box are used as positive samples.
根据本发明提供一种农业病虫害检测方法，所述根据匹配结果筛选出正样本、负样本和背景，包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, wherein the positive samples, negative samples and background are screened out according to the matching results, including:
分别计算每个GT框与每个先验框的框尺寸比值，所述框尺寸比值包括框宽比值和框高比值；Calculate the frame size ratio of each GT frame and each prior frame, the frame size ratio includes a frame width ratio and a frame height ratio;
若所述框尺寸比值的最大值小于预设比值阈值，则判定对应GT框为正样本；If the maximum value of the frame size ratio is less than a preset ratio threshold, it is determined that the corresponding GT frame is a positive sample;
若所述框尺寸比值的最大值不小于预设比值阈值，则判定对应GT框为负样本；If the maximum value of the frame size ratio is not less than a preset ratio threshold, it is determined that the corresponding GT frame is a negative sample;
若GT框与任一先验框的框尺寸比值小于等于1，则判定所述GT框为背景。If the frame size ratio of the GT frame to any prior frame is less than or equal to 1, it is determined that the GT frame is the background.
根据本发明提供一种农业病虫害检测方法，若一个GT框与多个先验框的框尺寸比值均大于预设比值阈值，包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided, if the frame size ratios of a GT frame and multiple prior frames are greater than the preset ratio threshold, including:
将大于预设比值阈值对应的多个先验框按照匹配代价从小到大排列，所述匹配代价根据预测结果和真实结果计算出的IoU损失与分类损失相加得到；Arranging a plurality of prior frames corresponding to a preset ratio threshold in ascending order of matching cost, the matching cost is obtained by adding the IoU loss and classification loss calculated according to the prediction result and the real result;
根据所述先验框的排列顺序依次筛选出预设数量阈值的先验框作为正样本。According to the arrangement order of the a priori frames, a priori frames with a preset number of thresholds are sequentially screened out as positive samples.
本发明还提供一种农业病虫害检测装置，包括：The present invention also provides a detection device for agricultural diseases and insect pests, comprising:
输入模块，用于将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；The input module is used to input the image of agricultural diseases and insect pests to be detected into the agricultural diseases and insect pests detection model based on YOLO v4;
输出模块，用于根据所述基于YOLO v4的农业病虫害检测模型得到检测结果；所述基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，所述检测结果根据所述引导头输出预测结果和辅助头输出预测结果加权求和得到。Output module, for obtaining the detection result according to the agricultural disease and pest detection model based on YOLO v4; The agricultural disease and pest detection model based on YOLO v4 includes a guide head and an auxiliary head, and the detection result outputs a prediction result according to the guide head and the output prediction results of the auxiliary head are weighted and summed.
本发明还提供一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如上述任一种所述的农业病虫害检测方法。The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, the agricultural diseases and insect pests as described in any one of the above can be realized. Detection method.
本发明还提供一种非暂态计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现如上述任一种所述的农业病虫害检测方法。The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for detecting agricultural diseases and insect pests as described in any one of the above-mentioned methods is realized.
本发明提供的基于YOLO v4的农业病虫害检测方法及装置，通过将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；根据基于YOLO v4的农业病虫害检测模型得到检测结果；基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据引导头输出预测结果和辅助头输出预测结果加权求和得到，通过增加辅助头，可以增加备选分类结果，避免漏检，提高农业病虫害检测的准确性。The method and device for detecting agricultural diseases and insect pests based on YOLO v4 provided by the present invention input the image of agricultural diseases and insect pests to be detected into the agricultural diseases and insect pests detection model based on YOLO v4; obtain the detection results according to the agricultural diseases and insect pests detection model based on YOLO v4; the agricultural diseases and insect pests detection model based on YOLO v4 The pest detection model includes a guide head and an auxiliary head. The detection result is obtained by the weighted summation of the output prediction results of the guide head and the output prediction results of the auxiliary head. By adding the auxiliary head, the alternative classification results can be increased to avoid missed detection and improve the detection of agricultural pests and diseases. accuracy.
附图说明Description of drawings
为了更清楚地说明本发明或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the present invention or the technical solutions in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are the present invention. For some embodiments of the invention, those skilled in the art can also obtain other drawings based on these drawings without creative effort.
图1是本发明提供的农业病虫害检测方法的流程示意图；Fig. 1 is a schematic flow sheet of the agricultural disease and insect pest detection method provided by the present invention;
图2是本发明提供的基于YOLO v4的农业病虫害检测模型的辅助训练模块结构示意图；Fig. 2 is the auxiliary training module structural representation of the agricultural disease and insect pest detection model based on YOLO v4 provided by the present invention;
图3是本发明提供的基于YOLO v4的农业病虫害检测模型中引导头和辅助头学习流程示意图；Fig. 3 is a schematic diagram of the learning process of the guide head and the auxiliary head in the YOLO v4-based agricultural pest detection model provided by the present invention;
图4是本发明提供的农业病虫害检测方法中模型训练流程图；Fig. 4 is a flow chart of model training in the agricultural disease and insect pest detection method provided by the present invention;
图5是本发明提供的农业病虫害检测方法中模型训练样本集生成流程图。Fig. 5 is a flow chart of generating a model training sample set in the method for detecting agricultural diseases and insect pests provided by the present invention.
图6是本发明提供的农业病虫害检测装置的结构示意图；Fig. 6 is a schematic structural view of an agricultural pest detection device provided by the present invention;
图7是本发明提供的电子设备的结构示意图。Fig. 7 is a schematic structural diagram of an electronic device provided by the present invention.
具体实施方式detailed description
为使本发明的目的、技术方案和优点更加清楚，下面将结合本发明中的附图，对本发明中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention , but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
图1为本发明实施例提供的农业病虫害检测方法的流程图，如图1所示，本发明实施例提供的农业病虫害检测方法包括：Fig. 1 is the flowchart of the detection method of agricultural diseases and insect pests provided by the embodiment of the present invention, as shown in Fig. 1, the detection method of agricultural diseases and insect pests provided by the embodiment of the present invention comprises:
步骤101、将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；
步骤102、根据基于YOLO v4的农业病虫害检测模型得到检测结果，基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据引导头输出预测结果和辅助头输出预测结果加权求和得到。Step 102: Obtain the detection result based on the YOLO v4-based agricultural disease and pest detection model. The YOLO v4-based agricultural disease and pest detection model includes a guide head and an auxiliary head, and the detection result is obtained by weighting and summing the output prediction results of the guide head and the auxiliary head output prediction results. .
在本发明实施例中，在网络的中间层增加额外的用于辅助训练的头，称之为辅助头(Aux Head)，基于YOLO v4的农业病虫害检测模型中YOLO Head结构如附图2所示，其工作原理包括：In the embodiment of the present invention, an additional head for auxiliary training is added to the middle layer of the network, which is called an auxiliary head (Aux Head). The structure of the YOLO Head in the agricultural disease and pest detection model based on YOLO v4 is shown in Figure 2 , which works by:
将从YOLO v4模型Neck模块中一些特征层输出的特征数据分别输入至不同的辅助头，将Neck模块中另一些特征层输出的特征数据分别输入至不同的引导头。辅助头和引导头均需对目标进行深度监督，通过引导头的预测来引导辅助头以及自身学习优化。Input the feature data output from some feature layers in the Neck module of the YOLO v4 model to different auxiliary heads, and input the feature data output from other feature layers in the Neck module to different guide heads. Both the auxiliary head and the guidance head need to carry out deep supervision on the target, and guide the auxiliary head and self-learning optimization through the prediction of the guidance head.
本发明实施例利用引导头的预测结果作为指导，生成从粗到细的层次标签，将这些层次标签分别用于辅助头(Aux Head)和引导头(Lead Head)的学习，改变YOLO v4原有的单方面YOLO Head输出方式，提高分类结果准确性。The embodiment of the present invention uses the prediction results of the lead head as a guide to generate hierarchical labels from coarse to fine, and use these hierarchical labels for the learning of the auxiliary head (Aux Head) and the lead head (Lead Head), changing the original YOLO v4 The unilateral YOLO Head output method improves the accuracy of classification results.
本发明实施例提供的农业病虫害检测方法通过将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型，根据基于YOLO v4的农业病虫害检测模型得到检测结果，基于YOLO v4的农业病虫害检测模型中辅助训练模块包括引导头和辅助头，利用引导头生成软标签，辅助头和引导头基于所述软标签和损失函数进行训练，以根据训练好的辅助头优化模型网络权重，根据训练好的引导头输出检测结果，一方面，通过辅助头优化模型网络权重，提高辅助模块作用，从而优化模型性能，提高检测结果准确性，另一方面，通过引导头的预测来指导辅助头以及自身预测结果，可以进一步提高对农业病虫害检测的准确性。The method for detecting agricultural diseases and insect pests provided by the embodiments of the present invention inputs the image of the agricultural diseases and insect pests to be detected into the agricultural diseases and insect pests detection model based on YOLO v4, obtains the detection results according to the agricultural diseases and insect pests detection model based on YOLO v4, and assists in the agricultural diseases and insect pests detection model based on YOLO v4 The training module includes a guiding head and an auxiliary head, and the guiding head is used to generate soft labels. The auxiliary head and the guiding head are trained based on the soft label and loss function, so as to optimize the model network weight according to the trained auxiliary head, and according to the trained guiding head To output the detection results, on the one hand, the weight of the model network is optimized through the auxiliary head, and the role of the auxiliary module is improved, thereby optimizing the performance of the model and improving the accuracy of the detection results; Further improve the accuracy of detection of agricultural diseases and insect pests.
辅助头和引导头的输入不同，辅助头和引导头均需对目标进行深度监督，通过引导头的预测来引导辅助头以及自身。首先使用引导头的预测作为指导，生成从粗到细的层次标签，再分别用于辅助头和引导头的学习，如图3所示，其中的层次标签需使用“标签分配器”，即将网络预测结果与真实结果一起考虑后再分配软标签的机制。The input of the auxiliary head and the guide head is different. Both the auxiliary head and the guide head need to carry out deep supervision on the target, and guide the auxiliary head and itself through the prediction of the guide head. Firstly, use the prediction of the guide head as a guide to generate hierarchical labels from coarse to fine, and then use them for the learning of the auxiliary head and the guide head respectively, as shown in Figure 3, where the hierarchical labels need to use the "label assigner", that is, the network A mechanism for assigning soft labels after prediction results are considered together with ground truth results.
在本发明实施例中，还包括：In an embodiment of the present invention, it also includes:
根据引导头生成粗标签和所述细标签，所述粗标签用于辅助辅导头进行训练，细标签用于辅助引导头进行训练。Generate coarse labels and the fine labels according to the guidance head, the coarse labels are used to assist the guidance head in training, and the fine labels are used to assist the guidance head in training.
细标签与传统YOLO v4模型中引导头在标签分配器上生成的软标签相同，用于指导分类结果输出；The fine label is the same as the soft label generated by the guide head on the label allocator in the traditional YOLO v4 model, and is used to guide the classification result output;
粗标签用于降低正样本分配约束，允许更多的网格作为正样本，可以提高分类结果的准确性。Coarse labels are used to reduce the distribution constraints of positive samples, allowing more grids to be used as positive samples, which can improve the accuracy of classification results.
通过引入coarse-to-fine(由粗到细)引导标签分配策略可以对病虫害进行更准确的分类和定位。由于传统YOLO v4模型中只具有引导头进行分类输出，易造成预测结果标签分配不准确，导致用户使用体验差。By introducing a coarse-to-fine (from coarse to fine) guide label assignment strategy, the pests and diseases can be classified and located more accurately. Since the traditional YOLO v4 model only has a leader for classification output, it is easy to cause inaccurate label assignment of prediction results, resulting in poor user experience.
基于上述任一实施例，如图4所示，基于YOLO v4的农业病虫害检测模型的训练过程包括：Based on any of the above-mentioned embodiments, as shown in Figure 4, the training process of the agricultural disease and insect pest detection model based on YOLO v4 includes:
步骤401、根据损失函数计算引导头输出预测结果与真实结果之间的第一损失值；
步骤402、根据第一损失值和细标签指导引导头输出预测结果；
步骤403、根据损失函数计算辅助头输出预测结果与真实结果之间的第二损失值；
步骤404、根据第二损失值和粗标签指导辅助头输出预测结果。
在本发明实施例中，损失函数公式为：In the embodiment of the present invention, the loss function formula is:
其中，
子损失函数，包括：Sub-loss functions, including:
坐标损失函数、目标置信度损失函数和分类损失函数，其中目标置信度损失函数和分类损失函数采用带log的二值交叉熵损失函数，坐标损失函数采用CIoU损失函数。Coordinate loss function, target confidence loss function, and classification loss function, where the target confidence loss function and classification loss function use the binary cross-entropy loss function with log, and the coordinate loss function uses the CIoU loss function.
坐标损失函数的表达式如下所示：The expression of the coordinate loss function is as follows:
其中，
目标置信度损失由正样本匹配得到的样本对计算，样本对包括预测框中的目标置信度分数po和预测框和与之对应的目标框的IoU值，其作为GT框，两者计算二进制交叉熵得到最终的目标置信度损失。目标置信度的公式如下：The target confidence loss is calculated by the sample pair obtained by positive sample matching. The sample pair includes the target confidence score po in the prediction frame and the IoU value of the prediction frame and the corresponding target frame, which is used as a GT frame. The two are calculated as binary Cross entropy to get the final target confidence loss. The formula for target confidence is as follows:
分类损失与目标置信度损失类似，通过预测结果的类别分数和目标类别的one-hot表现来计算分类损失，分类损失函数如下所示：The classification loss is similar to the target confidence loss. The classification loss is calculated by the category score of the predicted result and the one-hot performance of the target category. The classification loss function is as follows:
三个部分的损失均是通过匹配得到的正样本对来计算，每一个输出特征图相互独立，直接相加得到最终每一部分的损失值。The losses of the three parts are all calculated by matching positive sample pairs, and each output feature map is independent of each other, and directly added to obtain the final loss value of each part.
根据损失函数计算每个GT框和候选先验框损失，前期会加大分类损失权重，后期减低分类损失权重，并保留损失最小的前K个，去除掉同一个先验框被分配到多个GT框的情况。Calculate the loss of each GT frame and candidate a priori frame according to the loss function. The classification loss weight will be increased in the early stage, and the classification loss weight will be reduced in the later stage, and the top K with the smallest loss will be retained, and the same a priori frame will be assigned to multiple The case of the GT box.
需要说明的是，辅助头和引导头的损失函数相同，且辅助头输出预测结果的加权系数不能过大，否则会导致模型输出检测结果精度变低。It should be noted that the loss function of the auxiliary head and the leading head are the same, and the weighting coefficient of the auxiliary head output prediction results cannot be too large, otherwise the accuracy of the model output detection results will be reduced.
本发明实施例提供的农业病虫害检测方法将原有的没有深度检测的目标检测器架构进行改进，添加了辅助训练的辅助头，将辅助头和引导头的权重做融合，提高模型性能，同时引入新的标签分配方法，通过引导头的预测来引导辅助头以及自身，提高对农业病虫害检测的准确性。The agricultural disease and insect pest detection method provided by the embodiment of the present invention improves the original target detector architecture without in-depth detection, adds an auxiliary head for auxiliary training, fuses the weights of the auxiliary head and the guiding head, improves the performance of the model, and introduces The new label assignment method guides the auxiliary head and itself through the prediction of the leading head to improve the accuracy of agricultural pest detection.
基于上述任一实施例，如图5所示，基于YOLO v4的农业病虫害检测模型根据训练样本集进行训练得到，所述训练样本集生成过程包括：Based on any of the above-mentioned embodiments, as shown in Figure 5, the agricultural disease and insect pest detection model based on YOLO v4 is trained according to the training sample set, and the training sample set generation process includes:
步骤501、将训练数据集中的多个GT框进行聚类操作得到多个先验框；
在本发明实施例中，聚类算法例如为K-Means聚类算法，得到9个先验框。In the embodiment of the present invention, the clustering algorithm is, for example, the K-Means clustering algorithm, and nine prior frames are obtained.
步骤502、将每个GT框与每个先验框进行匹配；
步骤503、根据匹配结果筛选出正样本、负样本和背景；
在本发明实施例中，根据匹配结果筛选出正样本、负样本和背景，包括：In the embodiment of the present invention, positive samples, negative samples and background are screened out according to the matching results, including:
分别计算每个GT框与每个先验框的框尺寸比值，框尺寸比值包括框宽比值和框高比值；Calculate the box size ratio of each GT box and each prior box separately, the box size ratio includes the box width ratio and the box height ratio;
若所述框尺寸比值的最大值小于预设比值阈值，则判定对应GT框为正样本；If the maximum value of the frame size ratio is less than a preset ratio threshold, it is determined that the corresponding GT frame is a positive sample;
在本发明实施例中，分别计算每个GT框与每个预测框的框宽的比值和框高的比值；In the embodiment of the present invention, the ratio of the frame width and the frame height of each GT frame to each predicted frame are calculated respectively;
取多个框宽的比值中的最大值和多个框高的比值中的最大值；Take the maximum value of the ratio of multiple frame widths and the maximum value of the ratio of multiple frame heights;
若框宽的比值的最大值和框高的比值的最大值均小于预设比值阈值，则判定对应先验框为正样本。If both the maximum value of the ratio of the frame width and the maximum value of the ratio of the frame height are less than the preset ratio threshold, it is determined that the corresponding prior frame is a positive sample.
若所述框尺寸比值的最大值不小于预设比值阈值，则判定对应GT框为负样本；If the maximum value of the frame size ratio is not less than a preset ratio threshold, it is determined that the corresponding GT frame is a negative sample;
若GT框与任一先验框的框尺寸比值小于等于1，则判定所述GT框为背景。If the frame size ratio of the GT frame to any prior frame is less than or equal to 1, it is determined that the GT frame is the background.
若一个GT框也可能会没有任何先验框能匹配，此时的GT框会被默认为背景，不参与模型训练，此时本领域技术人员可以调整先验框的大小。If a GT frame may not be matched by any prior frame, the GT frame at this time will be defaulted as the background and will not participate in model training. At this time, those skilled in the art can adjust the size of the prior frame.
步骤504、根据正样本、负样本和背景生成训练样本集。
在本发明实施例中，训练样本集生成过程还包括：In an embodiment of the present invention, the training sample set generation process also includes:
将离GT框的中心位置最近的预设个数邻域网格作为正样本。The preset number of neighborhood grids closest to the center of the GT box are used as positive samples.
预设个数例如为2，通过增加预测框数量可以扩充正样本。把离GT框的中心位置最近的2个邻域网格也作为预测框，此时的GT框由先验框，2个相邻网格来预测，与传统的YOLOv4模型相比，正样本数量变为3倍(在本发明实施例中最大的匹配数由9个升至27个)。The preset number is, for example, 2, and positive samples can be expanded by increasing the number of prediction frames. The two neighboring grids closest to the center of the GT box are also used as prediction boxes. At this time, the GT box is predicted by the prior box and two adjacent grids. Compared with the traditional YOLOv4 model, the number of positive samples 3 times (in the embodiment of the present invention, the maximum number of matches rises from 9 to 27).
根据本发明提供一种农业病虫害检测方法，若一个GT框与多个先验框的框尺寸比值均大于预设比值阈值，匹配策略是使用SimOTA方法，包括：According to the present invention, a method for detecting agricultural diseases and insect pests is provided. If the frame size ratios of a GT frame and multiple prior frames are greater than the preset ratio threshold, the matching strategy is to use the SimOTA method, including:
将大于预设比值阈值对应的多个先验框按照匹配代价(cost)从小到大排列，匹配代价(cost)根据预测结果和真实结果计算出的IoU损失与分类损失相加得到；Arrange multiple a priori frames corresponding to the preset ratio threshold in ascending order of matching cost (cost), and the matching cost (cost) is obtained by adding the IoU loss and classification loss calculated according to the prediction result and the real result;
IoU(Intersection over Union)是一种测量在特定数据集中检测相应物体准确度的一个标准。IoU是两个区域重叠的部分除以两个区域的集合部分得出的结果，通过设定的阈值，与这个IoU计算结果比较。例如，将根据特征数据预测得到的预测框与真实框进行IoU比较，获取前10的最大的IoU的预测结果，将10个IoU的值进行加和操作后会得到一个值，这个值就是正样本数量，最小取1。IoU (Intersection over Union) is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. IoU is the result obtained by dividing the overlapping part of the two regions by the set part of the two regions, and is compared with the IoU calculation result through the set threshold. For example, compare the IoU of the prediction frame predicted according to the feature data with the real frame, obtain the top 10 largest IoU prediction results, and add the 10 IoU values to get a value, which is the positive sample The minimum value is 1.
根据先验框的排列顺序依次筛选出预设数量阈值的先验框作为正样本。According to the arrangement order of the priori boxes, the priori boxes with a preset number threshold are sequentially screened out as positive samples.
在本发明实施例中，模型训练结束后，通过实验分析对比模型对农业病虫害图像的识别精度、识别耗时及模型实时性能。使用Precision-Recall曲线、AP(检测精度)、mAP(AP值在所有类别下的均值)评价模型精度，其中AP值即为P-R曲线与坐标轴围成区域面积，mAP是所有类的AP平均值，P-R曲线中召回率(R)的公式和精准率(P)的公式如下所示：In the embodiment of the present invention, after the model training is completed, the recognition accuracy, recognition time consumption and real-time performance of the model for agricultural diseases and insect pest images are analyzed and compared through experiments. Use the Precision-Recall curve, AP (detection accuracy), mAP (average of AP values under all categories) to evaluate the accuracy of the model, where the AP value is the area enclosed by the P-R curve and the coordinate axes, and mAP is the average value of AP for all categories , the formulas of the recall rate (R) and the precision rate (P) in the P-R curve are as follows:
其中TP为被判为正类的正类，FN为被判为负类的正类，FP为被判为正类的负类，利用模型预测模块对测试集中的病虫害图像进行预测效果对比分析，获取最优检测模型。Among them, TP is the positive class judged as the positive class, FN is the positive class judged as the negative class, and FP is the negative class judged as the positive class, and the model prediction module is used to predict the pest images in the test set. Comparative analysis to obtain the optimal detection model.
本发明实施例提供的农业病虫害检测方法通过对模型训练样本集进行筛选，提高模型的学习性能，从而有利于提高对农业病虫害检测的准确性。The method for detecting agricultural diseases and insect pests provided by the embodiments of the present invention improves the learning performance of the model by screening the model training sample set, thereby helping to improve the accuracy of detecting agricultural diseases and insect pests.
下面对本发明提供的农业病虫害检测装置进行描述，下文描述的农业病虫害检测装置与上文描述的农业病虫害检测方法可相互对应参照。The agricultural disease and insect pest detection device provided by the present invention is described below, and the agricultural disease and insect pest detection device described below and the agricultural disease and insect pest detection method described above can be referred to in correspondence.
图6为本发明实施例提供的农业病虫害检测装置的示意图，如图6所示，本发明实施例提供的农业病虫害检测装置包括：Figure 6 is a schematic diagram of an agricultural pest detection device provided by an embodiment of the present invention. As shown in Figure 6, the agricultural pest detection device provided by an embodiment of the present invention includes:
输入模块601，用于将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；
输出模块602，用于根据基于YOLO v4的农业病虫害检测模型得到检测结果；基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据所述引导头输出预测结果和辅助头输出预测结果加权求和得到。The
本发明实施例提供的农业病虫害检测装置通过将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；根据基于YOLO v4的农业病虫害检测模型得到检测结果；基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据引导头输出预测结果和辅助头输出预测结果加权求和得到，通过增加辅助头，可以增加备选分类结果，避免漏检，提高农业病虫害检测的准确性。The agricultural disease and insect pest detection device provided by the embodiment of the present invention inputs the image of the agricultural disease and insect pest to be detected into the agricultural disease and insect pest detection model based on YOLO v4; the detection result is obtained according to the agricultural disease and insect pest detection model based on YOLO v4; the agricultural disease and insect pest detection model based on YOLO v4 includes For the guide head and the auxiliary head, the detection results are obtained based on the weighted summation of the output prediction results of the guide head and the output prediction results of the auxiliary head. By adding auxiliary heads, alternative classification results can be added to avoid missed detection and improve the accuracy of agricultural pest detection.
图7示例了一种电子设备的实体结构示意图，如图7所示，该电子设备可以包括：处理器(processor)710、通信接口(Communications Interface)720、存储器(memory)730和通信总线740，其中，处理器710，通信接口720，存储器730通过通信总线740完成相互间的通信。处理器710可以调用存储器730中的逻辑指令，以执行农业病虫害检测方法，该方法包括：将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；根据基于YOLO v4的农业病虫害检测模型得到检测结果；基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据引导头输出预测结果和辅助头输出预测结果加权求和得到。FIG. 7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 7, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730, and a
此外，上述的存储器730中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。In addition, the above-mentioned logic instructions in the
另一方面，本发明还提供一种非暂态计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现以执行上述各方法提供的农业病虫害检测方法，该方法包括：将待检测农业病虫害图像输入基于YOLO v4的农业病虫害检测模型；根据基于YOLO v4的农业病虫害检测模型得到检测结果；基于YOLO v4的农业病虫害检测模型中包括引导头和辅助头，检测结果根据引导头输出预测结果和辅助头输出预测结果加权求和得到。On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to perform the method for detecting agricultural diseases and insect pests provided by the above-mentioned methods, the method includes : Input the image of agricultural diseases and insect pests to be detected into the agricultural diseases and insect pests detection model based on YOLO v4; obtain the detection results according to the agricultural diseases and insect pests detection model based on YOLO v4; The head output prediction result and the auxiliary head output prediction result are weighted and summed.
以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性的劳动的情况下，即可以理解并实施。The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative effort.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行各个实施例或者实施例的某些部分所述的方法。Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims (12)
Translated fromChinese
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211338472.9ACN115546186B (en) | 2022-10-28 | 2022-10-28 | Agricultural pest detection method and device based on YOLO v4 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211338472.9ACN115546186B (en) | 2022-10-28 | 2022-10-28 | Agricultural pest detection method and device based on YOLO v4 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115546186Atrue CN115546186A (en) | 2022-12-30 |
| CN115546186B CN115546186B (en) | 2023-07-14 |
Family
ID=84718052
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211338472.9AActiveCN115546186B (en) | 2022-10-28 | 2022-10-28 | Agricultural pest detection method and device based on YOLO v4 |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115546186B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170307889A1 (en)* | 2016-02-23 | 2017-10-26 | Compedia Software and Hardware Development Ltd. | Vision-assisted input within a virtual world |
| US11176384B1 (en)* | 2020-09-18 | 2021-11-16 | XMotors.ai Inc. | Apparatus, system and method for object detection around vehicle and application of same |
| US20210366144A1 (en)* | 2020-05-21 | 2021-11-25 | Verizon Connect Ireland Limited | Systems and methods for utilizing a deep learning model to determine vehicle viewpoint estimations |
| CN114220035A (en)* | 2021-12-23 | 2022-03-22 | 中科合肥智慧农业协同创新研究院 | Rapid pest detection method based on improved YOLO V4 |
- 2022
- 2022-10-28CNCN202211338472.9Apatent/CN115546186B/enactiveActive
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170307889A1 (en)* | 2016-02-23 | 2017-10-26 | Compedia Software and Hardware Development Ltd. | Vision-assisted input within a virtual world |
| US20210366144A1 (en)* | 2020-05-21 | 2021-11-25 | Verizon Connect Ireland Limited | Systems and methods for utilizing a deep learning model to determine vehicle viewpoint estimations |
| US11176384B1 (en)* | 2020-09-18 | 2021-11-16 | XMotors.ai Inc. | Apparatus, system and method for object detection around vehicle and application of same |
| CN114220035A (en)* | 2021-12-23 | 2022-03-22 | 中科合肥智慧农业协同创新研究院 | Rapid pest detection method based on improved YOLO V4 |
Non-Patent Citations (1)
| Title |
|---|
| 郭守向等: "Ｙｏｌｏ－Ｃ：基于单阶段网络的 Ｘ光图像违禁品检测", 《／激光与光电子学进展》, vol. 58, no. 8, pages 1 - 10* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115546186B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110837870B (en) | Sonar image target recognition method based on active learning | |
| CN114758288B (en) | Power distribution network engineering safety control detection method and device | |
| US11436493B2 (en) | Chromosome recognition method based on deep learning | |
| CN108345911B (en) | Surface defect detection method of steel plate based on multi-level features of convolutional neural network | |
| CN109492561B (en) | Optical remote sensing image ship detection method based on improved YOLO V2 model | |
| CN114565048B (en) | Three-stage pest image recognition method based on adaptive feature fusion pyramid network | |
| CN111898432B (en) | Pedestrian detection system and method based on improved YOLOv3 algorithm | |
| CN111291826B (en) | A pixel-by-pixel classification method for multi-source remote sensing images based on correlation fusion network | |
| CN112819821B (en) | A kind of nuclear image detection method | |
| CN115546187A (en) | Agricultural pest detection method and device based on YOLO v5 | |
| CN114399780A (en) | Table detection method, table detection model training method and device | |
| WO2024032010A1 (en) | Transfer learning strategy-based real-time few-shot object detection method | |
| CN117576381B (en) | Target detection training method, electronic device and computer readable storage medium | |
| CN112465821A (en) | Multi-scale pest image detection method based on boundary key point perception | |
| WO2023116565A1 (en) | Method for intelligently designing network security architecture diagram | |
| CN116994287B (en) | Animal counting method and device and animal counting equipment | |
| CN115359264A (en) | Intensive distribution adhesion cell deep learning identification method | |
| CN117371511A (en) | Training method, device, equipment and storage medium for image classification model | |
| CN111291820B (en) | A Target Detection Method Combining Location Information and Classification Information | |
| CN112418358A (en) | Vehicle multi-attribute classification method for strengthening deep fusion network | |
| CN111368625B (en) | Pedestrian target detection method based on cascade optimization | |
| CN112183336A (en) | Expression recognition model training method and device, terminal equipment and storage medium | |
| CN111860601A (en) | Method and device for predicting macrofungal species | |
| CN110675382A (en) | Identification method of superheat degree of aluminum electrolysis based on CNN-LapsELM | |
| CN115546186A (en) | Agricultural pest detection method and device based on YOLO v4 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |