CN113114934B - Multi-focus video acquisition method and system for urine red blood cells - Google Patents
Multi-focus video acquisition method and system for urine red blood cellsDownload PDFInfo
- Publication number
- CN113114934B CN113114934BCN202110344671.XACN202110344671ACN113114934BCN 113114934 BCN113114934 BCN 113114934BCN 202110344671 ACN202110344671 ACN 202110344671ACN 113114934 BCN113114934 BCN 113114934B
- Authority
- CN
- China
- Prior art keywords
- image
- video
- definition
- sharpness
- dct coefficient
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000003743erythrocyteAnatomy0.000titleclaimsabstractdescription42
- 238000000034methodMethods0.000titleclaimsabstractdescription28
- 210000002700urineAnatomy0.000titleclaimsdescription46
- 238000011156evaluationMethods0.000claimsabstractdescription52
- 238000004364calculation methodMethods0.000claimsabstractdescription34
- 230000008859changeEffects0.000claimsabstractdescription26
- 230000002485urinary effectEffects0.000claimsabstractdescription6
- 239000011159matrix materialSubstances0.000claimsdescription66
- 238000013473artificial intelligenceMethods0.000abstractdescription2
- 230000002159abnormal effectEffects0.000description10
- 230000000877morphologic effectEffects0.000description10
- 239000006059cover glassSubstances0.000description6
- 238000010586diagramMethods0.000description6
- 230000008569processEffects0.000description6
- 238000004458analytical methodMethods0.000description5
- 210000004027cellAnatomy0.000description4
- 208000017169kidney diseaseDiseases0.000description4
- 238000012545processingMethods0.000description4
- 238000011160researchMethods0.000description4
- 238000013480data collectionMethods0.000description3
- 230000009466transformationEffects0.000description3
- 238000003745diagnosisMethods0.000description2
- 230000006872improvementEffects0.000description2
- 238000001228spectrumMethods0.000description2
- 230000000007visual effectEffects0.000description2
- 230000009286beneficial effectEffects0.000description1
- 238000012512characterization methodMethods0.000description1
- 230000007812deficiencyEffects0.000description1
- 238000002474experimental methodMethods0.000description1
- 238000012986modificationMethods0.000description1
- 230000004048modificationEffects0.000description1
- 238000004321preservationMethods0.000description1
- 238000012360testing methodMethods0.000description1
- 238000012800visualizationMethods0.000description1
Images
Classifications
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- Signal Processing (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Investigating Or Analysing Biological Materials (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本发明属于人工智能辅助医疗技术领域，具体涉及一种尿红细胞多焦距视频采集方法与系统。The invention belongs to the technical field of artificial intelligence assisted medical treatment, and in particular relates to a method and a system for collecting urine erythrocyte multifocal length video.
背景技术Background technique
肾病患者的临床表现通常包括尿液的变化，一般情况下通过显微镜可观察到肾病患者的尿液样本中除了存在正常的尿红细胞外，还存在着大量由肾小球和其他组织挤压形成的异常的尿红细胞。医生通常通过收集和观察患者尿液样本来诊断患者是否患有肾脏疾病。其中，一些异常的尿红细胞在显微镜的不同焦距下会呈现出不同的形态特征，甚至在一定焦距下可能出现与其他类型的异常尿红细胞形态特征相似的形态特征。对于此类异常尿红细胞的识别，如果观察时显微镜焦距不合适，那么极易将这类异常尿红细胞识别错误。此外，部分异常尿红细胞彼此间的细胞形态特征也极其相似，识别时极易混淆。The clinical manifestations of patients with nephropathy usually include changes in urine. Generally, in the urine samples of patients with nephropathy, in addition to the normal urine red blood cells, there are a large number of glomeruli and other tissues squeezed. Abnormal urine red blood cells. Doctors usually diagnose whether a patient has kidney disease by collecting and observing a sample of the patient's urine. Among them, some abnormal urinary erythrocytes will show different morphological features under different focal lengths of the microscope, and may even appear morphological features similar to other types of abnormal urinary erythrocytes under a certain focal length. For the identification of such abnormal urine red blood cells, if the focus of the microscope is not appropriate during observation, it is easy to misidentify such abnormal urine red blood cells. In addition, the cell morphological characteristics of some abnormal urine red blood cells are very similar to each other, and it is easy to be confused when identifying.
目前，已有公开发明专利提出了关于异常尿红细胞的分类统计方法与系统，并提出了显微镜下尿红细胞的多焦距视频数据的概念，但是所使用的视频数据为人工采集。人工采集数据通常存在着较多不可控因素，例如采集数据时外部对比环境的改变、相机的曝光度调整等问题。而在多焦距视频数据的采集过程中，出现的更重要的问题是人工采集数据时难以判断产生正确的异常尿红细胞特征时所对应的焦距，且判断标准具有主观性，无法获得统一标准的包含准确特征的异常尿红细胞的高清晰度的多焦距视频数据。At present, there have been published invention patents that propose methods and systems for classification and statistics of abnormal urine red blood cells, and the concept of multifocal video data of urinary red blood cells under a microscope, but the video data used is manually collected. There are usually many uncontrollable factors in manual data collection, such as changes in the external contrast environment and camera exposure adjustment when collecting data. In the process of collecting multi-focal length video data, the more important problem is that it is difficult to judge the focal length corresponding to the correct abnormal urine red blood cell characteristics when collecting data manually, and the judgment standard is subjective, and it is impossible to obtain the inclusion of a unified standard. High-definition multifocal video data for accurate characterization of abnormal urinary red blood cells.
在现有技术中，还未出现自动分析采集显微镜下尿红细胞的多焦距视频的视频数据采集设备与系统。因此，有必要提出一种能够快速实现智能分析、采集显微镜下尿红细胞的多焦距视频数据的视频数据采集设备与系统，为使用多焦距视频数据进行科学研究、医疗诊断的研究人员与医生提供具有统一标准的包含准确特征的多焦距视频数据，并提供视频中清晰度最高的帧，便于研究人员与医生进行分析、研究和诊断，有效减少专业医师的工作负担。In the prior art, there is no video data acquisition device and system that automatically analyzes and acquires a multi-focal length video of urine red blood cells under a microscope. Therefore, it is necessary to propose a video data acquisition device and system that can quickly realize intelligent analysis and collect multi-focal length video data of urine red blood cells under a microscope, so as to provide researchers and doctors with multi-focal length video data for scientific research and medical diagnosis. Unified standard multi-focal length video data containing accurate features, and provide the highest definition frame in the video, which is convenient for researchers and doctors to analyze, study and diagnose, effectively reducing the workload of professional doctors.
发明内容SUMMARY OF THE INVENTION
本发明克服现有技术存在的不足，所要解决的技术问题为：提供一种尿红细胞多焦距视频采集方法与系统，以提高尿红细胞多焦距视频采集的效率。The present invention overcomes the deficiencies in the prior art, and the technical problem to be solved is: providing a method and system for collecting urine erythrocyte multifocal length video, so as to improve the efficiency of urine erythrocyte multifocal length video collecting.
为了解决上述技术问题，本发明采用的技术方案为：一种尿红细胞多焦距视频采集方法，包括以下步骤：In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a method for collecting urine erythrocyte multifocal length video, comprising the following steps:
S1、连续调节显微镜焦距，采集样本的显微镜变焦视频；S1. Continuously adjust the focal length of the microscope, and collect the microscope zoom video of the sample;
S2、对变焦视频中各帧图像进行清晰度计算，所述清晰度计算采用改进的DCT的图像清晰度评价函数进行计算；S2, carry out definition calculation to each frame image in the zoom video, and the definition calculation adopts the image definition evaluation function of the improved DCT to calculate;
S3、根据计算得到清晰度值，绘制变焦视频的清晰度变化曲线；根据清晰度值变化曲线，确定视频采集的起点和终点，采集并保存显微镜变焦视频。S3. Draw the definition change curve of the zoom video according to the calculated definition value; determine the starting point and the end point of video acquisition according to the definition value change curve, and collect and save the microscope zoom video.
所述步骤S2具体包括如下步骤：The step S2 specifically includes the following steps:
S201、设置阈值；S201, setting a threshold;
S202、从DCT系数矩阵的左上角开始，计算DCT系数矩阵中各个元素的值，计算公式为；S202, starting from the upper left corner of the DCT coefficient matrix, calculate the value of each element in the DCT coefficient matrix, and the calculation formula is:
其中，f（i，j）表示图像中的一个像素点（i，j）对应的信号，μ，v分别表水平方向和垂直方向的频率，μ=0,1，……M-1，v=0,1,2，……N-1，M和N分别表示图像宽度与高度，F（μ，v）表示DCT系数矩阵块中第μ行，第v列元素的对应值；Among them,f (i, j) represents the signal corresponding to a pixel (i, j) in the image,μ ,v represent the frequency in the horizontal and vertical directions, respectively, μ=0,1,...M-1,v =0,1,2,...N-1, M and N represent the image width and height, respectively,F (μ ,v ) represents the corresponding value of theμ -th row and thev -th column element in the DCT coefficient matrix block;
S203、判断现有的DCT系数矩阵块中所有元素的均值是否大于阈值，若否，依次增加矩阵的行和列，通过步骤S202的公式，继续计算DCT系数；若是，则根据对应的DCT系数矩阵块，计算图像清晰度评价值G，计算公式为：S203, judge whether the mean value of all elements in the existing DCT coefficient matrix block is greater than the threshold value, if not, increase the row and column of the matrix in turn, and continue to calculate the DCT coefficient through the formula of step S202; if so, according to the corresponding DCT coefficient matrix block, calculate the image sharpness evaluation value G, the calculation formula is:
其中，m和n表示对应的DCT系数矩阵块的行数和列数。Among them, m and n represent the number of rows and columns of the corresponding DCT coefficient matrix block.
所述阈值为0.5±0.03。The threshold is 0.5±0.03.
所述步骤S203中，判断DCT系数矩阵块中所有元素的均值是否大于阈值时，从图像的M/3行或N/3行开始判断。In the step S203, when judging whether the mean value of all elements in the DCT coefficient matrix block is greater than the threshold value, the judgment starts from the M/3 row or the N/3 row of the image.
所述的一种尿红细胞多焦距视频采集方法中，确定视频采集的起点和终点时，在清晰度值变化曲线中，以曲线清晰度最大值点为分界线，分别找到曲线清晰度最大值点两侧对应的清晰度最小值点，将两侧的清晰度最小值点对应的图像焦距分别作为为采集视频的起点焦距和终止点焦距。In the method for multifocal video collection of urine red blood cells, when determining the starting point and the end point of the video collection, in the change curve of the sharpness value, the maximum point of the sharpness of the curve is used as the dividing line, and the maximum point of the sharpness of the curve is respectively found. For the minimum sharpness points on both sides, the image focal lengths corresponding to the minimum sharpness points on both sides are taken as the starting point focal length and the ending point focal length of the captured video, respectively.
此外，本发明还提供了一种尿红细胞多焦距视频采集系统，包括CCD工业相机、显微镜、变焦步进电机和控制单元，所述控制单元的输入端与所述CCD工业相机的输出端连接，输出端与所述变焦步进电机的控制端连接；所述控制单元用于驱动所述变焦步进电机调节显微镜焦距，还用于计算所述CCD工业相机的采集视频中各帧图像的清晰度获得清晰度变化曲线，并根据清晰度变化曲线确定视频采集的起点和终点，控制CCD工业相机进行显微镜变焦视频采集与保存。In addition, the present invention also provides a urine red blood cell multi-focal length video acquisition system, comprising a CCD industrial camera, a microscope, a zoom stepping motor and a control unit, the input end of the control unit is connected with the output end of the CCD industrial camera, The output end is connected with the control end of the zoom stepping motor; the control unit is used to drive the zoom stepping motor to adjust the focal length of the microscope, and is also used to calculate the sharpness of each frame image in the video collected by the CCD industrial camera Obtain the sharpness change curve, and determine the starting point and end point of video capture according to the sharpness change curve, and control the CCD industrial camera to capture and save the zoom video of the microscope.
所述控制单元计算各帧图像的清晰度的具体方法为：The specific method that the control unit calculates the sharpness of each frame of image is:
设置阈值；set threshold;
从DCT系数矩阵的左上角开始，计算DCT系数矩阵中各个元素的值，计算公式为；Starting from the upper left corner of the DCT coefficient matrix, calculate the value of each element in the DCT coefficient matrix, and the calculation formula is:
其中，f（i，j）表示图像中的一个像素点（i，j）对应的信号，μ，v分别表水平方向和垂直方向的频率，μ=0,1，……M-1，v=0,1,2，……N-1，M和N分别表示图像宽度与高度，F（μ，v）表示DCT系数矩阵块中第μ行，第v列元素的对应值；Among them,f (i, j) represents the signal corresponding to a pixel (i, j) in the image,μ ,v represent the frequency in the horizontal and vertical directions, respectively, μ=0,1,...M-1,v =0,1,2,...N-1, M and N represent the image width and height, respectively,F (μ ,v ) represents the corresponding value of theμ -th row and thev -th column element in the DCT coefficient matrix block;
判断现有的DCT系数矩阵块中所有元素的均值是否大于阈值，若否，依次增加矩阵的行和列，通过步骤S202的公式，继续计算DCT系数；若是，则根据对应的DCT系数矩阵块，计算图像清晰度评价值G，计算公式为：Determine whether the mean value of all elements in the existing DCT coefficient matrix block is greater than the threshold value, if not, increase the row and column of the matrix in turn, and continue to calculate the DCT coefficient through the formula of step S202; if so, then according to the corresponding DCT coefficient matrix block, Calculate the image sharpness evaluation value G, the calculation formula is:
其中，m和n表示对应的DCT系数矩阵块的行数和列数。Among them, m and n represent the number of rows and columns of the corresponding DCT coefficient matrix block.
所述阈值为0.5。The threshold is 0.5.
所述的一种尿红细胞多焦距视频采集系统中，控制模块确定视频采集的起点和终点时，在清晰度值变化曲线中，以曲线清晰度最大值点为分界线，分别找到曲线清晰度最大值点两侧对应的清晰度最小值点，将两侧的清晰度最小值点对应的图像焦距分别作为为采集视频的起点焦距和终止点焦距。In the described multi-focal length video collection system of urine red blood cells, when the control module determines the starting point and the end point of the video collection, in the change curve of the sharpness value, the maximum sharpness point of the curve is used as the dividing line to find the maximum sharpness value of the curve respectively. The minimum sharpness points corresponding to both sides of the value point, and the image focal lengths corresponding to the minimum sharpness points on both sides are taken as the starting focal length and the ending point focal length of the captured video respectively.
本发明与现有技术相比具有以下有益效果：Compared with the prior art, the present invention has the following beneficial effects:
本发明提供了一种尿红细胞的多焦距视频采集方法与系统，通过对基于DCT的图像清晰度评价函数的改进，在保持图像清晰度评价值相对不变的情况下提高了算法判断图像清晰度的速率，实现对图像的实时评价，利用图像的评价函数进行多焦距视频采集，提高了采集的精度和效率。本发明使用肾病患者的尿液样本在显微镜下的多焦距视频作为分析数据，将改进后的算法在人工采集的质量较好的尿红细胞多焦距视频数据集上进行测试，每个视频数据的图像序列共包含图像64张，每张数字图像像素均为1200×1600，通常的基于DCT的图像清晰度评价函数中的DCT系数矩阵的维数为1200×1600维，改进后的算法需要计算的DCT系数矩阵的维数则减少为450×600维，对整个视频数据的64张图像进行清晰度评价的速率从5帧/秒增加到31帧/秒，整个速率增加了近6倍，且从附图5可以看出，改进后的算法得到的对图像清晰度的评价值与基于DCT的图像清晰度评价函数得到的评价值基本保持一致，也就是说，改进后的算法在与原算法对图像的评价值基本保持一致的同时大幅度提高了对图像清晰度的评价速率。而在本发明中使用的CCD工业相机录制视频的速率为30帧/秒，改进后的算法的评价图片清晰度的速率与其基本一致，因此，本发明可以实现对图像进行实时的清晰度评价，为研究人员与医生提供具有统一标准的包含数据准确特征的多焦距视频数据，有效减少专业医师的工作负担，便于分析、研究和诊断。The invention provides a multi-focal length video acquisition method and system for urine red blood cells. By improving the image sharpness evaluation function based on DCT, the algorithm judges the image sharpness while keeping the image sharpness evaluation value relatively unchanged. It realizes the real-time evaluation of the image, and uses the evaluation function of the image for multi-focal length video acquisition, which improves the accuracy and efficiency of the acquisition. The invention uses the multi-focal length video of the urine sample of the kidney disease patient under the microscope as the analysis data, and tests the improved algorithm on the artificially collected multi-focal length video data set of urine red blood cells with better quality. The sequence contains 64 images in total, and the pixels of each digital image are 1200×1600. The dimension of the DCT coefficient matrix in the usual DCT-based image sharpness evaluation function is 1200×1600. The improved algorithm needs to calculate the DCT The dimension of the coefficient matrix is reduced to 450 × 600 dimensions, and the rate of sharpness evaluation for 64 images of the entire video data is increased from 5 frames/second to 31 frames/second, and the entire rate has increased nearly 6 times. As can be seen from Figure 5, the evaluation value of image clarity obtained by the improved algorithm is basically consistent with the evaluation value obtained by the image clarity evaluation function based on DCT. The evaluation value of the image is basically the same, and the evaluation rate of image sharpness is greatly improved. The CCD industrial camera used in the present invention records video at a rate of 30 frames per second, and the improved algorithm evaluates the image clarity at a rate basically the same. Therefore, the present invention can realize real-time image clarity evaluation. Provide researchers and doctors with multi-focal length video data with a unified standard containing accurate data features, effectively reducing the workload of professional doctors, and facilitating analysis, research and diagnosis.
附图说明Description of drawings
图1为本发明实施例中异形尿红细胞在显微镜下随着显微镜焦距的变化所呈现出的细胞形态特征变化图；Fig. 1 is the cell morphological characteristic change diagram presented under the microscope with the change of microscope focal length in the embodiment of the present invention the heteromorphic urine red blood cell;
图2为对尿红细胞图像的像素值矩阵进行DCT处理后得到的DCT系数矩阵的可视化灰度图像展示；Fig. 2 is a visual grayscale image display of the DCT coefficient matrix obtained after the pixel value matrix of the urine red blood cell image is subjected to DCT processing;
图3为本发明提出的改进的基于DCT图像清晰度评价函数的算法示意图；3 is a schematic diagram of an improved algorithm based on the DCT image sharpness evaluation function proposed by the present invention;
图4为改进的基于DCT图像清晰度评价函数的流程图；Fig. 4 is the flow chart of the improved DCT image sharpness evaluation function based on;
图5为基于DCT的图像清晰度评价函数与改进后的基于DCT的图像清晰度函数（IDCT）在人工采集的显微镜下尿红细胞的多焦距视频上的应用时得到的清晰度值对比；Figure 5 is a comparison of the sharpness values obtained when the DCT-based image sharpness evaluation function and the improved DCT-based image sharpness function (IDCT) are applied to the multifocal video of urine red blood cells under a microscope;
图6为本发明实施例中得到的改进后的基于DCT的图像清晰度函数（IDCT）的清晰度值变化曲线示意图；6 is a schematic diagram of a change curve of sharpness value of an improved DCT-based image sharpness function (IDCT) obtained in an embodiment of the present invention;
图7为本发明提出的智能分析、采集显微镜下尿红细胞的多焦距视频数据的设备及流程示意图。FIG. 7 is a schematic diagram of a device and a flow chart for intelligently analyzing and collecting multi-focal length video data of urine red blood cells under a microscope proposed by the present invention.
图中各序号的代指组件为：1-CCD工业相机，2-显微镜调焦轮，3-变焦步进电机，4-显微镜物镜，5-样本盖玻片，6-显微镜载物台，7-PC端。The components of each serial number in the figure are: 1-CCD industrial camera, 2-microscope focusing wheel, 3-zoom stepping motor, 4-microscope objective lens, 5-sample cover glass, 6-microscope stage, 7-microscope stage -PC side.
具体实施方式Detailed ways
为使本发明实施例的目的、技术方案和优点更加清楚，下面将对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明的一部分实施例，而不是全部的实施例；基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not All the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work, all belong to the protection scope of the present invention.
实施例一Example 1
本发明实施例一提供了一种尿红细胞多焦距视频采集方法，包括以下步骤：
S1、连续调节显微镜焦距，采集样本的显微镜变焦视频。S1. Continuously adjust the focal length of the microscope, and collect a microscope zoom video of the sample.
S2、对变焦视频中各帧图像进行清晰度计算，所述清晰度计算采用改进的DCT的图像清晰度评价函数进行计算。S2. Perform sharpness calculation on each frame of the image in the zoom video, where the sharpness calculation is performed by using the improved DCT image sharpness evaluation function.
本实施例中，判断图像清晰度部分使用的图像清晰度评价函数是改进的基于离散余弦变化（DCT）的图像清晰度评价函数。In this embodiment, the image sharpness evaluation function used in the image sharpness determination part is an improved discrete cosine change (DCT)-based image sharpness evaluation function.
视频数据是指连续的图像序列，其实质上是由一组连续的图像构成的，图1为本发明实施例中异常尿红细胞在显微镜下随着显微镜焦距的变化所呈现出的细胞形态特征变化图。因此，视频数据的采集、处理问题最终可简化为对于图像的采集、处理问题。而图像的清晰度是衡量图像质量优劣的重要指标，清晰度高的图像能够给人较好的主观感受。在显微镜下，图像形态特征的清晰度通常会随着焦距的变化经历从模糊到清晰再到模糊的过程。因此，在采集显微镜下尿红细胞的多焦距视频数据时，可以通过判断图像清晰度来确定录制视频数据的起始点与终点相对应的焦距与时刻。Video data refers to a continuous image sequence, which is essentially composed of a group of continuous images. Figure 1 shows the changes in cell morphological characteristics of abnormal urine red blood cells under the microscope with the change of the focal length of the microscope in the embodiment of the present invention. picture. Therefore, the problem of video data collection and processing can finally be simplified to the problem of image collection and processing. The clarity of the image is an important indicator to measure the quality of the image, and the image with high clarity can give people a better subjective feeling. Under the microscope, the sharpness of image morphological features typically undergoes a process from blurred to sharp to blurred as the focal length changes. Therefore, when collecting multi-focal length video data of urine red blood cells under the microscope, the focal length and time corresponding to the start point and end point of the recorded video data can be determined by judging the image clarity.
基于以上分析，本发明在显微镜下尿红细胞的多焦距视频数据采集设备时设置了判断图像清晰度的部分，对显微镜下尿红细胞的图像进行质量评价，便于采集图像质量高的视频数据。本实施例中，利用改进的基于DCT的图像清晰度评价函数，实现对图像清晰度的快速实时评价。基于DCT的图像清晰度评价函数是最常用的图像清晰度评价函数之一。对于一幅像素为M×N的图像，基于DCT的图像清晰度评价函数为：Based on the above analysis, the present invention provides a part for judging the image clarity in the multi-focal length video data acquisition device of urine red blood cells under the microscope, and evaluates the quality of the urine red blood cell image under the microscope, which is convenient for collecting video data with high image quality. In this embodiment, the improved DCT-based image sharpness evaluation function is used to realize fast real-time evaluation of image sharpness. The image sharpness evaluation function based on DCT is one of the most commonly used image sharpness evaluation functions. For an image with M×N pixels, the image sharpness evaluation function based on DCT is:
其中，G表示基于DCT的图像清晰度评价值，M,N分别表示图像宽度与高度，μ，v分别表示水平方向和垂直方向的频率，F是M×N维的离散余弦变换DCT的变换系数矩阵。通常来说，F（μ，v）的计算公式为：Among them, G represents the image sharpness evaluation value based on DCT, M and N represent the image width and height respectively,μ andv represent the frequency in the horizontal and vertical directions, respectively, andF is the transformation coefficient of the M×N-dimensional discrete cosine transform DCT. matrix. Generally speaking, the formula for calculatingF (μ ,v ) is:
其中，f（i，j）表示图像中的一个像素点（i，j）对应的信号，μ=0,1，……M-1，v=0,1,2，……N-1，当μ=v=0时，F（0，0）表示图像的直流分量/直流系数。其中，f（i，j）表示图像中的一个像素点（i，j）对应的信号，μ，v分别表示水平方向和垂直方向的频率，μ=0,1，……M-1，v=0,1,2，……N-1，M和N分别表示图像宽度与高度，F（μ，v）表示DCT系数矩阵块中第μ行，第v列元素的对应值。Among them,f (i, j) represents the signal corresponding to a pixel (i, j) in the image, μ=0,1,...M-1,v =0,1,2,...N-1, Whenμ =v = 0,F (0, 0) represents the DC component/DC coefficient of the image. Among them,f (i, j) represents the signal corresponding to a pixel (i, j) in the image,μ ,v represent the frequency in the horizontal and vertical directions, respectively, μ=0,1,...M-1,v =0,1,2,...N-1, M and N represent the image width and height, respectively,F (μ ,v ) represents the corresponding value of theμ -th row andv -th column elements in the DCT coefficient matrix block.
一般来说，通过对图像进行DCT处理获得的位于变换系数矩阵中最左上角的直流系数具有最大幅度；以直流系数为出发点，向下、向右的交流系数距离直流系数越远，频率越高，幅度值越小。换句话说，在DCT系数矩阵中，直流系数和低频系数位于左上角，高频系数位于右下角，中频系数位于中间区域。由于人眼对低频数据的敏感度高于对高频数据敏感度，因此低频系数的变化对图像的视觉影响要远大于高频数据。从图2的DCT系数矩阵的灰度可视化图可观察到左上角区域的亮度明显高于其他区域。对于DCT来说，图像的主要能量集中在DCT系数矩阵的一小部分，也就是说，大多数图像信息集中于直流系数及其附近的低频频谱中，远离直流系数的交流系数几乎不包含任何图像信息，甚至仅包含杂散波。同时根据现有的清晰度评价函数G的计算公式（1），可发现在计算清晰度评价函数时，需要对图像的DCT的变换系数矩阵的F（μ，v）进行逐个计算，对于像素值大的图片该过程计算量过大，难以实现对图像的实时评价。因此，本发明实施例中，对于算法的核心改进在于基于DCT变换系数矩阵的特点对其计算过程进行改进以减少算法计算量，提高算法的计算速率，实现对图像清晰度的实时评价。Generally speaking, the DC coefficient located in the upper left corner of the transformation coefficient matrix obtained by performing DCT processing on the image has the largest amplitude; taking the DC coefficient as the starting point, the farther the downward and right AC coefficients are from the DC coefficient, the higher the frequency. , the smaller the amplitude value. In other words, in the DCT coefficient matrix, the DC coefficients and low frequency coefficients are in the upper left corner, the high frequency coefficients are in the lower right corner, and the intermediate frequency coefficients are in the middle region. Since the human eye is more sensitive to low-frequency data than to high-frequency data, changes in low-frequency coefficients have a far greater visual impact on images than high-frequency data. From the grayscale visualization of the DCT coefficient matrix in Figure 2, it can be observed that the brightness of the upper left area is significantly higher than that of other areas. For DCT, the main energy of the image is concentrated in a small part of the DCT coefficient matrix, that is, most of the image information is concentrated in the DC coefficient and its nearby low-frequency spectrum, and the AC coefficient far from the DC coefficient hardly contains any image. information, even just containing stray waves. At the same time, according to the existing calculation formula (1) of the sharpness evaluation function G, it can be found that when calculating the sharpness evaluation function, it is necessary to calculateF (μ ,v ) of the transformation coefficient matrix of the DCT of the image one by one. For large images, this process requires too much calculation, and it is difficult to realize real-time evaluation of images. Therefore, in the embodiment of the present invention, the core improvement of the algorithm is to improve its calculation process based on the characteristics of the DCT transform coefficient matrix to reduce the calculation amount of the algorithm, increase the calculation rate of the algorithm, and realize real-time evaluation of image clarity.
由于DCT变换系数矩阵上显示大部分图像信息集中于矩阵最左上角的直流系数及其附近的低频频谱上，因此本发明实施例提出通过阈值的设置来确定DCT系数矩阵中保留图像主要信息的矩阵块，减少DCT系数矩阵的计算量，进而减少整个评价函数的计算时间。在计算DCT系数矩阵时，从左上角开始依次按行、列计算DCT系数，每次分别按行、列计算DCT变换系数，即DCT系数矩阵中的元素的值，然后计算该次得到的矩阵块的各个元素的均值。若计算到m行n列对应的DCT系数矩阵块时，m×n维的DCT系数矩阵的均值达到某个阈值时，则可停止计算。图3以7×7维的方阵为例，给出了改进的计算DCT系数矩阵的计算过程示意图。Since most of the image information displayed on the DCT transform coefficient matrix is concentrated on the DC coefficient in the upper left corner of the matrix and the low-frequency spectrum near it, the embodiment of the present invention proposes to determine the matrix that retains the main image information in the DCT coefficient matrix by setting a threshold. block, reduce the calculation amount of the DCT coefficient matrix, and then reduce the calculation time of the entire evaluation function. When calculating the DCT coefficient matrix, the DCT coefficients are calculated in rows and columns from the upper left corner, and the DCT transform coefficients are calculated by row and column each time, that is, the values of the elements in the DCT coefficient matrix, and then the matrix block obtained this time is calculated. the mean of each element of . If the DCT coefficient matrix block corresponding to m rows and n columns is calculated, and the mean value of the m×n-dimensional DCT coefficient matrix reaches a certain threshold, the calculation can be stopped. FIG. 3 shows a schematic diagram of an improved calculation process for calculating a DCT coefficient matrix by taking a 7×7-dimensional square matrix as an example.
如图4所示，为本发明实施例中，进行图像清晰度计算的流程示意图。具体地，本实施例中，步骤S2具体包括如下步骤：As shown in FIG. 4 , it is a schematic flowchart of image definition calculation in an embodiment of the present invention. Specifically, in this embodiment, step S2 specifically includes the following steps:
S201、设置阈值；S201, setting a threshold;
S202、从DCT系数矩阵的左上角开始，计算DCT系数矩阵中各个元素的值，计算公式为式（2）；S202, starting from the upper left corner of the DCT coefficient matrix, calculate the value of each element in the DCT coefficient matrix, and the calculation formula is formula (2);
S203、判断现有的DCT系数矩阵块中所有元素的均值是否大于阈值，若否，依次增加矩阵的行和列，通过步骤S202的公式，继续计算DCT系数；若是，则根据对应的DCT系数矩阵块，计算图像清晰度评价值G，计算公式为：S203, judge whether the mean value of all elements in the existing DCT coefficient matrix block is greater than the threshold value, if not, increase the row and column of the matrix in turn, and continue to calculate the DCT coefficient through the formula of step S202; if so, according to the corresponding DCT coefficient matrix block, calculate the image sharpness evaluation value G, the calculation formula is:
其中，m和n表示对应的DCT系数矩阵块的行数和列数，即均值达到阈值时，对应的DCT系数矩阵块的行数和列数。Wherein, m and n represent the number of rows and columns of the corresponding DCT coefficient matrix block, that is, the number of rows and columns of the corresponding DCT coefficient matrix block when the mean value reaches the threshold.
步骤S203中，计算DCT系数矩阵的元素时，行列的增加方式可以为，每次增加一行或者一列，并且，行列交替增加，进行计算DCT系数矩阵，也可以按照图像的宽高比的比例进行行列的交替增加；此外，循环计算DCT系数矩阵时，也可以直接计算到图像的特定行或者特定列以后，再开始判断DCT系数矩阵块中所有元素的均值是否大于阈值，例如，可以直接计算到m=M/3行或m=N/3行再开始判断。In step S203, when calculating the elements of the DCT coefficient matrix, the way of increasing the rows and columns may be, adding one row or one column at a time, and the rows and columns are increased alternately to calculate the DCT coefficient matrix, or the rows and columns may be calculated according to the ratio of the aspect ratio of the image. In addition, when calculating the DCT coefficient matrix cyclically, it can also be directly calculated to a specific row or column of the image, and then start to judge whether the mean value of all elements in the DCT coefficient matrix block is greater than the threshold, for example, it can be directly calculated to m =M/3 lines or m=N/3 lines and then start the judgment.
通过多次计算实验得出阈值为0.5（±0.03）时实现了在不影响图像清晰度评价函数的对图像评价值的前提下大幅度提升了评价速率。经过改进，DCT系数矩阵的维数由之前的(M×N)维减少到(m×n)维，计算清晰度时，计算量从(M×N)减少到(m×n)，大大减少了图像清晰度的计算量，节省了计算时间，提高了图像评价速率，使得可以进行图像实时评价。Through multiple calculation experiments, it is found that when the threshold is 0.5 (±0.03), the evaluation rate is greatly improved without affecting the image evaluation value of the image sharpness evaluation function. After improvement, the dimension of the DCT coefficient matrix is reduced from the previous (M×N) dimension to (m×n) dimension. When calculating the clarity, the calculation amount is reduced from (M×N) to (m×n), which is greatly reduced. The calculation amount of the image clarity is reduced, the calculation time is saved, and the image evaluation rate is improved, so that the real-time image evaluation can be performed.
如图5所示，改进后的图像清晰度函数对图像的评价值与基于DCT的图像评价函数对图像的评价值几乎一致。As shown in Fig. 5, the evaluation value of the image by the improved image sharpness function is almost the same as the evaluation value of the image by the DCT-based image evaluation function.
S3、根据计算得到清晰度值，绘制变焦视频的清晰度变化曲线；根据清晰度变化曲线，确定视频采集的起点和终点，采集并保存显微镜变焦视频。S3. Draw the definition change curve of the zoom video according to the calculated definition value; determine the starting point and the end point of video acquisition according to the definition change curve, and collect and save the microscope zoom video.
具体地，如图6所示，为本发明实施例中获得的清晰度值变化曲线示意图；确定视频采集的起点和终点时，确定方法可以为：在清晰度值变化曲线中，以曲线清晰度最大值点（A点）为分界线，分别找到曲线清晰度最大值点两侧对应的清晰度最小值点（B点和C点），将两个部分的清晰度最小值对应的图像焦距分别作为为采集视频的起点焦距和终止点焦距。由于在显微镜下，图像形态特征的清晰度通常会随着焦距的变化经历从模糊到清晰再到模糊的过程，且采集的尿红细胞的多焦距视频数据旨在获得清晰度高的可以看到准确形态特征的多焦距视频数据，因此本发明选取能够包含图像形态特征变化的焦距范围作为录制视频的焦距范围。而在图像清晰度评价值变化曲线中，图像的清晰度值表示了图像的清晰程度，起始点和终止点在图像清晰度评价值变化曲线中体现为以曲线清晰度最大值对应的点为界将横轴划分的两个区间的清晰度最小值对应的图像焦距，因此，通过上述方法确定录制视频的起点和终点，可以在保证视频图像形态特征的前提下，减小采集数据，提高采集效率。Specifically, as shown in FIG. 6 , it is a schematic diagram of the change curve of the sharpness value obtained in the embodiment of the present invention; when determining the starting point and the end point of the video capture, the determination method may be: The maximum point (point A) is the dividing line. Find the minimum sharpness points (point B and point C) corresponding to both sides of the maximum sharpness point of the curve, respectively. As the starting focal length and ending point focal length for capturing video. Because under the microscope, the sharpness of image morphological features usually undergoes a process from blurred to clear to blurred with the change of focal length, and the multifocal video data of the collected urine red blood cells are designed to obtain high-definition images that can be seen accurately. The multi-focal length video data of morphological features, so the present invention selects the focal length range that can include the change of the image morphological features as the focal length range of the recorded video. In the change curve of the image sharpness evaluation value, the sharpness value of the image represents the sharpness of the image, and the starting point and the end point in the image sharpness evaluation value change curve are represented by the point corresponding to the maximum sharpness value of the curve as the boundary The image focal length corresponding to the minimum sharpness value of the two intervals divided by the horizontal axis. Therefore, by determining the start and end points of the recorded video by the above method, the collected data can be reduced and the collection efficiency can be improved on the premise of ensuring the morphological characteristics of the video image. .
实施例二
如图7所示，本发明实施例二提供了一种尿红细胞多焦距视频采集系统，包括CCD工业相机1、显微镜、变焦步进电机3和控制单元7，所述控制单元7的输入端与所述CCD工业相机1的输出端连接，输出端与所述变焦步进电机3的控制端连接；所述控制单元7用于驱动所述变焦步进电机3调节显微镜焦距，还用于计算所述CCD工业相机1的采集视频中各帧图像的清晰度获得清晰度变化曲线，并将清晰度变化曲线中斜率最大时对应的图像焦距作为为采集视频的起点焦距，曲线斜率最小时所对应的图像焦距作为采集视频的终止点焦距，控制CCD工业相机1进行显微镜变焦视频采集与保存。As shown in FIG. 7 , the second embodiment of the present invention provides a multi-focal length video acquisition system for urine red blood cells, which includes a CCD
采集时，首先由操作员将尿液样本盖玻片5固定在显微镜载物台6上，随后变焦步进电机将3带动显微镜的显微镜调焦轮2使得显微镜物镜4从与尿液样本盖玻片5距离0.8mm的位置运动到与尿液样本盖玻片5距离1.5mm的位置，同时启动CCD工业相机1以30帧/秒的速度实时获取焦距变化时尿液样本的图像，并将获取的图像实时传输到PC端7使用改进的基于DCT的图像清晰度评价函数对该焦距下的样本图像的清晰度进行实时评价并记录，根据获得的图像清晰度评价值曲线明确多焦距视频录制的起始点和终止点。During collection, the operator first fixes the urine sample cover glass 5 on the
在该阶段结束后，PC端7确定了包含各类尿红细胞准确细胞特征的多焦距视频数据采集的起始点和终止点。随后，PC端7指挥变焦步进电机3控制显微镜调焦轮2使得显微镜物镜4从距尿液样本盖玻片5为1.5mm的位置运动到距尿液样本盖玻片5为0.8mm的位置，即开始反向变焦，同时CDD工业相机1从上一阶段的确定的起始点开始录制多焦距视频，到确定的终止点时停止录制。录制完成的显微镜变焦视频将通过千兆网线传输给PC端7进行保存。After the end of this stage, the PC terminal 7 determines the start point and the end point of the multi-focal length video data collection containing accurate cell characteristics of various types of urine red blood cells. Then, the PC terminal 7 instructs the
具体地，本实施例中，控制单元计算各帧图像的清晰度的具体方法为实施例一中的步骤S2所对应的方法。Specifically, in this embodiment, the specific method for the control unit to calculate the sharpness of each frame of image is the method corresponding to step S2 in the first embodiment.
综上所述，本发明提供了一种尿红细胞多焦距视频采集方法与系统，基于图像清晰度判别部分和确定焦距范围部分，而且，通过改进的清晰度评价算法，提高了图像评价的速度，实现对图像清晰度的快速实时评价。本发明通过对图像的智能分析获得视频的录制起点和终点，基于录制视频阶段获得需采集的显微镜下尿红细胞的多焦距视频数据，用以辅助专业医师对样本观察与分析，以及为相关领域研究人员提供稳定具有统一标准的显微镜下尿红细胞的多焦距视频数据，便于研究与应用。To sum up, the present invention provides a method and system for multi-focal length video acquisition of urine red blood cells, which is based on the image sharpness determination part and the focal length range determination part, and, through the improved sharpness evaluation algorithm, the speed of image evaluation is improved, Enables fast real-time evaluation of image clarity. The invention obtains the starting point and end point of video recording through intelligent analysis of the image, and obtains the multi-focal length video data of urine red blood cells under the microscope to be collected based on the recording video stage, which is used to assist professional physicians in observing and analyzing samples, and for research in related fields. The personnel provide stable multi-focal video data of urine red blood cells under the microscope with a unified standard, which is convenient for research and application.
最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.
Claims (7)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110344671.XACN113114934B (en) | 2021-03-31 | 2021-03-31 | Multi-focus video acquisition method and system for urine red blood cells |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110344671.XACN113114934B (en) | 2021-03-31 | 2021-03-31 | Multi-focus video acquisition method and system for urine red blood cells |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113114934A CN113114934A (en) | 2021-07-13 |
| CN113114934Btrue CN113114934B (en) | 2022-08-05 |
Family
ID=76712931
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110344671.XAActiveCN113114934B (en) | 2021-03-31 | 2021-03-31 | Multi-focus video acquisition method and system for urine red blood cells |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113114934B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114071129A (en)* | 2021-11-23 | 2022-02-18 | 杭州逗酷软件科技有限公司 | Test method, test equipment and computer storage medium |
| CN118688950A (en)* | 2024-08-23 | 2024-09-24 | 四川科莫生医疗科技有限公司 | A multi-focus image fusion microscope imaging method based on sputum slides |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2675330B1 (en)* | 1991-04-11 | 1995-12-15 | Telediffusion Fse | A METHOD AND SYSTEM FOR AUTOMATIC FOCUSING OF A CAMERA USING TCD. |
| US6590938B1 (en)* | 1999-09-30 | 2003-07-08 | Conexant Systems, Inc. | DCT domain conversion of a higher definition signal to lower definition signal |
| US6529634B1 (en)* | 1999-11-08 | 2003-03-04 | Qualcomm, Inc. | Contrast sensitive variance based adaptive block size DCT image compression |
| US7266150B2 (en)* | 2001-07-11 | 2007-09-04 | Dolby Laboratories, Inc. | Interpolation of video compression frames |
| KR20050041640A (en)* | 2003-10-31 | 2005-05-04 | 삼성전자주식회사 | Image photographing device and method |
| CN100399183C (en)* | 2006-07-10 | 2008-07-02 | 南京邮电大学 | Adaptive Autofocus Method in Digital Camera |
| CN101976340B (en)* | 2010-10-13 | 2013-04-24 | 重庆大学 | License plate positioning method based on compressed domain |
| CN102053339B (en)* | 2010-11-23 | 2014-04-16 | 天津市亚安科技股份有限公司 | Automatic focusing method |
| CN103888764B (en)* | 2014-03-14 | 2017-02-15 | 西安交通大学 | Self-adaptation compensation system and method for video compression distortion |
| CN106506953A (en)* | 2016-10-28 | 2017-03-15 | 山东鲁能智能技术有限公司 | The substation equipment image acquisition method of servo is focused on and is exposed based on designated area |
| CN108345084A (en)* | 2018-03-07 | 2018-07-31 | 北京理工大学 | It is a kind of to lead the zoom automatic focusing method passively combined and system |
| CN108682008B (en)* | 2018-05-07 | 2019-04-16 | 江苏康尚生物医疗科技有限公司 | A kind of Leukocyte Image clarity evaluation method and device |
| CN111047577B (en)* | 2019-12-12 | 2021-02-26 | 太原理工大学 | Abnormal urine red blood cell classification statistical method and system |
| CN112433339B (en)* | 2020-12-10 | 2022-04-15 | 济南国科医工科技发展有限公司 | Microscope fine focusing method based on random forest |
- 2021
- 2021-03-31CNCN202110344671.XApatent/CN113114934B/enactiveActive
Also Published As
| Publication number | Publication date |
|---|---|
| CN113114934A (en) | 2021-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3848891A1 (en) | Computer vision-based method and system for monitoring speed of intestinal lens removal in real time | |
| CN113114934B (en) | Multi-focus video acquisition method and system for urine red blood cells | |
| CN113781455B (en) | Cervical cell image anomaly detection method, device, equipment and medium | |
| WO2021147429A1 (en) | Endoscopic image display method, apparatus, computer device, and storage medium | |
| KR102155381B1 (en) | Method, apparatus and software program for cervical cancer decision using image analysis of artificial intelligence based technology | |
| CN103198501B (en) | A kind of tooth panoramic picture automatic reconstruction | |
| CN110738658B (en) | Image quality evaluation method | |
| CN106990518A (en) | A kind of blood film self-focusing micro imaging method | |
| CN110751636A (en) | A method for detecting retinal arteriosclerosis in fundus images based on an improved codec network | |
| CN113237881B (en) | Detection method and device for specific cells and pathological section detection system | |
| CN111077153A (en) | Multifunctional sperm quality analysis system | |
| CN113241154A (en) | Artificial intelligent blood smear cell labeling system and method | |
| CN107543788A (en) | A kind of urine erythrocyte abnormal rate detection method and system | |
| CN116071337A (en) | Endoscopic image quality evaluation method based on super-pixel segmentation | |
| CN116548942A (en) | Non-contact heart rate variability detection method and system | |
| CN116725478A (en) | Analysis method of fundus refractive power and topography generation method | |
| KR20210033902A (en) | Method, apparatus and software program for cervical cancer diagnosis using image analysis of artificial intelligence based technology | |
| CN111754503A (en) | Enteroscope retroreduction overspeed ratio monitoring method based on two-channel convolutional neural network | |
| CN104966282B (en) | A kind of image-pickup method and system for single blood erythrocyte by mocro detection | |
| CN103955941A (en) | Corneal endothelial cell analysis meter and analysis method | |
| CN111507948A (en) | Automatic intercepting system and method for key images of ultrasonic video stream based on machine vision | |
| CN114565538A (en) | An endoscope image processing method, system, storage medium and device | |
| CN119453950B (en) | Tongue image feature extraction and health assessment method based on deep learning | |
| CN118229555B (en) | Image fusion method, device, equipment and computer readable storage medium | |
| Lin et al. | Stitching method for panoramic nail fold images based on capillary contour enhancement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |