CN114882093A - Intraoperative three-dimensional point cloud generation method and device and intraoperative local structure navigation method - Google Patents

Abstract

Translated fromChinese

本发明公开了一种术中三维点云生成方法、装置和一种术中局部结构导航方法，属于手术导航领域，术中三维点云生成方法步骤包括，获取白光图像和荧光图像；从白光图像中识别出生理组织的二维像素点；从荧光图像中识别出荧光点；荧光点为在生理组织上预先点出荧光凝胶的位置在荧光图像上的成像；根据荧光点的亮度计算荧光凝胶的物距；根据荧光点在二维像素点中的对应位置和荧光凝胶的物距，匹配得出二维像素点的三维点云，该方法无需借助3D镜头，只用普通荧光内窥镜就能生成针对生理组织的三维点云，本发明的术中局部结构导航方法计算量小，抗干扰能力强，弥补了术中导航柔性组织匹配差、难以实时的问题。

The invention discloses an intraoperative three-dimensional point cloud generation method, a device and an intraoperative local structure navigation method, belonging to the field of surgical navigation. The intraoperative three-dimensional point cloud generation method includes the following steps: acquiring a white light image and a fluorescence image; Identify the two-dimensional pixel points of the physiological tissue; identify the fluorescent point from the fluorescent image; the fluorescent point is the imaging on the fluorescent image of the position where the fluorescent gel is pre-pointed on the physiological tissue; calculate the fluorescent gel according to the brightness of the fluorescent point The object distance of the gel; according to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel, the three-dimensional point cloud of the two-dimensional pixel point is obtained by matching. The mirror can generate three-dimensional point cloud for physiological tissue. The intraoperative local structure navigation method of the present invention has small calculation amount and strong anti-interference ability, which makes up for the problems of poor matching of flexible tissue during intraoperative navigation and difficulty in real-time.

Description

Translated fromChinese

术中三维点云生成方法、装置和术中局部结构导航方法Intraoperative three-dimensional point cloud generation method, device and intraoperative local structure navigation method

技术领域technical field

本发明涉及一种术中三维点云生成方法、装置和一种术中局部结构导航方法，属于手术导航领域。The invention relates to an intraoperative three-dimensional point cloud generation method and device, and an intraoperative local structure navigation method, belonging to the field of surgical navigation.

背景技术Background technique

随着内窥镜技术及手术机器人技术的不断发展，手术导航在微创外科手术中越来越普及。精准的手术导航在复杂的微创外科手术中可以明显改善医生术中组织识别能力，优化手术方案，减少手术创伤。由于术前CT或MRI影像与术中内窥镜影像的匹配问题，柔性组织的匹配难度很高。因此，目前临床使用的手术导航系统主要应用于神经外科、骨科等组织刚性强的相关手术中，而少见用于肝胆外科、泌尿外科等柔性组织的手术。With the continuous development of endoscopic technology and surgical robot technology, surgical navigation has become more and more popular in minimally invasive surgery. Precise surgical navigation can significantly improve the ability of doctors to identify intraoperative tissues, optimize surgical plans, and reduce surgical trauma in complex minimally invasive surgical operations. Due to the matching problem between preoperative CT or MRI images and intraoperative endoscopic images, the matching of flexible tissues is very difficult. Therefore, the currently used surgical navigation systems are mainly used in neurosurgery, orthopedics and other related surgeries with strong tissue rigidity, and are rarely used for flexible tissue surgeries such as hepatobiliary surgery and urology.

为了改善柔性组织术中导航的效果，目前逐渐发展出了术前三维重建模型图像与术中组织三维重建图像配准的技术，在手术初期，组织与模型的匹配度较高时有一定的效果，可以将组织内部病灶在目标组织中的位置显示出来。但在术中，组织被切割或摆动后，组织与模型差异较大，导航误差变大。In order to improve the effect of intraoperative navigation of flexible tissue, the technology of registering the preoperative 3D reconstructed model image and the intraoperative 3D reconstructed image of the tissue has been gradually developed. , which can display the position of the internal lesions in the target tissue. However, during the operation, after the tissue is cut or swung, the difference between the tissue and the model is large, and the navigation error becomes larger.

发明内容SUMMARY OF THE INVENTION

为了克服现有技术的不足，本发明提供一种术中三维点云生成方法和装置，并提供一种术中局部结构导航方法。In order to overcome the deficiencies of the prior art, the present invention provides an intraoperative three-dimensional point cloud generation method and device, and an intraoperative local structure navigation method.

本发明解决其技术问题所采用的技术方案是：The technical scheme adopted by the present invention to solve its technical problems is:

第一方面，本申请提供一种术中三维点云生成方法，包括以下步骤：In a first aspect, the present application provides an intraoperative 3D point cloud generation method, comprising the following steps:

获取白光图像和在预设激光强度照射下的荧光图像；所述白光图像和所述荧光图像由荧光内窥镜采集而得；acquiring a white light image and a fluorescent image under the irradiation of a preset laser intensity; the white light image and the fluorescent image are acquired by a fluorescent endoscope;

从所述白光图像中识别出生理组织的二维像素点；Identifying two-dimensional pixel points of physiological tissue from the white light image;

从所述荧光图像中识别出荧光点；所述荧光点为在生理组织上预先点出荧光凝胶的位置在所述荧光图像上的成像；Identify the fluorescent point from the fluorescent image; the fluorescent point is the imaging on the fluorescent image of the position where the fluorescent gel is pre-marked on the physiological tissue;

根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距；Calculate the object distance of the fluorescent gel according to the brightness of the fluorescent spot in the fluorescent image;

根据所述荧光点在所述二维像素点中的对应位置和所述荧光凝胶的物距，匹配得出至少部分所述二维像素点的三维点云。According to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel, a three-dimensional point cloud of at least part of the two-dimensional pixel point is obtained by matching.

其中，物距是指荧光凝胶到内窥镜镜头的距离。在特定条件下，荧光图像中，荧光凝胶成像得到的荧光点的亮度只与物距有关。特定条件是指，激光强度一定、荧光凝胶的厚度一定、荧光凝胶中荧光染料的浓度一定。对于光圈可调的荧光内窥镜，还要求光圈一定。The object distance refers to the distance from the fluorescent gel to the endoscope lens. Under certain conditions, in fluorescence images, the brightness of fluorescent spots obtained by fluorescent gel imaging is only related to the object distance. The specific conditions mean that the laser intensity is constant, the thickness of the fluorescent gel is constant, and the concentration of the fluorescent dye in the fluorescent gel is constant. For a fluorescent endoscope with an adjustable aperture, a certain aperture is also required.

本申请提供的术中三维点云生成方法无需利用3D镜头就能生成三维点云，荧光凝胶可随柔性组织移动，能方便地更新柔性组织的位置和形态。The intraoperative 3D point cloud generation method provided by the present application can generate a 3D point cloud without using a 3D lens, the fluorescent gel can move with the flexible tissue, and the position and shape of the flexible tissue can be conveniently updated.

进一步地，所述根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距的步骤包括：Further, the step of calculating the object distance of the fluorescent gel according to the brightness of the fluorescent dots in the fluorescent image includes:

根据所述荧光点在所述荧光图像中的亮度和预制的亮度-物距关系计算所述荧光凝胶的物距；Calculate the object distance of the fluorescent gel according to the brightness of the fluorescent dots in the fluorescent image and a prefabricated brightness-object distance relationship;

预制所述亮度-物距关系的步骤包括：The step of prefabricating the brightness-object distance relationship includes:

在一平面上点出具有预设厚度且含有预设浓度荧光染料的标准凝胶；Point out a standard gel with a preset thickness and containing a preset concentration of fluorescent dyes on a plane;

用荧光内窥镜在不同物距下拍摄同一预设激光强度照射下的所述标准凝胶得到多个标准图像；photographing the standard gel under the irradiation of the same preset laser intensity with a fluorescence endoscope at different object distances to obtain a plurality of standard images;

从所述标准图像中识别出标准荧光点；identifying standard fluorescent spots from the standard image;

获取所述标准荧光点的标准亮度；obtaining the standard brightness of the standard fluorescent spot;

关联所述标准亮度与所述物距得到所述亮度-物距关系。The brightness-object distance relationship is obtained by correlating the standard brightness and the object distance.

点出标准凝胶的平面可以是载波板，用荧光内窥镜在不同物距下拍摄同一预设激光强度照射下的所述标准凝胶时，不同物距的间隔越小，即物距取得越密集，后续得到的亮度-物距关系更加准确。The plane on which the standard gel is pointed can be a carrier plate. When using a fluorescence endoscope to photograph the standard gel under the irradiation of the same preset laser intensity at different object distances, the smaller the interval between different object distances is, that is, the object distance is obtained. The denser it is, the more accurate the subsequently obtained brightness-object distance relationship is.

进一步地，所述亮度-物距关系为记录有与所述标准亮度对应的所述物距的数据库，具体可以是亮度与物距对照的表格，调用亮度-物距关系时，直接查找数据库，读取与荧光亮度对应的距离。Further, the brightness-object distance relationship is a database in which the object distance corresponding to the standard brightness is recorded, specifically, a table comparing brightness and object distance, when calling the brightness-object distance relationship, the database is directly searched, Read the distance corresponding to the fluorescence brightness.

所述关联所述标准亮度与所述物距得到所述亮度-物距关系的步骤包括：The step of correlating the standard brightness and the object distance to obtain the brightness-object distance relationship includes:

拟合出有关物距的亮度函数。Fits a luminance function with respect to object distance.

亮度-物距关系为拟合函数的情况下，更便于测量未记录到亮度与物距关系的中间值，可以减少事前以不同物距拍摄标准凝胶时不同物距的取值间隔。When the relationship between brightness and object distance is a fitting function, it is more convenient to measure the intermediate value of the relationship between brightness and object distance, which can reduce the value interval of different object distances when shooting standard gels with different object distances in advance.

优选地，所述有关物距的亮度函数为多项式函数。Preferably, the luminance function related to the object distance is a polynomial function.

第二方面，本申请提供一种术中局部结构导航方法，包括以下步骤：In a second aspect, the present application provides an intraoperative local structure navigation method, comprising the following steps:

根据术前CT（电子计算机断层扫描）结果或MRI（核磁共振成像）结果对目标组织进行三维重建，得到三维模型；According to the preoperative CT (Computed Tomography) results or MRI (Magnetic Resonance Imaging) results, three-dimensional reconstruction of the target tissue is performed to obtain a three-dimensional model;

按照第一方面所述的术中三维点云生成方法得到三维点云；Obtain a 3D point cloud according to the intraoperative 3D point cloud generation method described in the first aspect;

将所述三维点云与所述三维模型进行点云配准performing point cloud registration on the 3D point cloud with the 3D model

根据配准结果改变所述三维模型的形态。The shape of the three-dimensional model is changed according to the registration result.

本申请提供的术中局部结构导航方法计算2D图像中单点荧光与相机之间的距离（即物距），求得了深度，实现不用3D摄像头、不用外置红外内窥镜跟踪就可以为任意目标生成三维点云，弥补了当前内窥镜的术中导航组织柔性形变匹配差，术中导航难以实时的问题，对于肝胆外科等柔性组织，可实现术中局部组织的准确实时导航。The intraoperative local structure navigation method provided in this application calculates the distance (ie, the object distance) between the single-point fluorescence and the camera in the 2D image, obtains the depth, and realizes the tracking without a 3D camera or an external infrared endoscope. The target generates a 3D point cloud, which makes up for the poor matching of the flexible deformation of the current intraoperative navigation of the endoscopic tissue, and the difficulty of real-time intraoperative navigation. For flexible tissues such as hepatobiliary surgery, accurate real-time intraoperative navigation of local tissues can be achieved.

进一步地，所述目标组织为血管，所述在生理组织上预先点出荧光凝胶的位置包括：Further, the target tissue is a blood vessel, and the position where the fluorescent gel is pre-pointed on the physiological tissue includes:

（1）、血管分叉点处；(1) At the bifurcation point of blood vessels;

（2）、血管弯曲形成夹角小于120°的夹角处；(2) The blood vessel is bent to form an angle less than 120°;

（3）、血管上特定相邻荧光凝胶的中间，所述特定相邻荧光凝胶为血管上已存在的且间距大于50mm的相邻荧光凝胶。(3) The middle of specific adjacent fluorescent gels on the blood vessel, where the specific adjacent fluorescent gels are adjacent fluorescent gels that already exist on the blood vessel and whose spacing is greater than 50 mm.

选择这些位置能以较少的荧光凝胶反映血管的结构和形态，在血管的位置发生变化后仍能准确地与三维模型配准，有利于减少术中导航的数据量。Selecting these positions can reflect the structure and shape of blood vessels with less fluorescent gel, and can still accurately register with the 3D model after the position of the blood vessels changes, which is beneficial to reduce the amount of data for intraoperative navigation.

优选地，所述将所述三维点云与所述三维模型进行点云配准时，采用ICP配准算法直接进行配准。Preferably, when performing point cloud registration on the three-dimensional point cloud and the three-dimensional model, the ICP registration algorithm is used to perform the registration directly.

优选地，所述将所述三维点云与所述三维模型进行点云配准时，先采用刚性配准方法进行粗配准，再使用ICP方法进行细配准。Preferably, when performing point cloud registration on the three-dimensional point cloud and the three-dimensional model, a rigid registration method is used for rough registration first, and then an ICP method is used for fine registration.

第三方面，本申请提供一种术中三维点云生成装置，配合荧光内窥镜使用，包括：In a third aspect, the present application provides an intraoperative three-dimensional point cloud generation device, used in conjunction with a fluorescent endoscope, including:

获取模块，用于获取白光图像和在预设激光强度照射下的荧光图像；an acquisition module for acquiring white light images and fluorescent images under the irradiation of preset laser intensity;

组织识别模块，用于从所述白光图像中识别出生理组织的二维像素点；a tissue identification module for identifying two-dimensional pixel points of physiological tissue from the white light image;

荧光点识别模块，用于从所述荧光图像中识别出荧光点；所述荧光点为在生理组织上预先点出荧光凝胶的位置在所述荧光图像上的成像；a fluorescent spot identification module, used for identifying the fluorescent spot from the fluorescent image; the fluorescent spot is the imaging on the fluorescent image of the position of the fluorescent gel pre-marked on the physiological tissue;

深度计算模块，用于根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距；a depth calculation module, configured to calculate the object distance of the fluorescent gel according to the brightness of the fluorescent point in the fluorescent image;

匹配模块，用于根据所述荧光点在所述二维像素点中的对应位置和所述荧光凝胶的物距，匹配得出至少部分所述二维像素点的三维点云。The matching module is configured to obtain a three-dimensional point cloud of at least part of the two-dimensional pixel points by matching according to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel.

该装置可整合于荧光内窥镜的图像处理单元中，只增加计算，无需增设光学测距硬件，能为使用荧光内窥镜实施术中导航提供条件。The device can be integrated into the image processing unit of the fluoroscopic endoscope, and only adds calculation without adding optical ranging hardware, and can provide conditions for the use of the fluoroscopic endoscope to implement intraoperative navigation.

本发明的有益效果是：本发明的术中三维点云生成方法无需借助3D镜头，只用普通荧光内窥镜就能生成针对生理组织的三维点云，本发明的术中局部结构导航方法对普通2D荧光内窥镜都适用，计算量小，抗干扰能力强，荧光凝胶可随柔性组织移动，能方便地更新柔性组织的位置和形态，弥补了当前内窥镜的术中导航组织柔性形变匹配差，术中导航难以实时的问题，对于肝胆外科等柔性组织，可实现术中局部组织的准确实时导航。The beneficial effects of the present invention are as follows: the intraoperative 3D point cloud generation method of the present invention does not require the aid of a 3D lens, and only a common fluorescent endoscope can generate a 3D point cloud for physiological tissues, and the intraoperative local structure navigation method of the present invention is suitable for It is suitable for ordinary 2D fluorescent endoscopes, with a small amount of calculation and strong anti-interference ability. The fluorescent gel can move with the flexible tissue, which can easily update the position and shape of the flexible tissue, making up for the flexibility of the current endoscopic intraoperative navigation tissue. Due to poor deformation matching and difficult real-time intraoperative navigation, for flexible tissues such as hepatobiliary surgery, accurate and real-time intraoperative navigation of local tissues can be achieved.

本申请的其他特征和优点将在随后的说明书阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请了解。本申请的目的和其他优点可通过在所写的说明书以及附图中所特别指出的结构来实现和获得。Other features and advantages of the present application will be set forth in the description that follows, and, in part, will be apparent from the description, or learned by practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.

附图说明Description of drawings

图1是本申请实施例提供的一种术中三维点云生成方法的流程图。FIG. 1 is a flowchart of an intraoperative 3D point cloud generation method provided by an embodiment of the present application.

图2是本申请实施例提供的一种术中局部结构导航方法的流程图。FIG. 2 is a flowchart of an intraoperative local structure navigation method provided by an embodiment of the present application.

图3是血管的三维形态图。Fig. 3 is a three-dimensional morphological view of a blood vessel.

具体实施方式Detailed ways

下面详细描述本发明的实施方式，所述实施方式的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本发明，而不能理解为对本发明的限制。Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention.

下文的公开提供了许多不同的实施方式或例子用来实现本发明的不同结构。为了简化本发明的公开，下文中对特定例子的部件和设置进行描述。当然，它们仅仅为示例，并且目的不在于限制本发明。此外，本发明可以在不同例子中重复参考数字和/或参考字母，这种重复是为了简化和清楚的目的，其本身不指示所讨论各种实施方式和/或设置之间的关系。The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are only examples and are not intended to limit the invention. Furthermore, the present disclosure may repeat reference numerals and/or reference letters in different instances for the purpose of simplicity and clarity and not in itself indicative of a relationship between the various embodiments and/or arrangements discussed.

目前术中导航的方案有：手动融合方案、基于规划路径的匹配方案、影像配准方案。The current intraoperative navigation schemes include: manual fusion scheme, matching scheme based on planned path, and image registration scheme.

其中，通用影像配准方案细分有两种：Among them, there are two subdivisions of general image registration schemes:

第一种，提取内窥镜组织影像2D轮廓，与三维模型投影的2D轮廓做配准。The first is to extract the 2D contour of the endoscopic tissue image and register it with the 2D contour projected by the 3D model.

第二种，利用双目内窥镜来生成目标组织三维点云，与三维模型的点云来配准。The second is to use a binocular endoscope to generate a three-dimensional point cloud of the target tissue, which is registered with the point cloud of the three-dimensional model.

第二种方案用到的内窥镜体积大、硬件成本高，数据量大。第一种方案数据量比第二种方案少，但提取得到的轮廓仍需要用较多像素点来表示，在配准时要配准多个像素，数据量仍然较大，且在柔性组织被摆动、切割后，提取得到的轮廓与三维模型的投影相差较大，难以继续准确导航。The endoscope used in the second solution is large in size, high in hardware cost, and large in data volume. The amount of data in the first scheme is less than that in the second scheme, but the extracted contour still needs to be represented by more pixels. During registration, multiple pixels need to be registered, the amount of data is still large, and the flexible tissue is oscillated. , After cutting, the extracted contour is quite different from the projection of the 3D model, and it is difficult to continue to navigate accurately.

针对上述问题，本发明在目标组织的主要拓扑结构上点胶特制的荧光凝胶，利用荧光内窥镜在特定条件下的测距功能，获取荧光凝胶的在白光图像中的相对位置，利用荧光图像获取深度信息，然后与通过图像处理分割出的目标组织进行匹配，得到目标组织三维点云，三维点云与术前重建的三维模型配准融合，实现术中的局部结构导航。该方法对任意2D荧光内窥镜适用，且计算量小，抗干扰能力强，十分适用于术中局部结构导航。In view of the above problems, the present invention dispenses a special fluorescent gel on the main topological structure of the target tissue, uses the ranging function of the fluorescent endoscope under specific conditions to obtain the relative position of the fluorescent gel in the white light image, and uses The depth information is obtained from the fluorescence image, and then matched with the target tissue segmented by image processing to obtain a 3D point cloud of the target tissue. The 3D point cloud is registered and fused with the preoperatively reconstructed 3D model to achieve intraoperative local structure navigation. This method is applicable to any 2D fluorescence endoscope, and has a small amount of calculation and strong anti-interference ability, which is very suitable for intraoperative local structure navigation.

参照图1，本发明提供一种术中三维点云生成方法，包括以下步骤：1, the present invention provides an intraoperative 3D point cloud generation method, comprising the following steps:

S11：获取白光图像和在预设激光强度照射下的荧光图像。其中，白光图像和荧光图像由荧光内窥镜采集而得。荧光内窥镜会采集到荧光图像和白光图像，其中荧光图像经过滤光，只采集到荧光光波。显示器所显示的图像为荧光图像与白光图像融合后的图像。S11: Acquire a white light image and a fluorescence image under irradiation with a preset laser intensity. Among them, the white light image and the fluorescence image are acquired by the fluorescence endoscope. A fluorescence endoscope captures both a fluorescence image and a white light image, where the fluorescence image is filtered to capture only the fluorescence light waves. The image displayed by the monitor is the image after fusion of the fluorescent image and the white light image.

S12：从白光图像中识别出生理组织的二维像素点。具体是通过深度学习的方法，将对应生理组织像素从内窥镜白光图中分割出，得到生理组织的实际二维组织结构信息，相当于从白光图像中截取出包含目标组织的像素。例如采用UNet对目标组织图像进行预训练得到网络模型，在术中时，利用训练好的模型实时推理出目标组织所在像素，并将该像素提取出。S12: Identify the two-dimensional pixel points of the physiological tissue from the white light image. Specifically, through the deep learning method, the corresponding physiological tissue pixels are segmented from the endoscopic white light image to obtain the actual two-dimensional tissue structure information of the physiological tissue, which is equivalent to cutting out the pixels containing the target tissue from the white light image. For example, UNet is used to pre-train the target tissue image to obtain a network model. During the operation, the trained model is used to infer the pixel where the target tissue is located in real time, and the pixel is extracted.

S13：从荧光图像中识别出荧光点。荧光点为在生理组织上预先点出荧光凝胶的位置在荧光图像上的成像。识别出荧光点的方法有多种，例如大津法。该荧光凝胶在术中能够附着在血管上，随血管一并移动，在术后一定时间内可被组织吸收。S13: Fluorescent spots are identified from the fluorescent image. The fluorescent spot is the imaging on the fluorescent image of the position where the fluorescent gel is pre-spotted on the physiological tissue. There are various methods for identifying fluorescent dots, such as the Otsu method. The fluorescent gel can be attached to the blood vessel during the operation, move with the blood vessel, and can be absorbed by the tissue within a certain period of time after the operation.

S14：根据荧光点在荧光图像中的亮度计算荧光凝胶的物距。即求出荧光凝胶到内窥镜镜头的距离，反映图像的深度。S14: Calculate the object distance of the fluorescent gel according to the brightness of the fluorescent dots in the fluorescent image. That is, the distance from the fluorescent gel to the endoscope lens is obtained to reflect the depth of the image.

S15：根据荧光点在二维像素点中的对应位置和荧光凝胶的物距，匹配得出至少部分二维像素点的三维点云。例如，可通过B样条插值的方法匹配。S15: According to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel, match to obtain a three-dimensional point cloud of at least part of the two-dimensional pixel point. For example, it can be matched by means of B-spline interpolation.

本发明主通过在术中内窥镜图像中对局部血管进行实时跟踪，并与术前三维模型进行融合，可实时反馈局部血管与周围血管、肿瘤等关键组织之间的相对关系。步骤S12中识别出生理组织的原理与现有技术中提取内窥镜组织影像2D轮廓的原理相同，而本申请不止要轮廓，包含目标组织的所有像素都需要。荧光内窥镜的特点是，荧光图像和白光图像拍摄到的物体位置完全相同，只是颜色不同，而预先点出的荧光凝胶是医生点在目标生理组织上的，因此，荧光图像上的荧光点必然就是目标生理组织上的一点，而在特定条件下，荧光点的亮度反映该点的深度，因此根据荧光点在二维像素点中的对应位置即可匹配出所有荧光凝胶的三维点云，相当于目标组织的部分三维点云。此时就可舍弃步骤S12中没有荧光点可对应的二维像素点，在后续与三维模型配准时就可大幅减少数据量。The present invention mainly tracks the local blood vessels in intraoperative endoscopic images in real time and fuses them with the preoperative three-dimensional model, so that the relative relationship between the local blood vessels and surrounding blood vessels, tumors and other key tissues can be fed back in real time. The principle of identifying the physiological tissue in step S12 is the same as the principle of extracting the 2D contour of the endoscopic tissue image in the prior art, and the present application requires not only the contour, but also all pixels including the target tissue. The feature of fluorescence endoscope is that the position of the object captured by the fluorescent image and the white light image is exactly the same, but the color is different, and the pre-pointed fluorescent gel is pointed by the doctor on the target physiological tissue. Therefore, the fluorescence on the fluorescent image is different. The point must be a point on the target physiological tissue, and under certain conditions, the brightness of the fluorescent point reflects the depth of the point, so all the three-dimensional points of the fluorescent gel can be matched according to the corresponding position of the fluorescent point in the two-dimensional pixel point. cloud, which is equivalent to a partial 3D point cloud of the target tissue. At this time, the two-dimensional pixel points that have no corresponding fluorescent points in step S12 can be discarded, and the amount of data can be greatly reduced in the subsequent registration with the three-dimensional model.

在特定条件下利用荧光点亮度反映该点物距的原理如下：The principle of using the brightness of the fluorescent point to reflect the object distance of the point under specific conditions is as follows:

公知的荧光强度定量公式如式（1）。The well-known quantitative formula of fluorescence intensity is as formula (1).

其中，

为荧光强度，

为荧光量子产率，

为吸收光强度。in,

is the fluorescence intensity,

is the fluorescence quantum yield,

is the absorbed light intensity.

由朗伯-比耳定律可得：According to the Lambert-Beer law:

其中，

为摩尔吸光系数，与吸收物质和入射波长相关，

为吸收层厚度，c为吸收物质浓度，

为入射光强。in,

is the molar absorption coefficient, which is related to the absorbing substance and the incident wavelength,

is the thickness of the absorbing layer, c is the concentration of the absorbing substance,

is the incident light intensity.

当物质浓度很低时，有：When the substance concentration is very low, there are:

此时，当固定入射光强时，荧光强度与物质浓度成线性关系。从以上公式可以发现，对于同一种物质制成的厚度相同的材料，如ICG（吲哚菁绿）凝胶，当固定入射光强和物质浓度时，其荧光强度是固定的。At this time, when the incident light intensity is fixed, the fluorescence intensity has a linear relationship with the substance concentration. From the above formula, it can be found that for a material with the same thickness made of the same substance, such as ICG (indocyanine green) gel, when the incident light intensity and substance concentration are fixed, the fluorescence intensity is fixed.

荧光内窥镜系统中，在同样参数下，获取的荧光图像亮度值

主要受荧光强度

和相机与发出荧光点的目标位置之间的距离D（即物距）相关，可用以下公式表达：In the fluorescence endoscope system, under the same parameters, the acquired fluorescence image brightness value

Mainly affected by fluorescence intensity

It is related to the distance D (ie the object distance) between the camera and the target position of the fluorescent point, which can be expressed by the following formula:

若以“*”统一表示函数的变量，则g(*)的函数表达了荧光强度、相机与目标位置的距离、荧光图像亮度值

的转换函数。If the variables of the function are uniformly represented by "*", the function of g(*) expresses the fluorescence intensity, the distance between the camera and the target position, and the brightness value of the fluorescence image.

conversion function.

在ICG荧光内窥镜相机中，当激光强度和ICG浓度固定时，物质本身属性固定，根据公式（3）可知，荧光强度

将会是一个固定值。此时公式（6）可描述为：In the ICG fluorescence endoscopic camera, when the laser intensity and the ICG concentration are fixed, the properties of the substance itself are fixed. According to formula (3), it can be known that the fluorescence intensity

will be a fixed value. At this time, formula (6) can be described as:

即此时荧光图像亮度值（

）只与相机与目标位置的相对距离相关。That is, the brightness value of the fluorescence image at this time (

) is only related to the relative distance of the camera from the target location.

因此只要计算出g(*)的反函数

，即可以根据荧光图像得到荧光点到相机的距离，公式表达如下：So just calculate the inverse function of g(*)

, that is, the distance from the fluorescent point to the camera can be obtained from the fluorescent image, and the formula is as follows:

根据相机成像原理，同等环境下，相对距离与荧光图像亮度值接近线性关系，相应的

也为近似线性关系，但为了更准确计算出荧光点到相机的距离与荧光图像亮度值的关系，可以采用拟合逼近的方式，将

拟合出，具体方案为：According to the camera imaging principle, under the same environment, the relative distance and the fluorescence image brightness value are close to a linear relationship, and the corresponding

It is also an approximate linear relationship, but in order to more accurately calculate the relationship between the distance from the fluorescent point to the camera and the brightness value of the fluorescent image, a fitting approximation method can be used to calculate

Fitted out, the specific scheme is:

固定激光强度和ICG浓度，固定ICG点胶厚度，在载玻片上点胶一个ICG凝胶，调整相机与凝胶距离，以

表示拍摄第一张荧光图像时的距离，以

表示拍摄最后一张荧光图像时的距离，在相机距离凝胶

范围内，每隔

，拍摄一张荧光图像，共得到

张荧光图像。固定点胶厚度的方法为，用同一个点胶机和喷嘴，令挤出凝胶的时间相同以保持点胶的厚度相同。尽管如此，凝胶挤出后在一平面上或一弧面上会成半球状或圆台状，中间厚，边缘薄，因此获取亮度时最好取平均值。Fix the laser intensity and ICG concentration, fix the ICG dispensing thickness, dispense an ICG gel on the slide, and adjust the distance between the camera and the gel to

represents the distance at which the first fluorescence image was taken, with

Indicates the distance at which the last fluorescence image was taken, at the camera distance gel

range, every

, taking a fluorescence image to obtain a total of

fluorescence image. The method of fixing the dispensing thickness is to use the same dispenser and nozzle, so that the time of extruding the gel is the same to keep the thickness of the dispensing the same. However, after extrusion, the gel will be hemispherical or truncated on a flat or arc surface, with a thick center and thin edges, so it is better to take the average value when obtaining brightness.

对上述得到的每张荧光图像，可以通过大津阈值法将点胶位置对应的荧光像素分割出来，并计算荧光像素平均值

，作为当前距离下对应的荧光图像像素点亮度。For each fluorescent image obtained above, the fluorescent pixels corresponding to the dispensing position can be segmented by the Otsu threshold method, and the average value of the fluorescent pixels can be calculated.

, as the pixel brightness of the corresponding fluorescent image at the current distance.

最后，根据每个距离

（

的取值范围在

范围内）所对应的荧光图像像素点亮度

，可以通过多项式拟合的方式，得到拟合函数：Finally, according to each distance

(

The range of values is in

range) corresponding to the pixel brightness of the fluorescent image

, the fitting function can be obtained by polynomial fitting:

拟合函数f(*)可视为

，

越小，得到的亮度-物距关系越准确。The fitting function f(*) can be regarded as

,

The smaller the value, the more accurate the obtained luminance-object distance relationship.

参照图2，一种术中局部结构导航方法，包括以下步骤：2, an intraoperative local structure navigation method, comprising the following steps:

S0：根据术前CT结果或MRI结果对目标组织进行三维重建，得到三维模型。S0: Perform three-dimensional reconstruction of the target tissue according to the preoperative CT results or MRI results to obtain a three-dimensional model.

S1：按照本发明的术中三维点云生成方法得到三维点云。S1: According to the intraoperative 3D point cloud generation method of the present invention, a 3D point cloud is obtained.

S2：将三维点云与三维模型进行点云配准。S2: Register the 3D point cloud with the 3D model.

S3：根据配准结果改变三维模型的形态。S3: Change the shape of the three-dimensional model according to the registration result.

由于没有荧光点可对应的二维像素点被舍弃，步骤S2配准时数据量少，配准速度快，能够做到实时导航。另一方面，荧光点的现状和颜色显然与白光图像不同，特异性强，抗干扰能力强，即使柔性组织被摆动或被切开，只要荧光凝胶仍附着，在荧光图像中就依然存在荧光点，仍可继续准确导航。Since no two-dimensional pixel points corresponding to the fluorescent points are discarded, the amount of data during the registration in step S2 is small, the registration speed is fast, and real-time navigation can be achieved. On the other hand, the status and color of the fluorescent dots are obviously different from those of the white light image, with strong specificity and strong anti-interference ability. Even if the flexible tissue is wiggled or cut, as long as the fluorescent gel is still attached, there is still fluorescence in the fluorescent image. point, you can continue to navigate accurately.

对于较为简单（例如需要荧光凝胶数量10点以下）的组织结构中，步骤S2可以直接采用三维点云跟术前获取的三维模型进行点云配准，完成三维模型与当前图像的融合实现导航。例如，采用ICP配准算法直接进行点云配准。For a relatively simple tissue structure (for example, the number of fluorescent gels required is less than 10 points), step S2 can directly use the 3D point cloud to perform point cloud registration with the 3D model obtained before surgery, and complete the fusion of the 3D model and the current image to realize navigation. . For example, the ICP registration algorithm is used to directly perform point cloud registration.

对于复杂（例如需要荧光凝胶数量超过10点，如图3可视为复杂结构）的组织结构，步骤S2可以先利用点胶位置与三维模型上标注的点胶位置进行快速粗配准，再采用三维点云信息与完整的三维模型进行细配准，从而提高配准速度和配准精度。例如采用基于点的刚性配准方法，如经典的仿射变换配准方法作为粗配准方案，实现点云之间的大致位置匹配，完成后再利用ICP点云配准方法作为细配准方案，实现点云之间的绝对位置匹配。For complex tissue structures (for example, the number of fluorescent gels required exceeds 10 points, which can be regarded as a complex structure in Figure 3), step S2 can first use the dispensing position and the dispensing position marked on the 3D model to perform rapid rough registration, and then The 3D point cloud information is used for fine registration with the complete 3D model, thereby improving the registration speed and registration accuracy. For example, a point-based rigid registration method, such as the classical affine transformation registration method, is used as a coarse registration scheme to achieve approximate position matching between point clouds, and then the ICP point cloud registration method is used as a fine registration scheme. , to achieve absolute position matching between point clouds.

完成三维模型与目标组织的匹配关系计算后，采用常用图像处理方法，通过调整三维模型属性将三维模型在内窥镜图像中的特定位置进行图像融合。在经过荧光点胶后，对任意术中获取的荧光内窥镜图像，均可以通过上述流程进行术中实时导航。After completing the calculation of the matching relationship between the 3D model and the target tissue, a common image processing method is used to perform image fusion at a specific position of the 3D model in the endoscopic image by adjusting the attributes of the 3D model. After fluorescent dispensing, intraoperative real-time navigation can be performed for any intraoperative fluorescence endoscopic image obtained through the above process.

当目标组织为血管时，目标组织细长、柔软，且多分支，比其他生理组织更难以较少的荧光凝胶体现其主要拓扑结构，优选地，参照图3，在生理组织上预先点出荧光凝胶的位置包括：When the target tissue is a blood vessel, the target tissue is slender, soft, and multi-branched, which is more difficult than other physiological tissues to reflect its main topology. Preferably, referring to FIG. Fluorescent gel locations include:

（1）、血管分叉点处，如图3中的A点所指；(1) At the bifurcation point of blood vessels, as indicated by point A in Figure 3;

（2）、血管弯曲形成夹角小于120°的夹角顶点处，如图3中的B点所指；(2) The apex of the included angle formed by the curvature of the blood vessel is less than 120°, as indicated by point B in Figure 3;

（3）、血管上特定相邻荧光凝胶的中间，特定相邻荧光凝胶为血管上已存在的且间距大于50mm的相邻荧光凝胶，如图3的C点所指。需要注意的是，此处所说的“中间”仍在血管上，不能将荧光凝胶点在其他组织上。(3) In the middle of specific adjacent fluorescent gels on the blood vessel, the specific adjacent fluorescent gels are adjacent fluorescent gels that already exist on the blood vessel and whose spacing is greater than 50 mm, as indicated by point C in Figure 3 . It should be noted that the "middle" mentioned here is still on the blood vessel, and the fluorescent gel cannot be spotted on other tissues.

对于结构简单的普通目标血管，只需要少数几个荧光凝胶就可以根据二维的白光图像生成三维血管点云数据。For common target blood vessels with simple structure, only a few fluorescent gels are needed to generate 3D blood vessel point cloud data from 2D white light images.

更优选地，使得相邻点胶距离尽量小于30mm。这样的点胶位置可以在数据量较少的前提下，充分体现出目标血管的结构和形态，在目标血管发生形态变化后，方便追踪血管的形态变化，抗干扰能力强，使得像血管这样复杂、易变形的结构都能实现术中导航。More preferably, the distance between adjacent glue dispensing is as small as possible to be less than 30mm. Such a dispensing position can fully reflect the structure and shape of the target blood vessel under the premise of a small amount of data. After the target blood vessel changes in shape, it is convenient to track the morphological change of the blood vessel, and the anti-interference ability is strong, which makes the blood vessels as complex as blood vessels. , Deformable structure can achieve intraoperative navigation.

本申请还提供一种术中三维点云生成装置，配合荧光内窥镜使用，包括：The application also provides an intraoperative three-dimensional point cloud generation device, used in conjunction with a fluorescent endoscope, including:

获取模块，用于获取白光图像和在预设激光强度照射下的荧光图像；an acquisition module for acquiring white light images and fluorescent images under the irradiation of preset laser intensity;

组织识别模块，用于从白光图像中识别出生理组织的二维像素点；Tissue identification module, used to identify the two-dimensional pixel points of physiological tissue from the white light image;

荧光点识别模块，用于从荧光图像中识别出荧光点；荧光点为在生理组织上预先点出荧光凝胶的位置在荧光图像上的成像；The fluorescent point identification module is used to identify the fluorescent point from the fluorescent image; the fluorescent point is the imaging on the fluorescent image of the position where the fluorescent gel is pre-marked on the physiological tissue;

深度计算模块，用于根据荧光点在荧光图像中的亮度计算荧光凝胶的物距；The depth calculation module is used to calculate the object distance of the fluorescent gel according to the brightness of the fluorescent point in the fluorescent image;

匹配模块，用于根据荧光点在二维像素点中的对应位置和荧光凝胶的物距，匹配得出至少部分二维像素点的三维点云。The matching module is used for matching and obtaining a three-dimensional point cloud of at least part of the two-dimensional pixel points according to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel.

该装置可整合于荧光内窥镜的图像处理单元中，只增加计算，无需增设光学测距硬件，能为使用荧光内窥镜实施术中导航提供条件。The device can be integrated into the image processing unit of the fluoroscopic endoscope, and only adds calculation without adding optical ranging hardware, and can provide conditions for the use of the fluoroscopic endoscope to implement intraoperative navigation.

在本说明书的描述中，参考术语“一个实施方式”“某些实施方式”“示意性实施方式”“示例”“具体示例”、或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。In the description of this specification, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples", etc. The particular feature, structure, material or characteristic described by example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

以上所述是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也视为本发明的保护范围。The above are the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications may also be regarded as It is the protection scope of the present invention.

Claims (10)

Translated fromChinese

1.一种术中三维点云生成方法，其特征在于，包括以下步骤：1. an intraoperative three-dimensional point cloud generation method, is characterized in that, comprises the following steps:获取白光图像和在预设激光强度照射下的荧光图像；所述白光图像和所述荧光图像由荧光内窥镜采集而得；acquiring a white light image and a fluorescent image under the irradiation of a preset laser intensity; the white light image and the fluorescent image are acquired by a fluorescent endoscope;从所述白光图像中识别出生理组织的二维像素点；Identifying two-dimensional pixel points of physiological tissue from the white light image;从所述荧光图像中识别出荧光点；所述荧光点为在生理组织上预先点出荧光凝胶的位置在所述荧光图像上的成像；Identify the fluorescent point from the fluorescent image; the fluorescent point is the imaging on the fluorescent image of the position where the fluorescent gel is pre-marked on the physiological tissue;根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距；Calculate the object distance of the fluorescent gel according to the brightness of the fluorescent spot in the fluorescent image;根据所述荧光点在所述二维像素点中的对应位置和所述荧光凝胶的物距，匹配得出至少部分所述二维像素点的三维点云。According to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel, a three-dimensional point cloud of at least part of the two-dimensional pixel point is obtained by matching.2.根据权利要求1所述的术中三维点云生成方法，其特征在于，所述根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距的步骤包括：2 . The intraoperative three-dimensional point cloud generation method according to claim 1 , wherein the step of calculating the object distance of the fluorescent gel according to the brightness of the fluorescent points in the fluorescent image comprises: 3 .根据所述荧光点在所述荧光图像中的亮度和预制的亮度-物距关系计算所述荧光凝胶的物距；Calculate the object distance of the fluorescent gel according to the brightness of the fluorescent dots in the fluorescent image and a prefabricated brightness-object distance relationship;预制所述亮度-物距关系的步骤包括：The step of prefabricating the brightness-object distance relationship includes:在一平面上点出具有预设厚度且含有预设浓度荧光染料的标准凝胶；Point out a standard gel with a preset thickness and containing a preset concentration of fluorescent dyes on a plane;用荧光内窥镜在不同物距下拍摄同一预设激光强度照射下的所述标准凝胶得到多个标准图像；photographing the standard gel under the irradiation of the same preset laser intensity with a fluorescence endoscope at different object distances to obtain a plurality of standard images;从所述标准图像中识别出标准荧光点；identifying standard fluorescent spots from the standard image;获取所述标准荧光点的标准亮度；obtaining the standard brightness of the standard fluorescent spot;关联所述标准亮度与所述物距得到所述亮度-物距关系。The brightness-object distance relationship is obtained by correlating the standard brightness and the object distance.3.根据权利要求2所述的术中三维点云生成方法，其特征在于，所述亮度-物距关系为记录有与所述标准亮度对应的所述物距的数据库。3 . The intraoperative 3D point cloud generation method according to claim 2 , wherein the luminance-object distance relationship is a database in which the object distance corresponding to the standard luminance is recorded. 4 .4.根据权利要求2所述的术中三维点云生成方法，其特征在于，所述关联所述标准亮度与所述物距得到所述亮度-物距关系的步骤包括：4. The intraoperative 3D point cloud generation method according to claim 2, wherein the step of correlating the standard brightness and the object distance to obtain the brightness-object distance relationship comprises:拟合出有关物距的亮度函数。Fits a luminance function with respect to object distance.5.根据权利要求4所述的术中三维点云生成方法，其特征在于，所述有关物距的亮度函数为多项式函数。5 . The intraoperative 3D point cloud generation method according to claim 4 , wherein the luminance function related to the object distance is a polynomial function. 6 .6.一种术中局部结构导航方法，其特征在于，包括以下步骤：6. An intraoperative local structure navigation method, characterized in that, comprising the following steps:根据术前CT结果或MRI结果对目标组织进行三维重建，得到三维模型；3D reconstruction of the target tissue based on preoperative CT results or MRI results to obtain a 3D model;按照权利要求1至5任一所述的术中三维点云生成方法得到三维点云；Obtain a three-dimensional point cloud according to the intraoperative three-dimensional point cloud generation method described in any one of claims 1 to 5;将所述三维点云与所述三维模型进行点云配准；performing point cloud registration on the three-dimensional point cloud and the three-dimensional model;根据配准结果改变所述三维模型的形态。The shape of the three-dimensional model is changed according to the registration result.7.根据权利要求6所述的术中局部结构导航方法，其特征在于，所述目标组织为血管，所述在生理组织上预先点出荧光凝胶的位置包括：7 . The intraoperative local structure navigation method according to claim 6 , wherein the target tissue is a blood vessel, and the pre-pointing position of the fluorescent gel on the physiological tissue comprises: 8 .（1）、血管分叉点处；(1) At the bifurcation point of blood vessels;（2）、血管弯曲形成夹角小于120°的夹角处；(2) The blood vessel is bent to form an angle less than 120°;（3）、血管上特定相邻荧光凝胶的中间，所述特定相邻荧光凝胶为血管上已存在的且间距大于50mm的相邻荧光凝胶。(3) The middle of specific adjacent fluorescent gels on the blood vessel, where the specific adjacent fluorescent gels are adjacent fluorescent gels that already exist on the blood vessel and whose spacing is greater than 50 mm.8.根据权利要求6所述的术中局部结构导航方法，其特征在于，所述将所述三维点云与所述三维模型进行点云配准时，采用ICP配准算法直接进行配准。8 . The intraoperative local structure navigation method according to claim 6 , wherein when performing point cloud registration on the three-dimensional point cloud and the three-dimensional model, the registration is performed directly by using an ICP registration algorithm. 9 .9.根据权利要求6所述的术中局部结构导航方法，其特征在于，所述将所述三维点云与所述三维模型进行点云配准时，先采用刚性配准方法进行粗配准，再使用ICP方法进行细配准。9 . The intraoperative local structure navigation method according to claim 6 , wherein, when performing point cloud registration on the 3D point cloud and the 3D model, a rigid registration method is first used to perform rough registration, 10 . Then use the ICP method for fine registration.10.一种术中三维点云生成装置，其特征在于，配合荧光内窥镜使用，包括：10. An intraoperative three-dimensional point cloud generation device, characterized in that, used in conjunction with a fluorescent endoscope, comprising:获取模块，用于获取白光图像和在预设激光强度照射下的荧光图像；an acquisition module for acquiring white light images and fluorescent images under the irradiation of preset laser intensity;组织识别模块，用于从所述白光图像中识别出生理组织的二维像素点；a tissue identification module for identifying two-dimensional pixel points of physiological tissue from the white light image;荧光点识别模块，用于从所述荧光图像中识别出荧光点；所述荧光点为在生理组织上预先点出荧光凝胶的位置在所述荧光图像上的成像；a fluorescent spot identification module, used for identifying the fluorescent spot from the fluorescent image; the fluorescent spot is the imaging on the fluorescent image of the position where the fluorescent gel is pre-marked on the physiological tissue;深度计算模块，用于根据所述荧光点在所述荧光图像中的亮度计算所述荧光凝胶的物距；a depth calculation module, configured to calculate the object distance of the fluorescent gel according to the brightness of the fluorescent point in the fluorescent image;匹配模块，用于根据所述荧光点在所述二维像素点中的对应位置和所述荧光凝胶的物距，匹配得出至少部分所述二维像素点的三维点云。The matching module is configured to obtain a three-dimensional point cloud of at least part of the two-dimensional pixel points by matching according to the corresponding position of the fluorescent point in the two-dimensional pixel point and the object distance of the fluorescent gel.

Images