CN114418960A - Image processing method, system, computer equipment and storage medium - Google Patents

Abstract

The present application relates to an image processing method, system, computer device and storage medium. The method comprises the following steps: acquiring a real-time image of a target area, wherein the real-time image comprises a first target object and a second target object; acquiring a pre-generated standard image; matching and fusing the standard image and the real-time image according to the first target object to obtain a current image; and displaying the second target object in the current image according to the position of the second target object in the real-time image. By adopting the method, the image with the implant in the operation can be updated in real time, and the intra-operation navigation is further realized through the image collected in the operation.

Description

Translated fromChinese

图像处理方法、系统、计算机设备和存储介质Image processing method, system, computer equipment and storage medium

技术领域technical field

本申请涉及计算机技术领域，特别是涉及一种图像处理方法、系统、计算机设备和存储介质。The present application relates to the field of computer technology, and in particular, to an image processing method, system, computer device and storage medium.

背景技术Background technique

随着计算机技术的不断发展，在医疗器械领域中，逐渐出现了集诸多学科为一体的新型医疗器械手术机器人辅助设备。这些手术机器人已在全世界广泛应用，手术种类涵盖泌尿科、妇产科、心脏外科、胸外科、肝胆外科、胃肠外科、耳鼻喉科等学科。With the continuous development of computer technology, in the field of medical equipment, a new type of medical equipment surgical robot auxiliary equipment that integrates many disciplines has gradually emerged. These surgical robots have been widely used all over the world, covering urology, obstetrics and gynecology, cardiac surgery, thoracic surgery, hepatobiliary surgery, gastrointestinal surgery, otolaryngology and other disciplines.

在精准外科领域，也有着一些手术机器人来辅助外科医生进行手术，相较于传统的人工外科手术，采用手术机器人的方法进行的外科手术具有伤害小、出血少、恢复快等特点。In the field of precision surgery, there are also some surgical robots to assist surgeons in performing operations. Compared with traditional manual surgery, surgical operations using surgical robots have the characteristics of less injury, less bleeding, and faster recovery.

现有的手术机器人只能通过手术前收集的图像进行术中导航，不够智能化。Existing surgical robots can only perform intraoperative navigation through images collected before surgery, and are not intelligent enough.

发明内容SUMMARY OF THE INVENTION

基于此，有必要针对上述技术问题，提供一种能够将显示带有植入物的图像的图像处理方法、装置、计算机设备和存储介质。Based on this, it is necessary to provide an image processing method, apparatus, computer equipment and storage medium capable of displaying an image with an implant in view of the above technical problems.

一种图像处理方法，所述方法包括：An image processing method, the method comprising:

获取目标区域的实时图像，所述实时图像包括第一目标对象以及第二目标对象；获取预先生成的标准图像；acquiring a real-time image of the target area, the real-time image includes a first target object and a second target object; acquiring a pre-generated standard image;

根据所述第一目标对象将所述标准图像与所述实时图像进行匹配融合得到当前图像；According to the first target object, the standard image and the real-time image are matched and fused to obtain a current image;

根据所述第二目标对象在所述实时图像中的位置，将所述第二目标对象显示在所述当前图像中。The second target object is displayed in the current image according to the position of the second target object in the real-time image.

在其中一个实施例中，所述获取目标区域的实时图像，包括：获取扫描得到的目标区域的扫描图像，所述扫描图像中包括第一目标对象以及第二目标对象，且所述扫描图像携带有与第一参照物对应的第一位置，所述第一参照物用于标定所述第一目标对象；根据所述第一位置和所述第一目标对象对所述扫描图像进行重建得到实时图像，所述实时图像包括第一目标对象和第二目标对象。In one embodiment, the acquiring a real-time image of the target area includes: acquiring a scanned image of the target area obtained by scanning, the scanned image includes a first target object and a second target object, and the scanned image carries There is a first position corresponding to the first reference object, and the first reference object is used to calibrate the first target object; the scanned image is reconstructed according to the first position and the first target object to obtain real-time An image, the real-time image includes a first target object and a second target object.

在其中一个实施例中，所述第一位置为图像采集设备与目标区域中的第一参照物的相对位置；所述根据所述第一位置和所述第一目标对象重建得到实时图像，包括：根据所述相对位置将所述扫描图像按照预设方向进行排序；识别所述扫描图像中的第一目标对象；基于排序后的所述扫描图像以及所述第一目标对象对所述扫描图像进行重建得到实时图像。In one of the embodiments, the first position is a relative position between the image acquisition device and a first reference object in the target area; the reconstructing and obtaining a real-time image according to the first position and the first target object, including : sort the scanned images in a preset direction according to the relative positions; identify the first target object in the scanned images; sort the scanned images based on the sorted scanned images and the first target object Perform reconstruction to obtain real-time images.

在其中一个实施例中，所述根据所述第一目标对象将所述标准图像与所述实时图像进行匹配融合得到当前图像，包括：识别所述标准图像中的第一目标对象；将所述标准图像中的第一目标对象以及所述实时图像中的第一目标对象进行匹配，以将所述标准图像与所述实时图像进行匹配融合得到当前图像。In one embodiment, the matching and fusing the standard image and the real-time image to obtain the current image according to the first target object includes: recognizing the first target object in the standard image; The first target object in the standard image and the first target object in the real-time image are matched to obtain a current image by matching and fusing the standard image and the real-time image.

在其中一个实施例中，其特征在于，所述标准图像中包括第一参照物；所述方法还包括：通过增强现实设备采集得到真实图像，所述真实图像中包括第一参照物；基于所述第一参照物将所述标准图像与所述真实图像进行匹配；根据所述标准图像与所述真实图像的匹配关系，以及所述标准图像与所述实时图像的匹配关系，将所述当前图像与所述真实图像匹配融合得到混合现实图像，并通过所述增强现实设备显示所述混合现实图像。In one of the embodiments, it is characterized in that the standard image includes a first reference object; the method further includes: acquiring a real image through an augmented reality device, where the real image includes the first reference object; The first reference object matches the standard image and the real image; according to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image, the current The image is matched and fused with the real image to obtain a mixed reality image, and the mixed reality image is displayed by the augmented reality device.

在其中一个实施例中，所述基于所述第一参照物将所述标准图像与所述真实图像进行匹配，包括：通过光学定位跟踪设备采集目标空间的参考图像，所述参考图像包括第一参照物；根据所述第一参照物将所述参考图像与所述增强现实设备的所采集的图像坐标系下的真实图像进行配准，以获取所述增强现实设备的图像坐标系下的真实图像与所述参考图像的匹配关系；根据所述参考图像与所述增强现实设备的所采集的图像坐标系下的真实图像匹配关系、所述图像坐标系与视觉空间坐标系的匹配关系，将所述参考图像与所述视觉空间坐标系下的真实图像进行配准；将与所述参考图像配准后的所述视觉空间坐标系下的真实图像与所述标准图像进行匹配。In one of the embodiments, the matching of the standard image with the real image based on the first reference object includes: collecting a reference image in a target space by an optical positioning and tracking device, the reference image including the first reference image. A reference object; the reference image is registered with the real image in the image coordinate system of the augmented reality device according to the first reference object, so as to obtain the real image in the image coordinate system of the augmented reality device The matching relationship between the image and the reference image; according to the matching relationship between the reference image and the real image in the image coordinate system collected by the augmented reality device, and the matching relationship between the image coordinate system and the visual space coordinate system, the The reference image is registered with the real image in the visual space coordinate system; the real image in the visual space coordinate system after being registered with the reference image is matched with the standard image.

在其中一个实施例中，所述真实图像中还包括第二参照物，所述第二参照物用于标定操作对象；所述方法还包括：根据所述标准图像与所述真实图像的匹配关系，以及所述标准图像与所述当前图像的匹配关系，将所述当前图像与所述真实图像匹配融合得到混合现实图像，将所述操作对象显示在所述混合现实图像中。In one of the embodiments, the real image further includes a second reference object, and the second reference object is used to calibrate the operation object; the method further includes: according to the matching relationship between the standard image and the real image , and the matching relationship between the standard image and the current image, the current image and the real image are matched and fused to obtain a mixed reality image, and the operation object is displayed in the mixed reality image.

在其中一个实施例中，所述方法还包括：接收针对标准图像的操作信息配置指令，并根据所述配置指令建立所述操作信息与所述标准图像的对应关系；基于所述操作信息与所述标准图像的对应关系、所述标准图像与所述真实图像的匹配关系以及所述标准图像与所述实时图像的匹配关系，将所述操作信息显示在所述混合现实图像中。In one embodiment, the method further includes: receiving an operation information configuration instruction for a standard image, and establishing a corresponding relationship between the operation information and the standard image according to the configuration instruction; and based on the operation information and the standard image The corresponding relationship between the standard image, the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image are displayed, and the operation information is displayed in the mixed reality image.

在其中一个实施例中，所述方法还包括：通过所述增强现实设备接收针对所述操作信息的第一编辑指令；根据所述第一编辑指令对所述操作信息进行编辑，所述第一编辑指令包括移动、对齐、重新定向、旋转、放大中的至少一个。In one of the embodiments, the method further includes: receiving, by the augmented reality device, a first editing instruction for the operation information; editing the operation information according to the first editing instruction, the first editing instruction The editing instructions include at least one of move, align, reorient, rotate, and enlarge.

在其中一个实施例中，所述方法还包括：对所述混合现实图像进行截图以得到截图图像；将所述增强现实设备视野划分为混合现实图像显示区域以及所述截图图像显示区域；通过所述混合现实图像显示区域显示所述混合现实图像，并通过所述截图图像显示区域显示所述截图图像；通过所述截图图像显示区域接收针对所述截图图像的第二编辑指令，所述第二编辑指令包括移动、对齐、旋转、放大中的至少一个。In one embodiment, the method further includes: taking a screenshot of the mixed reality image to obtain a screenshot image; dividing the field of view of the augmented reality device into a mixed reality image display area and the screenshot image display area; the mixed reality image display area displays the mixed reality image, and displays the screenshot image through the screenshot image display area; receives a second editing instruction for the screenshot image through the screenshot image display area, and the second The editing instruction includes at least one of move, align, rotate, and enlarge.

在其中一个实施例中，所述获取预先生成的标准图像之前，还包括：通过至少一种医学成像设备对包括有第一参照物的所述目标区域进行扫描得到医学图像；对所述医学图像进行三维重建得到标准图像。In one embodiment, before acquiring the pre-generated standard image, the method further includes: scanning the target area including the first reference object by at least one medical imaging device to obtain a medical image; Perform 3D reconstruction to obtain standard images.

在其中一个实施例中，所述方法还包括：获取生理监测系统采集的生理信号；根据所述生理信号生成报警信息；显示所述报警信息。In one embodiment, the method further includes: acquiring physiological signals collected by a physiological monitoring system; generating alarm information according to the physiological signals; and displaying the alarm information.

在其中一个实施例中，所述方法还包括：将所述生理信号和/或所述生理信号对应的报警信息发送至所述增强现实设备，并通过所述增强现实设备进行显示。In one of the embodiments, the method further includes: sending the physiological signal and/or alarm information corresponding to the physiological signal to the augmented reality device, and displaying it through the augmented reality device.

在其中一个实施例中，所述根据所述生理信号生成报警信息，包括：获取预先设置的报警等级参数；将所述生理信号与所述报警等级参数进行匹配以确定报警等级。In one of the embodiments, the generating the alarm information according to the physiological signal includes: acquiring a preset alarm level parameter; and matching the physiological signal with the alarm level parameter to determine the alarm level.

一种图像处理系统，所述系统包括图像处理设备以及图像采集设备，所述图像处理设备与所述图像采集设备相通信；所述图像采集设备用于扫描目标区域并生成所述目标区域的实时图像；所述图像处理设备用于实现上述的图像处理方法。An image processing system, the system includes an image processing device and an image acquisition device, the image processing device communicates with the image acquisition device; the image acquisition device is used to scan a target area and generate a real-time image of the target area an image; the image processing device is used to implement the above-mentioned image processing method.

在其中一个实施例中，所述图像采集设备为超声设备。In one of the embodiments, the image acquisition device is an ultrasound device.

在其中一个实施例中，所述系统还包括光学定位跟踪设备，所述光学定位跟踪设备与所述图像处理设备通信，所述光学定位跟踪设备用于采集参考图像，并将所述参考图像发送至所述图像处理设备。In one of the embodiments, the system further includes an optical positioning and tracking device, the optical positioning and tracking device is in communication with the image processing device, the optical positioning and tracking device is configured to collect a reference image and send the reference image to the image processing device.

在其中一个实施例中，所述还包括增强现实设备，所述增强现实设备与所述图像处理设备相通信，所述增强现实设备用于采集得到真实图像，并将所述真实图像发送至所述图像处理设备，所述真实图像中包括所述第一参照物。In one of the embodiments, the device further includes an augmented reality device, the augmented reality device communicates with the image processing device, and the augmented reality device is configured to acquire a real image and send the real image to the image processing device. The image processing device, wherein the real image includes the first reference object.

在其中一个实施例中，所述系统还包括操作对象，所述操作对象是通过第二参照物标定的。In one of the embodiments, the system further includes an operation object, and the operation object is calibrated by a second reference object.

在其中一个实施例中，所述系统还包括生理数据采集设备，所述生理数据采集设备与所述图像处理设备相通信；所述生理数据采集设备用于采集生理信号，并将所述生理信号发送至所述图像处理设备。In one of the embodiments, the system further includes a physiological data acquisition device, the physiological data acquisition device communicates with the image processing device; the physiological data acquisition device is used for collecting physiological signals, and converting the physiological signals sent to the image processing device.

在其中一个实施例中，所述系统还包括至少一种医学成像设备，至少一种所述医学成像设备与所述图像处理设备相通信，所述医学成像设备用于扫描得到医学图像，所述医学图像用于生成标准图像。In one of the embodiments, the system further comprises at least one medical imaging device, the at least one medical imaging device is in communication with the image processing device, the medical imaging device is used for scanning to obtain a medical image, the Medical images are used to generate standard images.

在其中一个实施例中，所述医学成像设备包括核磁共振设备和计算机断层扫描设备。In one embodiment, the medical imaging device includes a nuclear magnetic resonance device and a computed tomography device.

一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现上述的方法的步骤。A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when the processor executes the computer program.

一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述的方法的步骤。A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the above-mentioned method.

上述图像处理方法、系统、计算机设备和存储介质，通过获取手术区域的实时图像，实时图像中包括第一目标对象以及第二目标对象。获取预先生成的标准图像；根据第一目标对象将标准图像与实时图像进行匹配融合得到当前图像；根据重建得到的第二目标对象在实时图像中的位置，将第二目标对象显示在当前图像中，以将术中带有植入物的图像实时更新，进而实现通过手术中收集的图像进行术中操作的实时导航。The above-mentioned image processing method, system, computer equipment and storage medium obtain a real-time image of the operation area, and the real-time image includes a first target object and a second target object. Obtain a pre-generated standard image; match and fuse the standard image and the real-time image according to the first target object to obtain a current image; display the second target object in the current image according to the reconstructed position of the second target object in the real-time image , in order to update the intraoperative images with implants in real time, so as to realize the real-time navigation of intraoperative operations through the images collected during the operation.

附图说明Description of drawings

图1为一个实施例中的图像处理系统的示意图；1 is a schematic diagram of an image processing system in one embodiment;

图2为又一个实施例中的图像处理系统的示意图；2 is a schematic diagram of an image processing system in yet another embodiment;

图3为一个实施例中的图像处理系统的各个部分的关系示意图；3 is a schematic diagram of the relationship of various parts of an image processing system in one embodiment;

图4为另一个实施例中图像示意图；4 is a schematic diagram of an image in another embodiment;

图5为一个实施例中的图像处理方法的流程示意图；5 is a schematic flowchart of an image processing method in one embodiment;

图6为又一个实施例中的图像处理方法的流程示意图；6 is a schematic flowchart of an image processing method in yet another embodiment;

图7为又一个实施例中的图像处理方法的流程示意图；7 is a schematic flowchart of an image processing method in yet another embodiment;

图8为一个实施例中的图像处理方法的模块示意图；8 is a schematic block diagram of an image processing method in one embodiment;

图9为一个实施例中的术前规划的流程示意图；9 is a schematic flowchart of preoperative planning in one embodiment;

图10为一个实施例中的各模块之间的关系示意图；10 is a schematic diagram of the relationship between the modules in one embodiment;

图11为另一个实施例中的各模块之间的关系示意图；11 is a schematic diagram of the relationship between the modules in another embodiment;

图12为一个实施例中的图像处理方法的流程示意图；12 is a schematic flowchart of an image processing method in one embodiment;

图13为一个实施例中的操作对象角度显示示意图；FIG. 13 is a schematic diagram showing the angle display of an operation object in one embodiment;

图14为一个实施例中的实际操作示意图；Figure 14 is a schematic diagram of the actual operation in one embodiment;

图15为一个实施例中的设备通讯关系图；Figure 15 is a device communication relationship diagram in one embodiment;

图16为一个实施例中计算机设备的内部结构图。Figure 16 is a diagram of the internal structure of a computer device in one embodiment.

具体实施方式Detailed ways

为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

结合图1所示，本申请的一个实施例中提供一种图像处理系统，该图像处理系统至少包括图像处理设备以及图像采集设备，其中图像处理设备和图像采集设备相通信。结合图1所示，图1为一个实施例中的图像处理系统的示意图。其中，此处的图像处理设备可以理解为处理器，且该处理器并不仅限于在导航车里的处理器2，其也可以包括增强现实设备6的处理器。在一个优选的实施例中，该图像处理系统中可以包括至少两个处理器，至少两个处理器之间相互通信，以实现不同的功能。这里为了方便将存在处理功能的处理器统一称为图像处理设备。With reference to FIG. 1 , an embodiment of the present application provides an image processing system, the image processing system at least includes an image processing device and an image acquisition device, wherein the image processing device and the image acquisition device communicate. Referring to FIG. 1 , FIG. 1 is a schematic diagram of an image processing system in one embodiment. The image processing device here can be understood as a processor, and the processor is not limited to theprocessor 2 in the navigation car, but may also include the processor of the augmentedreality device 6 . In a preferred embodiment, the image processing system may include at least two processors, and the at least two processors communicate with each other to implement different functions. Here, for convenience, processors with processing functions are collectively referred to as image processing devices.

该图像处理系统包括图像处理设备和图像采集设备，其中，图像处理设备则可以参见上文限定，在此不做具体的限定。其中，在手术的时候，图像处理设备将通过图像采集设备采集手术区域的实时图像，该实时图像中包括第一目标对象和第二目标对象。图像处理设备获取与实时图像对应的预先生成的标准图像，该标准图像中也包括第一目标对象及第二目标对象，从而，根据第一目标对象可以将标准图像与实时图像进行匹配融合得到当前图像。这样根据第二目标对象在实时图像中的位置，实时图像与当前图像之间的匹配关系，将第二目标对象显示在当前图像中，以将术中带有植入物的图像实时更新，进而实现通过手术中收集的图像进行术中操作的实时导航。The image processing system includes an image processing device and an image acquisition device, wherein, the image processing device may refer to the above definition, and no specific limitation is made here. Wherein, during the operation, the image processing device will collect a real-time image of the operation area through the image acquisition device, and the real-time image includes the first target object and the second target object. The image processing device obtains a pre-generated standard image corresponding to the real-time image, and the standard image also includes the first target object and the second target object, so that the standard image and the real-time image can be matched and fused according to the first target object to obtain the current image. image. In this way, according to the position of the second target object in the real-time image and the matching relationship between the real-time image and the current image, the second target object is displayed in the current image, so as to update the intraoperative image with the implant in real time, and then Enables real-time navigation of intraoperative operations through images collected during surgery.

在一个实施例中，以图4为例，图像采集设备可以为超声设备5，在其他的实施例中，该图像采集设备可以为其他的医疗设备，在此不做具体限定。更为优选地，该图像采集设备可以直接采集三维实时图像，或者是该图像采集设备可以是先采集得到二维图像，然后对二维图像进行重建得到三维实时图像。在患者躺在手术台7上时，通过图2中的第一参照物8对患者的待手术部位进行标记。通过超声设备5可以采集到手术区域的扫描图像，该扫描图像中包括第一目标对象以及第二目标对象，且扫描图像携带有与第一参照物对应的第一位置，第一参照物用于标定第一目标对象。以图4为例，超声设备5可以采集到图4中的超声影像示意图，该超声影像示意图中包括具体手术部位脊柱区域和植入物脊柱椎弓钉。图像处理设备根据扫描图像中的第一位置和第一目标对象得到手术区域的三维实时图像。在其中一个实施例中，如图3所示，该图像处理系统还包括光学定位跟踪设备，光学定位跟踪设备(NDI)可以理解为位置采集处理子系统，该子系统包括图3中的光学定位根据设备1、图2中的患者脊柱段上的基准靶标8、操作对象上的靶标9以及超声设备的探头上的靶标10，该子系统中的各设备之间相互通信，以完整不同的功能。这里为了方便，将该子系统统称为光学定位跟踪设备。需要说明的是，光学定位跟踪设备中的靶标种类及数目在本实施例中并不做限定，满足实际定位需求即可。图像处理系统中的光学定位跟踪设备(NDI)与图像处理设备通信，光学定位跟踪设备用于采集参考图像，并将参考图像发送至图像处理设备。图像处理设备根据第一参照物将参考图像与增强现实设备的所采集的图像坐标系下的真实图像进行配准，以获取增强现实设备的图像坐标系下的真实图像与参考图像的匹配关系；根据参考图像与增强现实设备的所采集的图像坐标系下的真实图像匹配关系、图像坐标系与视觉空间坐标系的匹配关系，将参考图像与视觉空间坐标系下的真实图像进行配准；将与参考图像配准后的视觉空间坐标系下的真实图像与标准图像进行匹配。In one embodiment, taking FIG. 4 as an example, the image acquisition device may be anultrasound device 5 , and in other embodiments, the image acquisition device may be other medical equipment, which is not specifically limited herein. More preferably, the image acquisition device can directly acquire a three-dimensional real-time image, or the image acquisition device can first acquire a two-dimensional image, and then reconstruct the two-dimensional image to obtain a three-dimensional real-time image. When the patient is lying on the operating table 7 , the patient's site to be operated on is marked by thefirst reference object 8 in FIG. 2 . A scanned image of the surgical area can be collected by theultrasonic device 5 , the scanned image includes a first target object and a second target object, and the scanned image carries a first position corresponding to the first reference object, and the first reference object is used for Calibrate the first target object. Taking FIG. 4 as an example, theultrasonic device 5 can collect the schematic diagram of the ultrasonic image in FIG. 4 , and the schematic diagram of the ultrasonic image includes the spinal region of the specific operation site and the implanted spinal pedicle screw. The image processing device obtains a three-dimensional real-time image of the operation area according to the first position in the scanned image and the first target object. In one embodiment, as shown in FIG. 3 , the image processing system further includes an optical positioning and tracking device. The optical positioning and tracking device (NDI) can be understood as a position acquisition and processing subsystem, and the subsystem includes the optical positioning and tracking device in FIG. 3 . According to thedevice 1, thereference target 8 on the patient's spinal segment in Figure 2, thetarget 9 on the operating object, and thetarget 10 on the probe of the ultrasound device, the devices in this subsystem communicate with each other to complete different functions. . For convenience, the subsystem is collectively referred to as an optical positioning and tracking device. It should be noted that, the types and numbers of targets in the optical positioning and tracking device are not limited in this embodiment, as long as the actual positioning requirements are met. An optical positioning and tracking device (NDI) in the image processing system communicates with the image processing device, and the optical positioning and tracking device is used for collecting a reference image and sending the reference image to the image processing device. The image processing device registers the reference image with the real image in the image coordinate system collected by the augmented reality device according to the first reference object, so as to obtain a matching relationship between the real image and the reference image in the image coordinate system of the augmented reality device; According to the matching relationship between the reference image and the real image in the image coordinate system collected by the augmented reality device, and the matching relationship between the image coordinate system and the visual space coordinate system, the reference image is registered with the real image in the visual space coordinate system; The real image in the visual space coordinate system registered with the reference image is matched with the standard image.

在其中一个实施例中，该图像处理系统还包括增强现实设备，增强现实设备与图像处理设备相通信。结合图1所示，在一个具体的实施例中，增强现实设备为混合现实眼镜6，通过混合现实眼镜采集得到真实图像，并将真实图像发送至图像处理设备，真实图像中包括第一参照物8。图像处理设备基于第二参照物和第一参照物将标准图像与真实图像进行匹配；根据标准图像与真实图像的匹配关系，以及标准图像与当前图像的匹配关系，将当前图像与真实图像匹配融合得到混合现实图像，并通过增强现实设备显示混合现实图像。In one of the embodiments, the image processing system further includes an augmented reality device in communication with the image processing device. As shown in FIG. 1 , in a specific embodiment, the augmented reality device ismixed reality glasses 6 , a real image is acquired through the mixed reality glasses, and the real image is sent to the image processing device, and the real image includes afirst reference object 8. The image processing device matches the standard image and the real image based on the second reference object and the first reference object; according to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the current image, the current image and the real image are matched and fused Get mixed reality images and display mixed reality images through augmented reality devices.

在其中一个实施例中，该图像处理系统还包括操作对象，操作对象是通过第二参照物标定的。以图1为例，操作对象为机械臂4，机械臂4的位置是通过图2中的靶标9标定。In one of the embodiments, the image processing system further includes an operation object, and the operation object is calibrated by the second reference object. Taking FIG. 1 as an example, the operation object is therobotic arm 4 , and the position of therobotic arm 4 is calibrated by thetarget 9 in FIG. 2 .

在其中一个实施例中，该图像处理系统还包括生理数据采集设备，生理数据采集设备与图像处理设备相通信。结合图1所示，在该实施例中，生理数据采集设备则为神经电生理监测子系统3。神经电生理监测子系统3用于采集获取肌电信号数据，并将所采集的肌电信号数据发送至图像处理设备。图像处理设备根据肌电信号数据生成报警信息，并通过混合现实眼镜6显示报警信息。In one of the embodiments, the image processing system further includes a physiological data acquisition device, and the physiological data acquisition device is in communication with the image processing device. Referring to FIG. 1 , in this embodiment, the physiological data acquisition device is theneurophysiological monitoring subsystem 3 . The nerveelectrophysiology monitoring subsystem 3 is used for collecting and acquiring electromyographic signal data, and sending the collected electromyographic signal data to the image processing device. The image processing device generates alarm information according to the EMG signal data, and displays the alarm information through themixed reality glasses 6 .

在其中一个实施例中，该图像处理系统还包括至少一种医学成像设备，至少一种医学成像设备与图像处理设备相通信，医学成像设备用于扫描得到医学图像，医学图像用于生成标准图像。In one embodiment, the image processing system further includes at least one medical imaging device, the at least one medical imaging device communicates with the image processing device, the medical imaging device is used for scanning to obtain a medical image, and the medical image is used to generate a standard image .

在其中一个实施例中，该图像处理系统中的医学成像设备包括核磁共振设备和/或计算机断层扫描设备。In one of the embodiments, the medical imaging device in the image processing system includes a nuclear magnetic resonance device and/or a computed tomography device.

在一个实施例中，如图5所示，提供了一种图像处理方法，该方法应用于图1中的图像处理设备2。本实施例中，该方法包括以下步骤：In one embodiment, as shown in FIG. 5 , an image processing method is provided, and the method is applied to theimage processing apparatus 2 in FIG. 1 . In this embodiment, the method includes the following steps:

步骤102，获取目标区域的实时图像，实时图像包括第一目标对象以及第二目标对象。Step 102, acquiring a real-time image of the target area, where the real-time image includes a first target object and a second target object.

其中，实时图像为根据图像采集设备采集的手术区域的三维实时图像。手术区域为待进行手术的目标区域。第一目标对象为具体手术部位，本实施例并不对具体手术部位做限定，具体手术部位可以为脊骨、腰椎等人体任意位置，但为了便于说明，下文中以具体手术部位为脊柱进行说明。第二目标对象为术中在第一目标对象处的植入物，本实施例并不对该植入物做限定，可以为脊柱椎弓钉、接骨板、接骨螺钉、髓内钉、缝合线等，满足具体手术部位的手术需求即可，为了便于说明，下文中以第二目标对象为脊柱椎弓钉进行说明，可以理解的是，第二目标对象除脊柱椎弓钉外，也可以为其它的第一目标对象的植入物。Wherein, the real-time image is a three-dimensional real-time image of the operation area collected according to the image collection device. The surgical area is the target area to be operated on. The first target object is a specific surgical site. This embodiment does not limit the specific surgical site. The specific surgical site can be any position on the human body such as the spine and lumbar spine. However, for the convenience of description, the specific surgical site is described below as the spine. The second target object is an intraoperative implant at the first target object, which is not limited in this embodiment, and may be a spinal pedicle screw, a bone plate, a bone screw, an intramedullary nail, a suture, etc. It is enough to meet the surgical requirements of the specific surgical site. For the convenience of description, the second target object is described as the spinal pedicle screw. It can be understood that the second target object can be other than the spinal pedicle screw. Implants of the first target subject.

具体地，图像处理设备获取图像采集设备采集的手术区域的三维实时图像，三维实时图像中包括第一目标对象和第二目标对象。在其他的实施例中，该图像采集设备可以是先采集得到二维图像，然后对二维图像进行重建得到三维实时图像，在此不做具体的限定。Specifically, the image processing device acquires a three-dimensional real-time image of the surgical area collected by the image acquisition device, and the three-dimensional real-time image includes the first target object and the second target object. In other embodiments, the image acquisition device may first acquire a two-dimensional image, and then reconstruct the two-dimensional image to obtain a three-dimensional real-time image, which is not specifically limited herein.

步骤104，获取预先生成的标准图像。Step 104, acquiring a pre-generated standard image.

其中，标准图像为图像处理设备预先根据病人的术前医学影像进行三维重建得到的手术区域的术前三维图。例如术前通过医学成像设备采集得到医学影像，并对医学影像进行分割、重建以得到第一目标对象的标准三维图像，该标准三维图像是包括完整的第一目标对象的，其可以提供纵观全局的视图。Wherein, the standard image is a preoperative 3D image of the surgical area obtained by the image processing device in advance by performing 3D reconstruction according to the preoperative medical image of the patient. For example, a medical image is acquired by a medical imaging device before surgery, and the medical image is segmented and reconstructed to obtain a standard three-dimensional image of the first target object. The standard three-dimensional image includes the complete first target object and can provide an overview global view.

步骤106，根据第一目标对象将标准图像与实时图像进行匹配融合得到当前图像。Step 106: Match and fuse the standard image and the real-time image according to the first target object to obtain the current image.

其中，当前图像为三维图像，其融合了标准图像和实时图像中的信息，其中图像处理设备根据第一目标对象将标准图像和实时图像进行配准。具体地，图像处理设备从标准图像和实时图像中的第一目标对象中选取对应的特征点，然后根据对应的特征点将标准图像与实时图像进行配准，这样可以将标准图像与实时图像配准以得到融合了所有信息的当前图像。Wherein, the current image is a three-dimensional image, which fuses the information in the standard image and the real-time image, wherein the image processing device registers the standard image and the real-time image according to the first target object. Specifically, the image processing device selects corresponding feature points from the first target object in the standard image and the real-time image, and then registers the standard image and the real-time image according to the corresponding feature points, so that the standard image can be matched with the real-time image. to get the current image with all the information fused.

具体地，图像处理设备根据目标脊柱的特征对术前三维图像进行特征识别，基于特征匹配与融合方法将所识别出的特征与三维实时图像中的相应特征一一匹配融合，得到手术区域的当前图像。Specifically, the image processing device performs feature recognition on the preoperative three-dimensional image according to the features of the target spine, and matches and fuses the identified features with the corresponding features in the three-dimensional real-time image based on the feature matching and fusion method to obtain the current operation area. image.

步骤108，根据第二目标对象在实时图像中的位置，将第二目标对象显示在当前图像中。Step 108: Display the second target object in the current image according to the position of the second target object in the real-time image.

具体地，图像处理设备由于第二目标对象在实时图像中的位置已经通过实时图像标记，而标准图像和实时图像进行了配准得到当前图像，因此可以获取到第二目标对象与当前图像的位置关系，进而将第二目标对象显示在当前图像中。Specifically, since the position of the second target object in the real-time image has been marked by the real-time image, and the standard image and the real-time image are registered to obtain the current image, the image processing device can obtain the position of the second target object and the current image relationship, and then display the second target object in the current image.

优选地，以第二目标对象为手术中的植入物为例进行说明，其中植入物在实时图像，实时图像与术前三维图像进行了配准得到了当前图像，因此图像处理设备可以获取到植入物与当前图像的位置关系，进而可以将植入物显示在当前图像中，从而医生等可以实时看到植入物在整个手术区域的位置，以为医生提供通观全局的视野。Preferably, the second target object is an implant in operation as an example for illustration, wherein the implant is in the real-time image, and the real-time image is registered with the pre-operative three-dimensional image to obtain the current image, so the image processing device can obtain the current image. The positional relationship between the implant and the current image can be obtained, and the implant can be displayed in the current image, so that the doctor can see the position of the implant in the entire operation area in real time, so as to provide the doctor with a global vision.

在具体地实施过程中，图像处理设备根据得到的植入目标脊柱的脊柱椎弓钉在三维实时图像中的位置，将植入物显示在人体三维实时图像的对应位置中。In a specific implementation process, the image processing device displays the implant in the corresponding position of the three-dimensional real-time image of the human body according to the obtained position of the spinal pedicle screw implanted into the target spine in the three-dimensional real-time image.

继续获取CT/MRI的三维重建图即标准图像，将标准图像与三维实时图像同屏显示，通过特征识别方法识别多模式医学图像即标准图像中的特征，如脊椎骨棘突、横突、关节突及必要的肋骨和肋骨头结构，再基于特征匹配与融合方法，将上述识别出的特征与三维实时图像中的相应结构一一匹配，以将三维实时图像和CT/MRI的三维重建图像进行融合，得到实时的人体三维图即当前图像。将当前图像通过通信模块进行处理传递给混合现实眼镜，以得到并显示混合现实图像。Continue to obtain the 3D reconstructed image of CT/MRI, that is, the standard image, display the standard image and the 3D real-time image on the same screen, and identify the features in the multi-modal medical image, that is, the standard image, such as the spinous process, transverse process, and articular process of the vertebrae through the feature recognition method. and the necessary rib and rib head structures, and then based on the feature matching and fusion method, the above-identified features are matched with the corresponding structures in the 3D real-time image to fuse the 3D real-time image and the 3D reconstructed image of CT/MRI. , to obtain a real-time three-dimensional image of the human body, that is, the current image. The current image is processed and transmitted to the mixed reality glasses through the communication module, so as to obtain and display the mixed reality image.

上述图像处理方法、系统、计算机设备和存储介质，通过获取手术区域的实时图像，实时图像中包括第一目标对象以及第二目标对象。获取预先生成的标准图像；根据第一目标对象将标准图像与实时图像进行匹配融合得到当前图像；根据第二目标对象在实时图像中的位置，将第二目标对象显示在当前图像中，以将术中带有植入物的图像实时更新，进而实现通过手术中收集的图像进行术中操作的实时导航。The above-mentioned image processing method, system, computer equipment and storage medium obtain a real-time image of the operation area, and the real-time image includes a first target object and a second target object. Obtain a pre-generated standard image; match and fuse the standard image and the real-time image according to the first target object to obtain a current image; display the second target object in the current image according to the position of the second target object in the real-time image, so as to display the second target object in the current image. Intraoperative images with implants are updated in real time, thereby enabling real-time navigation of intraoperative operations through images collected during surgery.

在一个实施例中，获取目标区域的实时图像，包括：获取扫描得到的目标区域的扫描图像，扫描图像中包括第一目标对象以及第二目标对象，且扫描图像携带有与第一参照物对应的第一位置，第一参照物用于标定第一目标对象。In one embodiment, acquiring a real-time image of the target area includes: acquiring a scanned image of the target area obtained by scanning, where the scanned image includes a first target object and a second target object, and the scanned image carries information corresponding to the first reference object. The first position of the first reference object is used to calibrate the first target object.

在本实施例中，主要介绍图像采集设备先采集得到二维图像，然后对二维图像进行重建得到三维实时图像的实施例。In this embodiment, an embodiment in which an image acquisition device first acquires a two-dimensional image and then reconstructs the two-dimensional image to obtain a three-dimensional real-time image is mainly introduced.

其中，目标区域为目标器官、组织或骨骼区域，具体的为待进行手术的目标区域，简称为手术区域。扫描图像为手术区域的扫描图像，该扫描图像可以是通过图像采集设备采集的，优选地，该图像采集设备为超声设备，因此所采集的扫描图像为二维超声图像，且由于在手术之前，在手术区域预先放置有第一参照物，在超声设备上设置有对应第一参照物的第三参照物，通过第一参照物和第三参照物之间的距离可以标定手术区域与超声探头的相对位置距离，因此该超声图像携带有与第一参照物对应的第一位置，该第一位置用于标定手术区域。第一参照物为放置在第一目标对象上的基准靶标等，第一参照物的主要目的是为了标定第一目标对象，因此只要能够在图像中显示，并且标定第一目标对象的物体均可。The target area is a target organ, tissue or bone area, specifically a target area to be operated on, which is referred to as an operation area for short. The scanned image is a scanned image of the surgical area, and the scanned image can be acquired by an image acquisition device. Preferably, the image acquisition device is an ultrasonic device, so the acquired scanned image is a two-dimensional ultrasonic image. A first reference object is pre-placed in the operating area, and a third reference object corresponding to the first reference object is set on the ultrasonic device. The distance between the first reference object and the third reference object can be used to calibrate the distance between the operating area and the ultrasonic probe. The relative position distance, so the ultrasound image carries a first position corresponding to the first reference object, and the first position is used to demarcate the surgical area. The first reference object is a reference target placed on the first target object, etc. The main purpose of the first reference object is to calibrate the first target object, so as long as it can be displayed in the image and the object that calibrates the first target object can be .

具体地，图像处理设备获取图像采集设备扫描手术区域得到扫描图像，扫描图像中包括具体手术部位及具体手术部位的植入物，且扫描图像中携带有与放置在具体手术部位上的基准靶标对应的第一位置，放置在具体手术部位上的标志物可以提供具体手术部位与图像采集设备的相对位置。Specifically, the image processing device acquires the image acquisition device to scan the surgical area to obtain a scanned image, the scanned image includes the specific surgical site and the implants at the specific surgical site, and the scanned image carries the reference target placed on the specific surgical site corresponding to In the first position, the marker placed on the specific surgical site can provide the relative position of the specific surgical site and the image acquisition device.

在具体地实施过程中，图像采集设备为超声设备，相应地，采集的扫描图像为手术区域对应的二维超声图像，具体手术部位为脊柱，对应的植入物为脊柱椎弓钉。即图像处理设备获取超声设备扫描得到的目标脊柱的至少一张二维超声图像，这些二维超声图像中包括目标脊柱及植入目标脊柱中的脊柱椎弓钉。且这些二维超声图像中携带有超声设备与目标脊柱的相对位置，相对位置的获取方法包括：首先，通过在目标脊柱上放置基准靶标，超声设备的超声探头上也放置有反光球靶标，其次，超声设备对目标脊柱及其上放置基准靶标进行扫描，一方面可以获取到目标脊柱的二维超声图像，另一方面可以确定基准靶标和反光球靶标的相对位置，由此获得超声设备与目标脊柱的相对位置。需要说明的是，图像采集设备可以不是超声设备，是其它的可以获取手术区域及对应的植入物的设备，例如红外感应图像获取设备等，本实施例并不对此进行限定，举此例仅为方便解释说明。In a specific implementation process, the image acquisition device is an ultrasonic device, and correspondingly, the acquired scan image is a two-dimensional ultrasonic image corresponding to the operation area, the specific operation site is the spine, and the corresponding implant is a spinal pedicle screw. That is, the image processing device acquires at least one two-dimensional ultrasound image of the target spine scanned by the ultrasound device, and these two-dimensional ultrasound images include the target spine and the spinal pedicle screws implanted in the target spine. And these two-dimensional ultrasound images carry the relative positions of the ultrasound equipment and the target spine. The relative position acquisition method includes: first, by placing a reference target on the target spine, and a reflective ball target is also placed on the ultrasound probe of the ultrasound equipment, and secondly , the ultrasound equipment scans the target spine and the reference target placed on it. On the one hand, the two-dimensional ultrasound image of the target spine can be obtained, and on the other hand, the relative position of the reference target and the reflective ball target can be determined, thereby obtaining the ultrasonic equipment and the target. The relative position of the spine. It should be noted that the image acquisition device may not be an ultrasound device, but other devices that can acquire the surgical area and corresponding implants, such as an infrared sensing image acquisition device, which is not limited in this embodiment, and this example is only For ease of explanation.

根据第一位置和第一目标对象对扫描图像进行重建得到实时图像，实时图像包括第一目标对象和第二目标对象。A real-time image is obtained by reconstructing the scanned image according to the first position and the first target object, and the real-time image includes the first target object and the second target object.

其中，实时图像为根据扫描图像以及第一位置进行重建得到的三维图像，其中扫描图像中包括第一目标对象，图像处理设备可以根据第一位置将扫描图像进行排列，并对各个扫描图像进行分割得到第一目标对象，根据各个扫描图像中的第一目标对象以及扫描图像的排列顺序进行重建得到实时图像。优选地，该实时图像为三维超声实时图像。The real-time image is a three-dimensional image reconstructed according to the scanned image and the first position, wherein the scanned image includes the first target object, and the image processing device can arrange the scanned images according to the first position, and segment each scanned image A first target object is obtained, and a real-time image is obtained by reconstructing the first target object in each scanned image and the arrangement order of the scanned images. Preferably, the real-time image is a three-dimensional ultrasound real-time image.

其中，仍以脊柱为例进行说明，放置有反光求靶标的超声设备对放置有基准靶标的脊柱进行扫描，这样可以得到脊柱区域的扫描图像，并且根据反光球靶标和基准靶标可以得到脊柱与超声探头的相位位置，根据相对位置对所得到的扫描图像进行排序，并识别扫描图像中的脊柱，这样根据识别得到的脊柱以及相对位置进行三维重建即可以得到三维超声实时图像。Among them, the spine is still used as an example to illustrate. The ultrasound equipment with the reflective target placed on the spine scans the spine with the reference target, so that the scanned image of the spine area can be obtained, and the spine and ultrasound can be obtained according to the reflective ball target and the reference target. According to the phase position of the probe, the obtained scan images are sorted according to the relative position, and the spine in the scan image is identified, so that the three-dimensional ultrasound real-time image can be obtained by performing three-dimensional reconstruction according to the identified spine and relative position.

在具体的实施过程中，以图6和图1以及图2为例，在手术过程中，如图1和图2所示，带有反光球靶标10的超声设备5在术中对目标脊柱进行扫描，扫描方法为纵切扫描目标脊柱区域和横切扫描目标脊柱区域，以获得目标脊柱区域的二维超声图像。其中，纵切扫描是指线阵超声探头的长轴与脊柱中线平行的方式进行扫描；横切是指线阵超声探头的长轴与脊柱中线垂直的方式进行扫描；脊柱中线是指真实人体的脊柱中轴线。所采集的二维超声图像中包括目标脊柱和脊柱椎弓钉以及与基准靶标8对应的第一位置，基准靶标8如用于标定目标脊柱的位置。根据图1中的光学定位跟踪设备1获取目标脊柱上的基准靶标8及反光球靶标10获取的位置信息，以得到超声设备与目标脊柱上的基准靶标的相对位置。其中，目标脊柱区域的二维超声图像中同时包含超声探头靶标和人体基准靶标相对空间位置信息，其空间位置信息包括6个自由度，即其所在坐标系中x、y、z轴的距离位置信息和旋转角度信息；其中，目标脊柱区域是指以脊柱脊髓病灶点为中心，包含其上、下2-3个脊柱节段，左、右至两侧横突边缘的区域范围。对一节椎骨而言，应至少包括棘突和两侧横突结构的二维超声图像信息；对于一次扫描未能获得全部椎骨信息的区域或脊柱节段，应进行多次反复扫描，直至获得所需内容。In the specific implementation process, taking FIG. 6 and FIG. 1 and FIG. 2 as examples, during the operation, as shown in FIG. 1 and FIG. Scanning, the scanning method is to longitudinally scan the target spine region and transversely scan the target spine region to obtain a two-dimensional ultrasound image of the target spine region. Among them, longitudinal scanning refers to scanning in a way that the long axis of the linear array ultrasound probe is parallel to the midline of the spine; transverse scanning refers to scanning in a way that the long axis of the linear array ultrasound probe is perpendicular to the midline of the spine; the midline of the spine refers to the real human body. Spinal axis. The acquired two-dimensional ultrasound image includes the target spine, the spinal pedicle screw, and the first position corresponding to thereference target 8 , for example, thereference target 8 is used to demarcate the position of the target spine. The position information obtained by thereference target 8 and thereflective ball target 10 on the target spine is obtained according to the optical positioning andtracking device 1 in FIG. 1 to obtain the relative position of the ultrasound device and the reference target on the target spine. Among them, the two-dimensional ultrasound image of the target spine region contains both the relative spatial position information of the ultrasound probe target and the human reference target, and the spatial position information includes 6 degrees of freedom, that is, the distance position of the x, y, and z axes in the coordinate system where it is located. Information and rotation angle information; among them, the target spinal area refers to the area range from the left and right to the edges of the transverse processes on both sides, centered on the spinal cord lesion point, including the upper and lower 2-3 spinal segments. For a vertebra, at least two-dimensional ultrasound image information of the spinous process and lateral process structures on both sides should be included; for areas or spinal segments that cannot obtain all vertebral information in one scan, repeated scans should be performed until the required content.

重建得到三维超声重建图即三维超声实时图像的具体方法：获取目标脊柱区域对应的至少一张二维超声图像，将上述二维超声图像进行去噪和图像增强，提取与目标脊柱区域相对应的第一位置信息，并按照第一位置信息将各二维超声图像沿脊柱中线方向排布，基于边缘识别和图像分割提取目标脊柱的解剖结构信息，重建得到三维超声实时图像。上述图像处理方法，通过获取扫描得到的目标区域的扫描图像，扫描图像中包括第一目标对象以及第二目标对象，且扫描图像携带有与第一参照物对应的第一位置，第一参照物用于标定第一目标对象，以获得术中包括植入物的实时图像。通过根据第一位置和第一目标对象重建得到实时图像，实时图像中包括重建得到的第二目标对象；获取预先生成的标准图像；根据第一目标对象将标准图像与实时图像进行匹配融合得到当前图像；根据重建得到的第二目标对象在实时图像中的位置，将第二目标对象显示在当前图像中，以将术中带有植入物的图像实时更新，进而实现通过手术中收集的图像进行术中操作的实时导航。A specific method for reconstructing a 3D ultrasound reconstruction map, that is, a 3D ultrasound real-time image: obtaining at least one 2D ultrasound image corresponding to the target spine region, performing denoising and image enhancement on the above-mentioned 2D ultrasound image, and extracting the first 2D ultrasound image corresponding to the target spine region. According to the first position information, each two-dimensional ultrasound image is arranged along the midline of the spine, the anatomical structure information of the target spine is extracted based on edge recognition and image segmentation, and the three-dimensional ultrasound real-time image is reconstructed. The above image processing method, by acquiring a scanned image of a target area obtained by scanning, the scanned image includes a first target object and a second target object, and the scanned image carries a first position corresponding to the first reference object, the first reference object. For calibrating the first target object to obtain intraoperative real-time images including implants. Obtain a real-time image by reconstructing according to the first position and the first target object, and the real-time image includes the reconstructed second target object; obtain a pre-generated standard image; match and fuse the standard image and the real-time image according to the first target object to obtain the current Image; according to the reconstructed position of the second target object in the real-time image, the second target object is displayed in the current image, so as to update the image with the implant in real time during the operation, and then realize the image collected during the operation. Perform real-time navigation of intraoperative procedures.

在一个实施例中，第一位置为图像采集设备与目标区域中的第一参照物的相对位置；根据第一位置和第一目标对象重建得到实时图像，包括：根据相对位置将扫描图像按照预设方向进行排序；识别扫描图像中的第一目标对象；基于排序后的扫描图像以及第一目标对象对扫描图像进行重建得到实时图像。In one embodiment, the first position is the relative position of the image acquisition device and the first reference object in the target area; reconstructing the real-time image according to the first position and the first target object includes: according to the relative position Set the direction to sort; identify the first target object in the scanned image; reconstruct the scanned image based on the sorted scanned image and the first target object to obtain a real-time image.

其中，预设方向为用户根据具体手术部位预先通过图像处理设备设置好的具体方向，本实施例并不对预设方向的方向做限定，具体的方向根据具体手术部位确定。以具体手术部位为特定脊柱为例，预设方向为脊柱中线方向。The preset direction is a specific direction pre-set by the user through the image processing device according to the specific surgical site. This embodiment does not limit the direction of the preset direction, and the specific direction is determined according to the specific surgical site. Taking the specific surgical site as a specific spine as an example, the preset direction is the direction of the midline of the spine.

具体地，图像处理设备根据超声设备与基准靶标的相对位置将至少一张二维超声图像按照脊柱中线方向进行排序，并根据相关算法识别二维超声图像中的目标脊柱结构信息，以基于排序后的二维超声图像以及目标脊柱的结构信息进行重建获得三维超声实时图像。本实施例并不对识别二维超声图像中的目标脊柱结构信息的相关算法做限定，可选地，可采取基于边缘识别和图像分割的方法识别二维超声图像中的目标脊柱结构信息。Specifically, the image processing device sorts at least one two-dimensional ultrasound image according to the direction of the spine midline according to the relative position of the ultrasound device and the reference target, and identifies the target spine structure information in the two-dimensional ultrasound image according to a related algorithm, and based on the sorted two The three-dimensional ultrasound image and the structural information of the target spine are reconstructed to obtain a three-dimensional ultrasound real-time image. This embodiment does not limit the relevant algorithm for identifying the target spine structure information in the two-dimensional ultrasound image. Optionally, a method based on edge recognition and image segmentation may be adopted to identify the target spine structure information in the two-dimensional ultrasound image.

在本实施例中，图像处理设备通过根据相对位置将扫描图像按照预设方向进行排序，识别扫描图像中的第一目标对象，以系统化的方式按序识别第一目标对象，使得重建得到实时图像更准确。In this embodiment, the image processing device recognizes the first target object in the scanned image by sorting the scanned images in a preset direction according to the relative position, and sequentially recognizes the first target object in a systematic manner, so that the reconstruction can be obtained in real time. Images are more accurate.

在一个实施例中，根据第一目标对象将标准图像与实时图像进行匹配融合得到当前图像，包括：识别标准图像中的第一目标对象；将标准图像中的第一目标对象以及实时图像中的第一目标对象进行匹配，以将标准图像与实时图像进行匹配融合得到当前图像。In one embodiment, matching and fusing the standard image and the real-time image according to the first target object to obtain the current image includes: identifying the first target object in the standard image; combining the first target object in the standard image and the real-time image The first target object is matched to obtain the current image by matching and fusing the standard image and the real-time image.

具体地，图像处理设备识别术前三维图中的第一目标对象，例如具体手术部位，将标准图像中的具体手术部位与对应的三维超声实时图像中的具体手术部位进行匹配，以将标准图像与三维超声实时图像进行匹配融合得到手术区域的三维实时图像。其中，这里的标准图像可以理解为术前三维图。Specifically, the image processing device identifies the first target object in the preoperative three-dimensional image, such as a specific surgical site, and matches the specific surgical site in the standard image with the specific surgical site in the corresponding three-dimensional ultrasound real-time image, so as to convert the standard image The three-dimensional real-time image of the surgical area is obtained by matching and fusing with the three-dimensional ultrasound real-time image. Among them, the standard image here can be understood as a preoperative three-dimensional image.

其中以脊柱为例进行说明，图像处理设备识别术前三维图中的目标脊柱的特征，这些特征可以为脊椎骨棘突、横突、关节突及必要的肋骨和肋骨头结构，将术前三维图中的目标脊柱的特征与三维超声实时图像中的目标脊柱的相应特征一一匹配融合，以确定术前三维图与实时图像的匹配关系，进而得到融合了术前三维图和实时图像的三维实时图像。其中，术前三维图样例可参照图4的侧面影像正面影像，三维超声实时图像的样例可参照图4的超声影像示意图。The spine is taken as an example to illustrate. The image processing equipment identifies the features of the target spine in the preoperative 3D image. These features can be the spinous process, transverse process, articular process of the vertebrae, and necessary rib and rib head structures. The features of the target spine in the 3D ultrasound real-time image are matched and fused one by one with the corresponding features of the target spine in the 3D ultrasound real-time image to determine the matching relationship between the preoperative 3D image and the real-time image, and then obtain a 3D real-time 3D real-time image that combines the preoperative 3D image and the real-time image. image. For an example of a preoperative 3D image, please refer to the frontal image of the side image in FIG. 4 , and for an example of a real-time 3D ultrasound image, please refer to the schematic diagram of the ultrasound image in FIG. 4 .

在本实施例中，图像处理设备通过将标准图像中的第一目标对象以及实时图像中的第一目标对象进行匹配，以将标准图像与实时图像进行匹配融合，对同一目标进行匹配，使得融合后的当前图像的准确度更高，更贴近真实。In this embodiment, the image processing device matches and fuses the standard image and the real-time image by matching the first target object in the standard image and the first target object in the real-time image to match the same target, so that the fusion The resulting current image is more accurate and closer to reality.

以上实施例中，对第二目标对象进行了实时的跟踪显示，以给医生以指导，且为了方便医生实时查看第二目标对象的位置，在后续的实施例中引入了增强现实设备，通过增强现实设备的视觉空间显示包括第二目标对象的混合现实图像，以更加方便医生查看第二目标对象的位置。In the above embodiment, the second target object is tracked and displayed in real time to guide the doctor, and in order to facilitate the doctor to check the position of the second target object in real time, an augmented reality device is introduced in the subsequent embodiments. The visual space of the reality device displays the mixed reality image including the second target object, so that it is more convenient for the doctor to check the position of the second target object.

在其中一个实施例中，标准图像中包括第一参照物；图像处理方法还包括：通过增强现实设备采集得到真实图像，真实图像中包括第一参照物；基于第一参照物将标准图像与真实图像进行匹配；根据标准图像与真实图像的匹配关系，以及标准图像与实时图像的匹配关系，将当前图像与真实图像匹配融合得到混合现实图像，并通过增强现实设备显示混合现实图像。In one embodiment, the standard image includes a first reference object; the image processing method further includes: acquiring a real image through an augmented reality device, where the real image includes a first reference object; and comparing the standard image with the real image based on the first reference object Image matching; according to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image, the current image and the real image are matched and fused to obtain a mixed reality image, and the mixed reality image is displayed through an augmented reality device.

其中，增强现实设备为能实现增强现实技术(AR)的设备，优选地，增强现实设备为增强现实眼镜(AR眼镜)。真实图像为AR眼镜中的图像获取设备获取的手术区域的图像。混合现实图像显示在增强现实设备的视觉空间的图像，其中该混合现实图像是将实时图像和真实图像进行匹配融合后得到的。The augmented reality device is a device capable of implementing augmented reality technology (AR). Preferably, the augmented reality device is augmented reality glasses (AR glasses). The real image is the image of the surgical area acquired by the image acquisition device in the AR glasses. The mixed reality image is displayed in the visual space of the augmented reality device, wherein the mixed reality image is obtained by matching and fusing the real-time image and the real image.

具体地，图像处理设备通过AR眼镜采集手术区域的真实图像，此真实图像中包括具体手术部位上的基准靶标的位置。图像处理设备基于基准靶标分别在标准图像与真实图像的位置进行匹配，获取标准图像与真实图像的匹配关系。根据标准图像与真实图像的匹配关系以及根据标准图像与实时图像的匹配关系，以标准图像为桥梁，获得实时图像与真实图像的映射关系，以此将实时图像以及标准图像映射到真实图像中，融合得到混合显示图像，并通过增强现实设备显示混合现实图像。Specifically, the image processing device collects a real image of the surgical area through the AR glasses, and the real image includes the position of the reference target on the specific surgical site. The image processing device matches the positions of the standard image and the real image respectively based on the reference target, and obtains the matching relationship between the standard image and the real image. According to the matching relationship between the standard image and the real image and the matching relationship between the standard image and the real-time image, the standard image is used as a bridge to obtain the mapping relationship between the real-time image and the real image, so as to map the real-time image and the standard image to the real image, The mixed display image is obtained by fusion, and the mixed reality image is displayed through the augmented reality device.

具体地，图像处理设备通过AR眼镜采集手术区域的图像，此图像中包括具体手术部位上的基准靶标的位置。术前三维图则包括与基准靶标位置对应的标志物，这样图像处理设备基于基准靶标以及术前三维图中的对应的标志物分别将术前三维图与AR眼镜采集的真实图像进行匹配，获取术前三维图与AR眼镜采集的真实图像的匹配关系，根据术前三维图与AR眼镜采集的真实图像的匹配关系以及根据术前三维图与三维实时图像的匹配关系，通过术前三维图这个桥梁使得三维实时图像与AR眼镜采集手术区域的图像相匹配，融合得到手术过程中的手术区域的混合现实图像。Specifically, the image processing device collects an image of the surgical area through the AR glasses, and the image includes the position of the reference target on the specific surgical site. The preoperative 3D map includes markers corresponding to the position of the reference target, so that the image processing device respectively matches the preoperative 3D map with the real image collected by the AR glasses based on the reference target and the corresponding markers in the preoperative 3D map, and obtains The matching relationship between the preoperative 3D image and the real image collected by the AR glasses, according to the matching relationship between the preoperative 3D image and the real image collected by the AR glasses, and the matching relationship between the preoperative 3D image and the 3D real-time image, through the preoperative 3D image. The bridge makes the three-dimensional real-time image match the image of the surgical area collected by the AR glasses, and fuses the mixed reality image of the surgical area during the operation.

在本实施例中，图像处理设备通过增强现实设备采集得到真实图像，将当前图像与真实图像匹配融合得到混合现实图像，并通过增强现实设备显示混合现实图像，使得用户可以通过增强现实眼镜看到真实环境下的三维实时图像。In this embodiment, the image processing device acquires a real image through an augmented reality device, matches and fuses the current image with the real image to obtain a mixed reality image, and displays the mixed reality image through the augmented reality device, so that the user can see through the augmented reality glasses 3D real-time images of real environments.

在一个实施例中，基于第一参照物将标准图像与真实图像进行匹配，包括：通过光学定位跟踪设备采集目标空间的参考图像，参考图像包括第一参照物；根据第一参照物将参考图像与增强现实设备的所采集的图像坐标系下的真实图像进行配准，以获取增强现实设备的图像坐标系下的真实图像与参考图像的匹配关系；根据参考图像与增强现实设备的所采集的图像坐标系下的真实图像匹配关系、图像坐标系与视觉空间坐标系的匹配关系，将参考图像与视觉空间坐标系下的真实图像进行配准；将与参考图像配准后的视觉空间坐标系下的真实图像与标准图像进行匹配。In one embodiment, matching the standard image with the real image based on the first reference object includes: collecting a reference image in the target space through an optical positioning and tracking device, where the reference image includes the first reference object; and matching the reference image according to the first reference object It is registered with the real image in the image coordinate system of the augmented reality device to obtain the matching relationship between the real image and the reference image in the image coordinate system of the augmented reality device; The matching relationship between the real image in the image coordinate system, the matching relationship between the image coordinate system and the visual space coordinate system, the reference image is registered with the real image in the visual space coordinate system; the visual space coordinate system after registration with the reference image is registered. The real image below is matched with the standard image.

其中，光学定位跟踪设备为定位设备。目标空间为手术区域的三维空间。参考图像为手术区域的三维空间图像，其是光学定位跟踪设备采集的，也即收取区域的真实的位置的图像。图像坐标系为描述AR眼镜采集的图像维度的坐标系，即将真实空间转换为AR眼镜的图像坐标系。视觉空间坐标系为AR眼镜显示的手术区域内的三维空间坐标系。The optical positioning and tracking device is a positioning device. The target space is the three-dimensional space of the surgical area. The reference image is a three-dimensional space image of the surgical area, which is acquired by the optical positioning and tracking device, that is, an image of the real position of the acquisition area. The image coordinate system is a coordinate system that describes the dimensions of the image captured by the AR glasses, that is, the real space is converted into the image coordinate system of the AR glasses. The visual space coordinate system is the three-dimensional space coordinate system in the surgical area displayed by the AR glasses.

具体地，图像处理设备通过光学定位跟踪设备采集手术区域的三维空间图像，此三维空间图像包括第一参照物即基准靶标的三维位置坐标。图像处理设备根据基准靶标的三维位置坐标和增强现实设备的图像坐标系下的基准靶标的图像位置坐标进行配准，以获取手术区域的真实空间图像与增强现实设备的图像坐标系的空间变换关系。其中，三维位置坐标为基准靶标在手术区域的真实空间图像中的坐标，图像位置坐标为基准靶标在增强现实设备的图像坐标系下的坐标。此外，图像处理设备还可以通过特征识别的方法确定增强现实设备的图像坐标系和视觉空间坐标系之间的匹配关系，这样可以得到真实空间图像与增强现实设备的视觉空间下的混合显示图像的匹配关系。Specifically, the image processing device collects a three-dimensional space image of the operation area through the optical positioning and tracking device, and the three-dimensional space image includes the three-dimensional position coordinates of the first reference object, that is, the fiducial target. The image processing device performs registration according to the three-dimensional position coordinates of the reference target and the image position coordinates of the reference target under the image coordinate system of the augmented reality device, so as to obtain the spatial transformation relationship between the real space image of the surgical area and the image coordinate system of the augmented reality device . The three-dimensional position coordinates are the coordinates of the reference target in the real space image of the surgical area, and the image position coordinates are the coordinates of the reference target in the image coordinate system of the augmented reality device. In addition, the image processing device can also determine the matching relationship between the image coordinate system of the augmented reality device and the visual space coordinate system by means of feature recognition, so that the mixed display image of the real space image and the visual space of the augmented reality device can be obtained. matching relationship.

在具体地实施过程中，以图4和图7为例进行说明。图像处理设备通过光学定位跟踪设备和各靶标之间地交互关系采集手术区域的三维空间图像，即采集手术区域内的各物体在三维空间内地绝对位置，并根据混合现实眼镜在三维空间内的位置建立世界坐标系{W}，通过世界坐标系来描述三维空间图像内的各物体的位置，且此三维空间图像包括第一参照物即基准靶标的三维位置坐标。图像处理设备将根据病人的术前三维图建立三维影像坐标系，并将三维影像坐标系记作{I}。医生在病人的目标脊柱上放置基准靶标，以作为标记点对目标脊柱进行标记。混合现实眼镜拍摄真实图像，真实图像中包括目标脊柱上的基准靶标。根据视觉重建算法，真实图像在世界坐标系{W}中表示为Wpc，医学影像三维图即术前三维图在三维影像坐标系{I}中表示为Ipc。其中，真实图像在世界坐标系{W}中表示为Wpc即为真实图像与三维空间图像配准后的结果。根据真实图像建立图像坐标系。利用系统软件配准模块中的配准算法将Wpc、Ipc以及图像坐标系配准，得到影像坐标系{I}到世界坐标系{W}的空间变化关系，通过可视化模块处理在混合现实眼镜中呈现虚拟的三维重建图叠加在真实场景中病人脊柱图即在混合现实眼镜中呈现混合现实图像。在本实施例中，图像处理设备通过术前获取的标准图像和术中真实图像进行配准，并将标准图像叠加显示到真实图像中，通过虚拟现实设备进行显示，在术中植入植入物的过程中不断重复该过程，真实图像中即会包含植入物体的影像，利用上述方式，医生能够直观了解钉在骨骼中的姿态和位置。In the specific implementation process, FIG. 4 and FIG. 7 are used as examples for description. The image processing device collects the three-dimensional space image of the surgical area through the interactive relationship between the optical positioning and tracking device and each target, that is, the absolute position of each object in the surgical area in the three-dimensional space is collected, and the position of the mixed reality glasses in the three-dimensional space is collected according to the position of the mixed reality glasses. A world coordinate system {W} is established, and the position of each object in the three-dimensional space image is described by the world coordinate system, and the three-dimensional space image includes the three-dimensional position coordinates of the first reference object, that is, the reference target. The image processing equipment will establish a three-dimensional image coordinate system according to the patient's preoperative three-dimensional map, and record the three-dimensional image coordinate system as {I}. The physician places a fiducial target on the patient's target spine as a marker point to mark the target spine. Mixed reality glasses capture real images that include fiducial targets on the target's spine. According to the visual reconstruction algorithm, the real image is represented as Wpc in the world coordinate system {W}, and the 3D medical image, ie the preoperative 3D image, is represented as Ipc in the 3D image coordinate system {I}. Among them, the real image is represented as Wpc in the world coordinate system {W}, which is the result of the registration of the real image and the three-dimensional space image. The image coordinate system is established based on the real image. Using the registration algorithm in the registration module of the system software to register the Wpc, Ipc and the image coordinate system, the spatial relationship between the image coordinate system {I} and the world coordinate system {W} is obtained, which is processed in the mixed reality glasses through the visualization module. Presenting the virtual 3D reconstruction image superimposed on the real scene of the patient's spine image is presenting the mixed reality image in the mixed reality glasses. In this embodiment, the image processing device performs registration through the standard image obtained before surgery and the real image during surgery, and superimposes and displays the standard image on the real image, displays it through the virtual reality device, and implants the implant during the surgery. Repeat this process continuously during the process of implanting the implant, and the real image will contain the image of the implanted object. Using the above method, the doctor can intuitively understand the posture and position of the nail in the bone.

可选地，光学定位跟踪设备选用NDI光学定位跟踪系统。Optionally, the optical positioning and tracking device selects the NDI optical positioning and tracking system.

在一个实施例中，真实图像中还包括第二参照物，第二参照物用于标定操作对象；上述图像处理方法还包括：根据标准图像与真实图像的匹配关系，以及标准图像与当前图像的匹配关系，将当前图像与真实图像匹配融合得到混合现实图像，将操作对象显示在混合现实图像中。In one embodiment, the real image further includes a second reference object, and the second reference object is used to calibrate the operation object; the above-mentioned image processing method further includes: according to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the current image Matching relationship, the current image is matched and fused with the real image to obtain a mixed reality image, and the operation object is displayed in the mixed reality image.

其中，第二参照物为操作对象上的靶标，用于获取操作对象的位置。操作对象为针对具体手术部位进行操作的机械。本实施例并不对操作对象上的靶标及操作对象的种类做限定，可以获取操作对象位置的任意种类靶标均可，操作对象的种类根据具体手术类型确定，满足对应手术类型即可。优选地，靶标选用光学靶标。Wherein, the second reference object is a target on the operation object, which is used to obtain the position of the operation object. The operation object is a machine that operates on a specific surgical site. This embodiment does not limit the target on the operation object and the type of the operation object. Any type of target that can obtain the position of the operation object can be used. The type of the operation object is determined according to the specific operation type and only needs to meet the corresponding operation type. Preferably, the target is an optical target.

具体地，AR眼镜中的图像获取设备获取的手术区域的真实图像中包括操作对象上的靶标，此靶标用于标识确定操作对象及其位置。图像处理设备根据标准图像与真实图像的匹配关系，将真实图像中的操作对象映射到标准图像中，进而根据标准图像与当前图像的匹配关系，将映射在标准图像中的操作对象映射到当前图像中，进而，在图像处理设备将当前图像与真实图像匹配融合得到混合现实图像后，操作对象显示在混合现实图像中。Specifically, the real image of the surgical area acquired by the image acquisition device in the AR glasses includes a target on the operation object, and the target is used to identify and determine the operation object and its position. The image processing device maps the operation object in the real image to the standard image according to the matching relationship between the standard image and the real image, and then maps the operation object mapped in the standard image to the current image according to the matching relationship between the standard image and the current image. , and further, after the image processing device matches and fuses the current image with the real image to obtain a mixed reality image, the operation object is displayed in the mixed reality image.

在本实施例中，图像处理设备将操作对象显示在混合现实图像中，使得操作者在同一视角可以同时观测到术中的实时图像及操作对象的位置，使得操作者的专注度更高，且便于调整。In this embodiment, the image processing device displays the operation object in the mixed reality image, so that the operator can simultaneously observe the intraoperative real-time image and the position of the operation object from the same viewing angle, so that the operator's concentration is higher, and Easy to adjust.

在一个实施例中，还包括：接收针对标准图像的操作信息配置指令，并根据配置指令建立操作信息与标准图像的对应关系；基于操作信息与标准图像的对应关系、标准图像与真实图像的匹配关系以及标准图像与实时图像的匹配关系，将操作信息显示在当前图像中。In one embodiment, the method further includes: receiving an operation information configuration instruction for the standard image, and establishing a corresponding relationship between the operation information and the standard image according to the configuration instruction; based on the corresponding relationship between the operation information and the standard image, the standard image and the real image are matched relationship and the matching relationship between the standard image and the real-time image, and display the operation information in the current image.

其中，操作信息为术前规划，其为手术方案，可以包括待手术部位的解剖信息以及与植入物相关的操作信息。The operation information is a preoperative plan, which is an operation plan, and may include anatomical information of the site to be operated and operation information related to the implant.

具体地，在获取标准图像后，手术执行者根据标准图像制定术前规划即手术计划，图像处理设备接收手术执行者针对标准图像的术前规划配置指令，并根据配置指令建立术前规划与标准图像的对应关系即根据配置指令将术前规划绘制到标准图像上。图像处理设备基于术前规划与标准图像的对应关系、标准图像与真实图像的匹配关系以及标准图像与实时图像的匹配关系，将术前规划显示在混合现实图像中。Specifically, after acquiring the standard image, the operator formulates a preoperative plan, that is, an operation plan, based on the standard image, and the image processing device receives the operator's preoperative planning configuration instruction for the standard image, and establishes the preoperative plan and standard according to the configuration instruction. The corresponding relationship of the images is to draw the preoperative planning on the standard image according to the configuration instructions. The image processing device displays the preoperative planning in the mixed reality image based on the corresponding relationship between the preoperative planning and the standard image, the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image.

在本实施例中，图像处理设备接收针对标准图像的操作信息配置指令，并根据配置指令建立操作信息与标准图像的对应关系，以在标准图像上制定术前规划。图像处理设备通过基于操作信息与标准图像的对应关系、标准图像与真实图像的匹配关系以及标准图像与实时图像的匹配关系，将操作信息显示在混合现实图像中，实现了将术前规划显示在术中的混合现实图像中，可以明确看到术前规划与现实情况是否存在较大偏差，确定术前规划与当前情况的适配性，增加手术的成功率。In this embodiment, the image processing device receives an operation information configuration instruction for the standard image, and establishes a corresponding relationship between the operation information and the standard image according to the configuration instruction, so as to formulate a preoperative plan on the standard image. The image processing equipment displays the operation information in the mixed reality image based on the corresponding relationship between the operation information and the standard image, the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image. In the mixed reality images during the operation, it is possible to clearly see whether there is a large deviation between the preoperative planning and the actual situation, to determine the adaptability of the preoperative planning to the current situation, and to increase the success rate of the operation.

在一个实施例中，还包括：通过增强现实设备接收针对操作信息的第一编辑指令；根据第一编辑指令对操作信息进行编辑，第一编辑指令包括移动、对齐、重新定向、旋转、放大中的至少一个。In one embodiment, the method further includes: receiving a first editing instruction for the operation information through an augmented reality device; editing the operation information according to the first editing instruction, where the first editing instruction includes moving, aligning, reorienting, rotating, and enlarging at least one of.

具体地，图像处理设备通过AR眼镜接收针对术前规划的包括移动、对齐、重新定向、旋转、放大中的至少一个第一编辑指令，并根据第一编辑指令对术前规划进行编辑。本实施例并不对接收第一编辑指令的方式做限定，满足接收指令的功能即可。Specifically, the image processing device receives at least one first editing instruction for the preoperative planning including movement, alignment, reorientation, rotation, and magnification through the AR glasses, and edits the preoperative planning according to the first editing instruction. This embodiment does not limit the manner of receiving the first editing instruction, as long as the function of receiving the instruction is satisfied.

在本实施例中，通过第一编辑指令对术前规划在AR眼镜中的显示进行调整，以对术前规划进行多样化显示，满足不同的需求场景。In this embodiment, the display of the preoperative plan in the AR glasses is adjusted through the first editing instruction, so as to display the preoperative plan in a diversified manner to meet different demand scenarios.

在一个实施例中，还包括：对混合现实图像进行截图以得到截图图像；将增强现实设备视野划分为混合现实图像显示区域以及截图图像显示区域；通过混合现实图像显示区域显示混合现实图像，并通过截图图像显示区域显示截图图像；通过截图图像显示区域接收针对截图图像的第二编辑指令，第二编辑指令包括移动、对齐、旋转、放大中的至少一个。In one embodiment, the method further includes: taking a screenshot of the mixed reality image to obtain the screenshot image; dividing the field of view of the augmented reality device into a mixed reality image display area and a screenshot image display area; displaying the mixed reality image through the mixed reality image display area, and The screenshot image is displayed through the screenshot image display area; the second editing instruction for the screenshot image is received through the screenshot image display area, and the second editing instruction includes at least one of moving, aligning, rotating, and zooming in.

具体地，图像处理设备接收对混合现实图像进行截图的指令，对混合现实图像进行截图以得到截图图像，将AR眼镜视野划分为混合现实图像显示区域以及截图图像显示区域。图像处理设备通过混合现实图像显示区域显示混合现实图像，并通过截图图像显示区域显示截图图像。图像处理设备通过截图图像显示区域接收针对截图图像的包括移动、对齐、旋转、放大中的至少一个第二编辑指令。优选地，图像处理设备接收第二编辑指令的方式为在虚拟面板上选择。本实施例并不对接收第二编辑指令的方式做限定，满足接收指令的功能即可。Specifically, the image processing device receives an instruction to take a screenshot of the mixed reality image, takes a screenshot of the mixed reality image to obtain the screenshot image, and divides the AR glasses field of view into a mixed reality image display area and a screenshot image display area. The image processing device displays the mixed reality image through the mixed reality image display area, and displays the screenshot image through the screenshot image display area. The image processing device receives at least one second editing instruction for the screenshot image including moving, aligning, rotating, and zooming in through the screenshot image display area. Preferably, the manner in which the image processing device receives the second editing instruction is selection on the virtual panel. This embodiment does not limit the manner of receiving the second editing instruction, as long as the function of receiving the instruction is satisfied.

在本实施例中，通过第二编辑指令对术前规划在AR眼镜中的截图图像的显示方式进行调整，以对术前规划进行多样化显示，满足不同的需求场景。In this embodiment, the display mode of the screenshot images of the preoperative planning in the AR glasses is adjusted through the second editing instruction, so as to display the preoperative planning in a variety of ways to meet different demand scenarios.

在一个实施例中，获取预先生成的标准图像之前，还包括：通过至少一种医学成像设备对包括有第一参照物的目标区域进行扫描得到医学图像；对医学图像进行三维重建得到标准图像。In an embodiment, before acquiring the pre-generated standard image, the method further includes: scanning a target area including the first reference object by at least one medical imaging device to obtain a medical image; and performing three-dimensional reconstruction on the medical image to obtain a standard image.

其中，医学成像设备用于获取医学图像的成像设备。医学图像，用于当前手术中的图像。本实施例并不对医学成像设备做限定，可以获取医学图像的任意设备均可。Among them, the medical imaging device is an imaging device for acquiring medical images. Medical images, for images in current surgery. This embodiment does not limit the medical imaging device, and any device that can acquire medical images may be used.

优选地，医学成像设备为核磁共振设备(MRI)和/或计算机断层扫描设备(CT)。Preferably, the medical imaging device is a nuclear magnetic resonance device (MRI) and/or a computed tomography device (CT).

可选地，通过MRI对包括基准靶标的手术区域进行扫描得到对应的医学图像，对上述医学图像进行三维重建得到标准图像。Optionally, a corresponding medical image is obtained by scanning the operation area including the reference target by MRI, and a standard image is obtained by performing three-dimensional reconstruction on the medical image.

可选地，通过CT对包括基准靶标的手术区域进行扫描得到对应的医学图像，对上述医学图像进行三维重建得到标准图像。Optionally, a corresponding medical image is obtained by scanning the operation area including the reference target by CT, and a standard image is obtained by performing three-dimensional reconstruction on the above medical image.

优选地，通过MRI和CT分别对包括基准靶标的手术区域进行扫描得到对应的医学图像，将上述两种医学图像进行多模态配准融合实现影像的三维重建得到标准图像。Preferably, the corresponding medical image is obtained by scanning the surgical area including the reference target by MRI and CT respectively, and the above two medical images are subjected to multi-modal registration and fusion to achieve three-dimensional reconstruction of the image to obtain the standard image.

在本实施例中，图像处理设备获取术前至少一种医学图像以进行三维重建得到标准图像，使得手术区域对应的标准图像更加真实、准确。In this embodiment, the image processing device acquires at least one preoperative medical image to perform three-dimensional reconstruction to obtain a standard image, so that the standard image corresponding to the operation area is more realistic and accurate.

在一个实施例中，如图8所示，在术前，图像处理设备获取根据医学影像处理模块生成的标准图像。图像处理设备通过手术规划模块在标准图像上生成术前规划。在术中，获取三维超声实时图像，三维超声图像中包括具体手术部位的植入物，通过配准模块将三维超声实时图像与标准图像进行配准后通过视觉处理模块得到当前图像，混合现实眼镜获取手术区域的真实图像，配准模块将真实图像与当前图像再次进行配准后通过视觉处理模块生成混合现实图像，将混合现实图像传递给混合现实眼镜的可视化模块进行处理后显示，医生穿戴混合现实眼镜进行手术。其中，跟踪引导模块负责处理随动，使得带有术前计划的当前图像映射到真实图像即真实脊柱时可以随着混合现实眼镜的移动而随动。各模块之间的信息传递通过通信模块进行。In one embodiment, as shown in FIG. 8 , before surgery, the image processing device acquires a standard image generated according to a medical image processing module. The image processing equipment generates the preoperative plan on the standard image through the surgical planning module. During the operation, the 3D ultrasound real-time image is obtained, and the 3D ultrasound image includes the implant of the specific surgical site. The 3D ultrasound real-time image is registered with the standard image through the registration module, and then the current image is obtained through the vision processing module. Mixed reality glasses The real image of the surgical area is acquired, the registration module re-registers the real image with the current image, and then generates a mixed reality image through the vision processing module, and transmits the mixed reality image to the visualization module of the mixed reality glasses for processing and display. Realistic glasses for surgery. Among them, the tracking and guidance module is responsible for processing the follow-up, so that when the current image with the preoperative plan is mapped to the real image, that is, the real spine, it can follow the movement of the mixed reality glasses. The information transfer between the modules is carried out through the communication module.

在本实施例中，图像处理设备通过获取的三维超声图像中包括具体手术部位的植入物将上述植入物显示在混合现实图像中，实现了将术中带有植入物的图像实时更新，进而实现通过手术中收集的图像进行术中操作的实时导航，且对操作的过程实时可视。In this embodiment, the image processing device displays the implant in the mixed reality image by including the implant at the specific surgical site in the acquired three-dimensional ultrasound image, so that the intraoperative image with the implant is updated in real time , so as to realize real-time navigation of intraoperative operations through images collected during the operation, and real-time visualization of the operation process.

在一个实施例中，如图9所示，术前规划的具体制定过程如下：In one embodiment, as shown in Figure 9, the specific formulation process of the preoperative planning is as follows:

手术前，在患者具体手术部位上放置基准靶标的位置点放置标记物，对患者进行CT和MRI扫描以得到患者术前影像数据，并将上述数据发送至医学影像处理模块进行多模态配准融合，实现影像的三维重建并得到标准图像，且标准图像带有第一目标对象，例如，标准图像中包括目标脊柱。图像处理设备将上述标准图像进行两个处理，一个是将上述标准图像发送至配准模块，另一个是接收手术执行者在标准图像的基础上制定的术前规划，并将带有术前规划的标准图像发送至配准模块。医生在佩戴好混合现实眼镜后，进行手术前，混合现实眼镜中的摄像头获取实时的患者具体手术部位的真实图像，例如获取目标脊柱对应的真实骨骼图像，且真实图像带有三维空间坐标系对应的坐标，将上述真实图像发送至配准模块。在配准模块中，利用配准算法将标准图像与真实图像进行配准，通过可视化模块将带有术前规划的标准图像叠加在目标脊柱对应的真实骨骼图像上，生成术前混合现实图像。混合现实眼镜显示上述术前混合现实图像，使医生可以同时观看患者组织结构信息与实际物理信息以及术前规划。同时，虚拟现实眼镜可根据医生发出的第一编辑指令调整术前规划，例如，如对术前重构的三维骨骼图像数据进行移动、对齐、重新定向、旋转、平移、放大等或者更改术前规划中的器械轴向等。图像处理设备将修改后的术前规划发送至配准模块进行重新配准后，发送至混合现实眼镜进行显示。Before surgery, place markers at the positions where the reference target is placed on the specific surgical site of the patient, perform CT and MRI scans on the patient to obtain preoperative image data of the patient, and send the above data to the medical image processing module for multimodal registration. Fusion, realizes the three-dimensional reconstruction of the image and obtains a standard image, and the standard image contains the first target object, for example, the standard image includes the target spine. The image processing equipment performs two processes on the above-mentioned standard images, one is to send the above-mentioned standard images to the registration module, and the other is to receive the preoperative planning made by the operator on the basis of the standard images, with the preoperative planning. The standard image is sent to the registration module. After the doctor wears the mixed reality glasses and before the operation, the camera in the mixed reality glasses obtains the real image of the specific surgical site of the patient in real time, such as obtaining the real bone image corresponding to the target spine, and the real image corresponds to the three-dimensional space coordinate system. The coordinates of the above-mentioned real image are sent to the registration module. In the registration module, the registration algorithm is used to register the standard image with the real image, and the standard image with preoperative planning is superimposed on the real bone image corresponding to the target spine through the visualization module to generate a preoperative mixed reality image. Mixed reality glasses display the aforementioned preoperative mixed reality images, allowing doctors to simultaneously view patient tissue information along with actual physical information and preoperative planning. At the same time, the virtual reality glasses can adjust the preoperative planning according to the first editing instructions issued by the doctor, for example, such as moving, aligning, reorienting, rotating, translating, zooming, etc. on the preoperatively reconstructed 3D bone image data or changing the preoperative planning The planned instrument axis, etc. The image processing device sends the modified preoperative plan to the registration module for re-registration, and then sends it to the mixed reality glasses for display.

在一个实施例中，通过图10说明图像处理设备中各模块的交互关系。In one embodiment, the interaction relationship of each module in the image processing device is illustrated by FIG. 10 .

真实手术空间包括手术器械即手术对象即患者。The real surgical space includes surgical instruments, that is, the surgical object, that is, the patient.

在术前，医学影像处理模块处理术前拍摄的CT和MRI的影像，将上述影像进行图像融合和三维重建生成标准图像，将标准图像及其包含的空间坐标提供给配准模块。通过手术规划模块，医生可以根据标准图像进行术前手术规划，并将带有手术规划的标准图像提供给配准模块，混合现实眼镜拍摄真实目标脊柱图像即真实图像并提供到配准模块。配准模块根据标准图像和真实图像以及根据混合现实眼镜建立的世界坐标系即参考图像进行配准，并将配准后的结果提供给视觉处理模块以将标准图像与真实图像进行处理生成混合现实图像，将混合现实图像通过通信模块传输给术前混合现实眼镜并进行显示。Before the operation, the medical image processing module processes the CT and MRI images taken before the operation, performs image fusion and 3D reconstruction on the above images to generate a standard image, and provides the standard image and its contained spatial coordinates to the registration module. Through the surgical planning module, doctors can perform preoperative surgical planning based on standard images, and provide the standard images with surgical planning to the registration module. Mixed reality glasses capture real target spine images, that is, real images, and provide them to the registration module. The registration module performs registration according to the standard image and the real image and the world coordinate system established according to the mixed reality glasses, that is, the reference image, and provides the registered result to the vision processing module to process the standard image and the real image to generate mixed reality. The mixed reality image is transmitted to the preoperative mixed reality glasses through the communication module and displayed.

在术中，混合现实眼镜继续获取包括目标脊柱和手术器械图像和坐标的真实图像，并将上述真实图像提供给配准模块，图像处理模块将术中超声设备拍摄的多体位二维超声图像进行三维重建并跟术前的CT/MRI三维重建图进行融合生成当前图像，将当前图像提供给配准模块。配准模块根据标准图像、三维超声重建图像、真实图像以及根据混合现实眼镜建立的世界坐标系即参考图像进行配准，并将配准后的结果提供给视觉处理模块以将当前图像与真实图像进行处理生成术中混合现实图像，将术中混合现实图像通过通信模块传输给混合现实眼镜并进行显示。During the operation, the mixed reality glasses continue to obtain real images including the images and coordinates of the target spine and surgical instruments, and provide the above real images to the registration module, and the image processing module processes the multi-position two-dimensional ultrasound images captured by the intraoperative ultrasound equipment 3D reconstruction and fusion with the preoperative CT/MRI 3D reconstruction image to generate the current image, and provide the current image to the registration module. The registration module performs registration according to the standard image, the 3D ultrasound reconstruction image, the real image, and the world coordinate system established according to the mixed reality glasses, that is, the reference image, and provides the registered result to the vision processing module to compare the current image with the real image. Perform processing to generate an intraoperative mixed reality image, and transmit the intraoperative mixed reality image to the mixed reality glasses through the communication module for display.

跟踪引导模块负责处理随动，使得带有术前规划的标准图形映射到真实图像时可以随着混合现实眼镜的移动而随动。The tracking and guidance module is responsible for processing the follow-up, so that the standard graphics with preoperative planning can follow the movement of the mixed reality glasses when they are mapped to the real image.

在一个实施例中，还包括：获取生理监测系统采集的生理信号；根据生理信号生成报警信息；显示报警信息。In one embodiment, the method further includes: acquiring physiological signals collected by the physiological monitoring system; generating alarm information according to the physiological signals; and displaying the alarm information.

其中，生理监测系统为监测患者生理信号的监测系统。生理信号为监测系统监测的对应的生理信号。本实施例并不对生理监测系统的类型做限定，满足对应手术的监测需求即可。The physiological monitoring system is a monitoring system for monitoring physiological signals of a patient. The physiological signal is the corresponding physiological signal monitored by the monitoring system. This embodiment does not limit the type of the physiological monitoring system, as long as the monitoring requirements of the corresponding operation are met.

具体的，以脊柱手术为例，图像处理设备还获取神经电生理监测系统采集的肌电信号数据，根据肌电信号数据生成报警信息并将其显示。Specifically, taking spine surgery as an example, the image processing device also acquires the electromyographic signal data collected by the neurophysiological monitoring system, and generates and displays alarm information according to the electromyographic signal data.

在一个实施例中，具体的，以脊柱手术为例，如图11所示，在术中，由于脊椎中有重要神经和血管，在植入植入物的过程中图像处理设备还利用神经电生理监测系统进行肌电信号数据的实时监测，并配合实时图像所显示的植入物的位置，使得在穿刺过程中可以及时有效检测神经根位置，有效避免神经损伤，也使医生能够准确判断患者的手术过程是否顺利。In one embodiment, specifically, taking spine surgery as an example, as shown in FIG. 11 , during surgery, since there are important nerves and blood vessels in the spine, the image processing device also uses neuroelectrical The physiological monitoring system performs real-time monitoring of EMG data, and cooperates with the position of the implant displayed by the real-time image, so that the position of the nerve root can be detected in a timely and effective manner during the puncture process, effectively avoiding nerve damage, and enabling doctors to accurately judge the patient. whether the surgery went well.

在本实施例中，根据采集的生理信号辅助判断手术过程是否顺利，并及时根据生理信号生成报警信息并进行对应的提示。In this embodiment, the collected physiological signals are used to assist in judging whether the surgical procedure is successful, and alarm information is generated in time according to the physiological signals and corresponding prompts are given.

在一个实施例中，还包括：将生理信号和/或生理信号对应的报警信息发送至增强现实设备，并通过增强现实设备进行显示。In one embodiment, the method further includes: sending the physiological signal and/or alarm information corresponding to the physiological signal to the augmented reality device, and displaying it through the augmented reality device.

具体地，图像处理设备将生理信号和/或生理信号对应的报警信息发送至AR眼镜，并提供AR眼镜进行显示。Specifically, the image processing device sends the physiological signal and/or alarm information corresponding to the physiological signal to the AR glasses, and provides the AR glasses for display.

在本实施例中，提供AR眼镜将生理信号和/或生理信号对应的报警信息进行显示，使得手术执行者可以第一时间获取到相应的数据，以对手术及时进行调整。In this embodiment, AR glasses are provided to display the physiological signal and/or alarm information corresponding to the physiological signal, so that the operator can obtain the corresponding data at the first time, so as to adjust the operation in time.

在一个实施例中，包括：获取预先设置的报警等级参数；将生理信号与报警等级参数进行匹配以确定报警等级。In one embodiment, it includes: acquiring preset alarm level parameters; and matching physiological signals with the alarm level parameters to determine the alarm level.

其中，预先设置的报警等级参数为根据生理信号的取值范围预先划分等级。Wherein, the preset alarm level parameter is to divide the level in advance according to the value range of the physiological signal.

具体地，预先根据生理信号的取值范围划分对应的等级参数，图像处理设备获取预先设置的报警等级参数，将当前生理信号与报警等级参数进行匹配以确定报警等级。Specifically, the corresponding level parameters are divided in advance according to the value range of the physiological signal, the image processing device obtains the preset alarm level parameter, and matches the current physiological signal with the alarm level parameter to determine the alarm level.

在具体地实施过程中，以图12为例，根据手术部位选择适宜的术中神经电生理监测方案。其中，体感诱发电位监测需分侧进行，刺激部位根据手术要求而定，可选地，上肢刺激电极的阳极置于腕横纹处、阴极置于距腕横纹2～3cm处，刺激正中神经或尺神经；下肢感觉刺激选择胫后神经，刺激电极的阳极置于胫后神经走行的内踝远端2～3cm处、阴极置于内踝与跟腱之间的近脚踝处，也可选择腓神经或坐骨神经。例如，在脊柱脊髓手术中体感诱发电位监测通常采用恒流电刺激，脉宽0.20～0.30ms，刺激频率为5Hz，刺激强度以观察到明显的手指或足趾活动为宜，上肢15～35mA、下肢30～60mA，并在监测过程中保持稳定。根据呈现的峰值波幅设定安全等级，传输给图像处理设备进行处理，并经过通信模块显示在混合现实眼镜。同时根据分类的安全等级发出预警信号，并根据预警信号生成对应的反馈指令，经通信模块将反馈指令传递给机械臂。In the specific implementation process, taking FIG. 12 as an example, an appropriate intraoperative neurophysiological monitoring scheme is selected according to the surgical site. Among them, the somatosensory evoked potential monitoring needs to be carried out side by side, and the stimulation site is determined according to the surgical requirements. Or ulnar nerve; lower limb sensory stimulation choose posterior tibial nerve, the anode of the stimulation electrode is placed 2-3 cm distal to the medial malleolus where the posterior tibial nerve runs, and the cathode is placed near the ankle between the medial malleolus and the Achilles tendon, and the peroneal nerve can also be selected or the sciatic nerve. For example, in spinal cord surgery, somatosensory evoked potential monitoring usually uses constant current electrical stimulation with a pulse width of 0.20 to 0.30 ms and a stimulation frequency of 5 Hz. Lower extremity 30 ~ 60mA, and remained stable during the monitoring process. The security level is set according to the presented peak amplitude, transmitted to the image processing device for processing, and displayed on the mixed reality glasses through the communication module. At the same time, an early warning signal is issued according to the classified security level, and a corresponding feedback command is generated according to the early warning signal, and the feedback command is transmitted to the robotic arm through the communication module.

在本实施例中，图像处理设备提供对生理信号划分等级，来显示病人的生理状况，以此显示手术的状况，便于及时调整。In this embodiment, the image processing device provides levels of physiological signals to display the physiological condition of the patient, so as to display the condition of the operation and facilitate timely adjustment.

优选地，混合现实眼镜还可以显示手术器械的角度，如图13所示。Preferably, the mixed reality glasses can also display the angle of the surgical instrument, as shown in FIG. 13 .

图14，为一个实施例的手术操作图。图15为各设备之间通信方式。Figure 14 is a diagram of a surgical operation of one embodiment. Figure 15 shows the communication mode between the devices.

应该理解的是，虽然图5、6、8、10-13的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图5、6、8、10-13中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。It should be understood that although the steps in the flowcharts of FIGS. 5 , 6 , 8 , and 10-13 are displayed in sequence according to the arrows, these steps are not necessarily executed in the sequence indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in Figs. 5, 6, 8, 10-13 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be executed at different times, The order of execution of these steps or stages is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages in the other steps.

在一个实施例中，提供了一种计算机设备，该计算机设备可以是图像处理设备，其内部结构图可以如图16所示。该计算机设备包括通过系统总线连接的处理器、存储器、通信接口、显示屏和输入装置。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的通信接口用于与外部的图像处理设备进行有线或无线方式的通信，无线方式可通过WIFI、运营商网络、NFC(近场通信)或其他技术实现。该计算机程序被处理器执行时以实现一种图像处理方法。该计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该计算机设备的输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。In one embodiment, a computer device is provided, and the computer device may be an image processing device, and its internal structure diagram may be as shown in FIG. 16 . The computer equipment includes a processor, memory, a communication interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with an external image processing device, and the wireless communication can be realized by WIFI, operator network, NFC (Near Field Communication) or other technologies. The computer program implements an image processing method when executed by a processor. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

本领域技术人员可以理解，图16中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。Those skilled in the art can understand that the structure shown in FIG. 16 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

在一个实施例中，还提供了一种计算机设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现上述各方法实施例中的步骤。In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述各方法实施例中的步骤。In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory，ROM)、磁带、软盘、闪存或光存储器等。易失性存储器可包括随机存取存储器(Random Access Memory，RAM)或外部高速缓冲存储器。作为说明而非局限，RAM可以是多种形式，比如静态随机存取存储器(Static Random Access Memory，SRAM)或动态随机存取存储器(Dynamic Random Access Memory，DRAM)等。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided in this application may include at least one of non-volatile and volatile memory. The non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, and the like. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM).

以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (24)

Translated fromChinese

1.一种图像处理方法，其特征在于，所述方法包括：1. an image processing method, is characterized in that, described method comprises:获取目标区域的实时图像，所述实时图像包括第一目标对象以及第二目标对象；acquiring a real-time image of the target area, where the real-time image includes a first target object and a second target object;获取预先生成的标准图像；Get pre-generated standard images;根据所述第一目标对象将所述标准图像与所述实时图像进行匹配融合得到当前图像；According to the first target object, the standard image and the real-time image are matched and fused to obtain a current image;根据所述第二目标对象在所述实时图像中的位置，将所述第二目标对象显示在所述当前图像中。The second target object is displayed in the current image according to the position of the second target object in the real-time image.2.根据权利要求1所述的方法，其特征在于，所述获取目标区域的实时图像，包括：2. The method according to claim 1, wherein the acquiring the real-time image of the target area comprises:获取扫描得到的目标区域的扫描图像，所述扫描图像中包括第一目标对象以及第二目标对象，且所述扫描图像携带有与第一参照物对应的第一位置，所述第一参照物用于标定所述第一目标对象；Obtain a scanned image of the target area obtained by scanning, the scanned image includes a first target object and a second target object, and the scanned image carries a first position corresponding to a first reference object, the first reference object for calibrating the first target object;根据所述第一位置和所述第一目标对象对所述扫描图像进行重建得到实时图像，所述实时图像包括第一目标对象和第二目标对象。A real-time image is obtained by reconstructing the scanned image according to the first position and the first target object, and the real-time image includes a first target object and a second target object.3.根据权利要求2所述的方法，其特征在于，所述第一位置为图像采集设备与目标区域中的第一参照物的相对位置；所述根据所述第一位置和所述第一目标对象重建得到实时图像，包括：3. The method according to claim 2, wherein the first position is a relative position between the image acquisition device and the first reference object in the target area; The target object is reconstructed to obtain a real-time image, including:根据所述相对位置将所述扫描图像按照预设方向进行排序；Sorting the scanned images in a preset direction according to the relative position;识别所述扫描图像中的第一目标对象；identifying the first target object in the scanned image;基于排序后的所述扫描图像以及所述第一目标对象对所述扫描图像进行重建得到实时图像。A real-time image is obtained by reconstructing the scanned image based on the sorted scanned image and the first target object.4.根据权利要求3所述的方法，其特征在于，所述根据所述第一目标对象将所述标准图像与所述实时图像进行匹配融合得到当前图像，包括：4. The method according to claim 3, wherein the obtaining a current image by matching and fusing the standard image and the real-time image according to the first target object, comprising:识别所述标准图像中的第一目标对象；identifying the first target object in the standard image;将所述标准图像中的第一目标对象以及所述实时图像中的第一目标对象进行匹配，以将所述标准图像与所述实时图像进行匹配融合得到当前图像。The first target object in the standard image and the first target object in the real-time image are matched to obtain a current image by matching and fusing the standard image and the real-time image.5.根据权利要求1至4任意一项所述的方法，其特征在于，所述标准图像中包括第一参照物；所述方法还包括：5. The method according to any one of claims 1 to 4, wherein the standard image includes a first reference object; the method further comprises:通过增强现实设备采集得到真实图像，所述真实图像中包括第一参照物；A real image is acquired through an augmented reality device, and the real image includes a first reference object;基于所述第一参照物将所述标准图像与所述真实图像进行匹配；matching the standard image with the real image based on the first reference;根据所述标准图像与所述真实图像的匹配关系，以及所述标准图像与所述实时图像的匹配关系，将所述当前图像与所述真实图像匹配融合得到混合现实图像，并通过所述增强现实设备显示所述混合现实图像。According to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image, the current image and the real image are matched and fused to obtain a mixed reality image, and through the enhancement A reality device displays the mixed reality image.6.根据权利要求5所述的方法，其特征在于，所述基于所述第一参照物将所述标准图像与所述真实图像进行匹配，包括：6. The method according to claim 5, wherein the matching the standard image with the real image based on the first reference object comprises:通过光学定位跟踪设备采集目标空间的参考图像，所述参考图像包括第一参照物；Collect a reference image of the target space by using an optical positioning and tracking device, the reference image includes a first reference object;根据所述第一参照物将所述参考图像与所述增强现实设备的所采集的图像坐标系下的真实图像进行配准，以获取所述增强现实设备的图像坐标系下的真实图像与所述参考图像的匹配关系；The reference image is registered with the real image in the image coordinate system collected by the augmented reality device according to the first reference object, so as to obtain the real image in the image coordinate system of the augmented reality device and the real image in the image coordinate system. The matching relationship of the reference image;根据所述参考图像与所述增强现实设备的所采集的图像坐标系下的真实图像匹配关系、所述图像坐标系与视觉空间坐标系的匹配关系，将所述参考图像与所述视觉空间坐标系下的真实图像进行配准；According to the matching relationship between the reference image and the real image in the image coordinate system collected by the augmented reality device, and the matching relationship between the image coordinate system and the visual space coordinate system, the reference image and the visual space coordinate system are matched. The real image under the system is registered;将与所述参考图像配准后的所述视觉空间坐标系下的真实图像与所述标准图像进行匹配。The real image in the visual space coordinate system after being registered with the reference image is matched with the standard image.7.根据权利要求5所述的方法，其特征在于，所述真实图像中还包括第二参照物，所述第二参照物用于标定操作对象；所述方法还包括：7. The method according to claim 5, wherein the real image further comprises a second reference object, and the second reference object is used to calibrate the operation object; the method further comprises:根据所述标准图像与所述真实图像的匹配关系，以及所述标准图像与所述当前图像的匹配关系，将所述当前图像与所述真实图像匹配融合得到混合现实图像，将所述操作对象显示在所述混合现实图像中。According to the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the current image, the current image and the real image are matched and fused to obtain a mixed reality image, and the operation object displayed in the mixed reality image.8.根据权利要求5所述的方法，其特征在于，所述方法还包括：8. The method according to claim 5, wherein the method further comprises:接收针对标准图像的操作信息配置指令，并根据所述配置指令建立所述操作信息与所述标准图像的对应关系；receiving an operation information configuration instruction for a standard image, and establishing a corresponding relationship between the operation information and the standard image according to the configuration instruction;基于所述操作信息与所述标准图像的对应关系、所述标准图像与所述真实图像的匹配关系以及所述标准图像与所述实时图像的匹配关系，将所述操作信息显示在所述混合现实图像中。Based on the corresponding relationship between the operation information and the standard image, the matching relationship between the standard image and the real image, and the matching relationship between the standard image and the real-time image, the operation information is displayed in the mixed in real images.9.根据权利要求8所述的方法，其特征在于，所述方法还包括：9. The method according to claim 8, wherein the method further comprises:通过所述增强现实设备接收针对所述操作信息的第一编辑指令；receiving, by the augmented reality device, a first editing instruction for the operation information;根据所述第一编辑指令对所述操作信息进行编辑，所述第一编辑指令包括移动、对齐、重新定向、旋转、放大中的至少一个。The operation information is edited according to the first editing instruction, where the first editing instruction includes at least one of moving, aligning, reorienting, rotating, and zooming in.10.根据权利要求9所述的方法，其特征在于，所述方法还包括：10. The method according to claim 9, wherein the method further comprises:对所述混合现实图像进行截图以得到截图图像；taking a screenshot of the mixed reality image to obtain a screenshot image;将所述增强现实设备视野划分为混合现实图像显示区域以及所述截图图像显示区域；dividing the field of view of the augmented reality device into a mixed reality image display area and the screenshot image display area;通过所述混合现实图像显示区域显示所述混合现实图像，并通过所述截图图像显示区域显示所述截图图像；Display the mixed reality image through the mixed reality image display area, and display the screenshot image through the screenshot image display area;通过所述截图图像显示区域接收针对所述截图图像的第二编辑指令，所述第二编辑指令包括移动、对齐、旋转、放大中的至少一个。A second editing instruction for the screenshot image is received through the screenshot image display area, and the second editing instruction includes at least one of moving, aligning, rotating, and zooming in.11.根据权利要求1所述的方法，其特征在于，所述获取预先生成的标准图像之前，还包括：11. The method according to claim 1, wherein before acquiring the pre-generated standard image, the method further comprises:通过至少一种医学成像设备对包括有第一参照物的所述目标区域进行扫描得到医学图像；Scan the target area including the first reference object by at least one medical imaging device to obtain a medical image;对所述医学图像进行三维重建得到标准图像。Three-dimensional reconstruction is performed on the medical image to obtain a standard image.12.根据权利要求1所述的方法，其特征在于，所述方法还包括：12. The method of claim 1, wherein the method further comprises:获取生理监测系统采集的生理信号；Obtain the physiological signals collected by the physiological monitoring system;根据所述生理信号生成报警信息；generating alarm information according to the physiological signal;显示所述报警信息。The alarm message is displayed.13.根据权利要求12所述的方法，其特征在于，所述方法还包括：13. The method of claim 12, wherein the method further comprises:将所述生理信号和/或所述生理信号对应的报警信息发送至所述增强现实设备，并通过所述增强现实设备进行显示。The physiological signal and/or alarm information corresponding to the physiological signal is sent to the augmented reality device, and displayed by the augmented reality device.14.根据权利要求12所述的方法，其特征在于，所述根据所述生理信号生成报警信息，包括：14. The method according to claim 12, wherein the generating alarm information according to the physiological signal comprises:获取预先设置的报警等级参数；Get the preset alarm level parameters;将所述生理信号与所述报警等级参数进行匹配以确定报警等级。The physiological signal is matched to the alarm level parameter to determine an alarm level.15.一种图像处理系统，其特征在于，所述系统包括图像处理设备以及图像采集设备，所述图像处理设备与所述图像采集设备相通信；15. An image processing system, wherein the system comprises an image processing device and an image acquisition device, and the image processing device communicates with the image acquisition device;所述图像采集设备用于扫描目标区域并生成所述目标区域的实时图像；The image acquisition device is used to scan a target area and generate a real-time image of the target area;所述图像处理设备用于实现权利要求1至14任意一项所述的图像处理方法。The image processing device is used to implement the image processing method according to any one of claims 1 to 14.16.根据权利要求15所述的系统，其特征在于，所述图像采集设备为超声设备。16. The system of claim 15, wherein the image acquisition device is an ultrasound device.17.根据权利要求15所述的系统，其特征在于，所述系统还包括光学定位跟踪设备，所述光学定位跟踪设备与所述图像处理设备通信，所述光学定位跟踪设备用于采集参考图像，并将所述参考图像发送至所述图像处理设备。17. The system of claim 15, wherein the system further comprises an optical positioning and tracking device, the optical positioning and tracking device is in communication with the image processing device, the optical positioning and tracking device is used to collect reference images , and send the reference image to the image processing device.18.根据权利要求15所述的系统，其特征在于，所述还包括增强现实设备，所述增强现实设备与所述图像处理设备相通信，所述增强现实设备用于采集得到真实图像，并将所述真实图像发送至所述图像处理设备，所述真实图像中包括所述第一参照物。18. The system according to claim 15, characterized in that, it further comprises an augmented reality device, the augmented reality device communicates with the image processing device, the augmented reality device is used to acquire a real image, and The real image is sent to the image processing device, and the real image includes the first reference object.19.根据权利要求15所述的系统，其特征在于，所述系统还包括操作对象，所述操作对象是通过第二参照物标定的。19. The system according to claim 15, characterized in that, the system further comprises an operation object, and the operation object is calibrated by a second reference object.20.根据权利要求15所述的系统，其特征在于，所述系统还包括生理数据采集设备，所述生理数据采集设备与所述图像处理设备相通信；20. The system of claim 15, further comprising a physiological data acquisition device in communication with the image processing device;所述生理数据采集设备用于采集生理信号，并将所述生理信号发送至所述图像处理设备。The physiological data acquisition device is used for collecting physiological signals and sending the physiological signals to the image processing device.21.根据权利要求20所述的系统，其特征在于，所述系统还包括至少一种医学成像设备，至少一种所述医学成像设备与所述图像处理设备相通信，所述医学成像设备用于扫描得到医学图像，所述医学图像用于生成标准图像。21. The system of claim 20, wherein the system further comprises at least one medical imaging device, the at least one medical imaging device being in communication with the image processing device, the medical imaging device using Medical images are obtained from scanning, and the medical images are used to generate standard images.22.根据权利要求21所述的系统，其特征在于，所述医学成像设备包括核磁共振设备和计算机断层扫描设备。22. The system of claim 21, wherein the medical imaging device comprises a nuclear magnetic resonance device and a computed tomography device.23.一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至14中任一项所述的方法的步骤。23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 14 when the processor executes the computer program. step.24.一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至14中任一项所述的方法的步骤。24. A computer-readable storage medium on which a computer program is stored, wherein the computer program implements the steps of the method according to any one of claims 1 to 14 when the computer program is executed by a processor.

Images