CN114565741A - Operation-assisted data processing method, device, equipment, medium and system - Google Patents
Operation-assisted data processing method, device, equipment, medium and systemDownload PDFInfo
- Publication number
- CN114565741A CN114565741ACN202111630612.5ACN202111630612ACN114565741ACN 114565741 ACN114565741 ACN 114565741ACN 202111630612 ACN202111630612 ACN 202111630612ACN 114565741 ACN114565741 ACN 114565741A
- Authority
- CN
- China
- Prior art keywords
- model
- target object
- data processing
- surgical
- dynamic model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/003—Navigation within 3D models or images
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/101—Computer-aided simulation of surgical operations
- A61B2034/105—Modelling of the patient, e.g. for ligaments or bones
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Radar, Positioning & Navigation (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Robotics (AREA)
- Remote Sensing (AREA)
- Radiology & Medical Imaging (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本发明涉及医疗领域，尤其涉及一种手术辅助的数据处理方法、装置、设备、介质与系统。The present invention relates to the medical field, and in particular, to a method, device, equipment, medium and system for data processing of surgical assistance.
背景技术Background technique
在为手术提供辅助的手段中，可针对病人的目标对象(例如支气管树、血管、皮肤、心脏等)建立虚拟模型。In an aid to surgery, a virtual model can be created for a patient's target object (eg, bronchial tree, blood vessels, skin, heart, etc.).
然而，在真实的手术场景中，现有技术中的虚拟模型只能体现出目标对象在人体中的大体位置及目标对象的大体形状，信息的种类较少。However, in a real surgical scene, the virtual model in the prior art can only reflect the general position of the target object in the human body and the general shape of the target object, and there are few types of information.
发明内容SUMMARY OF THE INVENTION
本发明提供一种手术辅助的数据处理方法、装置、设备、介质与系统，以解决信息的种类较少的问题。The present invention provides a data processing method, device, equipment, medium and system for surgical assistance to solve the problem of less types of information.
根据本发明的第一方面，提供了一种手术辅助的数据处理方法，包括：According to a first aspect of the present invention, there is provided a data processing method for surgical assistance, comprising:
确定目标对象的三维动态模型，所述三维动态模型被用于表征所述目标对象随时间的动态变化；所述目标对象包括：目标人体内的器官和/或组织；determining a three-dimensional dynamic model of the target object, the three-dimensional dynamic model being used to characterize the dynamic changes of the target object over time; the target object includes: organs and/or tissues in the target human body;
获取所述目标对象对应的至少覆盖一个呼吸周期的呼吸监测信号；acquiring a respiratory monitoring signal corresponding to the target object covering at least one respiratory cycle;
基于所述呼吸监测信号以及所述三维动态模型，构建所述目标对象的呼吸运动模型，所述呼吸运动模型被用于预测所述目标对象随呼吸周期的动态变化。Based on the breathing monitoring signal and the three-dimensional dynamic model, a breathing motion model of the target object is constructed, and the breathing motion model is used to predict the dynamic change of the target object with the breathing cycle.
可选的，确定目标对象的三维动态模型，包括：Optionally, determine the three-dimensional dynamic model of the target object, including:
获取所述目标对象的CT数据，所述CT数据包括所述目标对象的多个时刻的CT图像；所述多个时刻的跨度覆盖了采集所述CT数据时所述目标对象的至少一个呼吸周期；Acquiring CT data of the target object, the CT data includes CT images of the target object at multiple times; the span of the multiple times covers at least one breathing cycle of the target object when the CT data is collected ;
基于所述CT数据，在虚拟坐标系中，构建出所述三维动态模型。Based on the CT data, in a virtual coordinate system, the three-dimensional dynamic model is constructed.
可选的，基于所述CT数据，在虚拟坐标系中，构建出所述动态模型，包括：Optionally, based on the CT data, in a virtual coordinate system, the dynamic model is constructed, including:
基于所述CT数据，构建每个时刻所述目标对象的静态模型；所述静态模型在所述虚拟坐标系中的位置被用于表征：对应时刻所述目标对象在所述目标人体内的位置，所述静态模型的形状被用于表征：对应时刻所述目标对象的形状；Based on the CT data, a static model of the target object is constructed at each moment; the position of the static model in the virtual coordinate system is used to represent: the position of the target object in the target body at the corresponding moment , the shape of the static model is used to characterize: the shape of the target object at the corresponding moment;
基于所述多个时刻的静态模型，构建所述三维动态模型。The three-dimensional dynamic model is constructed based on the static models at the multiple times.
可选的，基于所述多个时刻的静态模型，构建所述三维动态模型，包括：Optionally, building the three-dimensional dynamic model based on the static models at the multiple moments, including:
基于任意两个相邻时刻的静态模型，拟合出静态模型在所述任意两个相邻时刻间发生的变化，并基于拟合出的变化与所述静态模型，构建所述三维动态模型。Based on the static models at any two adjacent moments, the changes of the static model between the any two adjacent moments are fitted, and the three-dimensional dynamic model is constructed based on the fitted changes and the static model.
可选的，基于所述CT数据，在虚拟坐标系中，构建出所述三维动态模型，包括：Optionally, based on the CT data, in a virtual coordinate system, the three-dimensional dynamic model is constructed, including:
将所述CT数据输入经训练的生成模型，并获取所述生成模型生成的所述三维动态模型。The CT data is input into a trained generative model, and the three-dimensional dynamic model generated by the generative model is obtained.
可选的，所述的手术辅助的数据处理方法，还包括：基于获取到的所述目标对象的至少一个特定呼吸监测信号，控制所述呼吸运动模型调整至指定状态。Optionally, the surgically assisted data processing method further includes: controlling the breathing motion model to adjust to a specified state based on the acquired at least one specific breathing monitoring signal of the target object.
可选的，所述的手术辅助的数据处理方法，还包括：Optionally, the surgically assisted data processing method further includes:
在显示设备的显示视野中显示出虚拟对象，所述虚拟对象包括所述目标对象所属目标人体的模型组合，所述模型组合中的多个模型包括所述呼吸运动模型。A virtual object is displayed in the display field of view of the display device, and the virtual object includes a model combination of a target human body to which the target object belongs, and a plurality of models in the model combination includes the breathing motion model.
可选的，所述显示设备为混合现实头戴式显示设备；Optionally, the display device is a mixed reality head-mounted display device;
所述数据处理方法，还包括：The data processing method further includes:
在所述显示视野中显示出对应于所述目标人体的术中图像，所述虚拟对象叠加显示于所述术中图像上。An intraoperative image corresponding to the target human body is displayed in the display field of view, and the virtual object is superimposed and displayed on the intraoperative image.
可选的，所述模型组合在所述显示视野中的显示位置适配于所述显示设备与所述目标人体的相对位姿。Optionally, the display position of the model combination in the display field of view is adapted to the relative pose of the display device and the target human body.
可选的，所述虚拟对象还包括以下至少之一：Optionally, the virtual object further includes at least one of the following:
设于所述模型组合的通往病灶点的导航路径；A navigation path leading to the lesion point set on the model combination;
设于所述模型组合的匹配于病灶点的模拟穿刺点；A simulated puncture point matching the lesion point set in the model combination;
设于所述模型组合的虚拟器械，所述虚拟器械在所述模型组合的位置和 /或姿态适配于真实器械。A virtual instrument provided in the model assembly, the position and/or attitude of the virtual instrument in the model assembly is adapted to the real instrument.
根据本发明的第二方面，提供了一种手术辅助的数据处理系统，包括：According to a second aspect of the present invention, there is provided a surgically assisted data processing system, comprising:
处理服务器，用于执行第一方面及其可选方案涉及的数据处理方法并输出所述呼吸运动模型；a processing server, configured to execute the data processing method involved in the first aspect and its optional solutions and output the breathing motion model;
混合现实头戴式设备，用于显示对应于所述目标对象的术中图像，并将接收到的由所述处理服务器传输的所述呼吸运动模型叠加显示在所述术中图像上。The mixed reality head-mounted device is used for displaying an intraoperative image corresponding to the target object, and superimposing and displaying the received respiratory motion model transmitted by the processing server on the intraoperative image.
根据本发明的第三方面，提供了一种手术辅助的数据处理装置，包括：According to a third aspect of the present invention, there is provided a surgically assisted data processing device, comprising:
动态模型确定模块，用于确定目标对象的三维动态模型，所述三维动态模型被用于表征所述目标对象随时间的动态变化；所述目标对象包括：目标人体内的器官和/或组织；a dynamic model determination module, configured to determine a three-dimensional dynamic model of the target object, the three-dimensional dynamic model is used to characterize the dynamic changes of the target object over time; the target object includes: organs and/or tissues in the target human body;
呼吸监测模块，用于获取所述目标对象对应的至少覆盖一个呼吸周期的呼吸监测信号；A breathing monitoring module, used for acquiring a breathing monitoring signal corresponding to the target object covering at least one breathing cycle;
呼吸模型控制模块，用于基于所述呼吸监测信号以及所述三维动态模型，构建所述目标对象的呼吸运动模型，所述呼吸运动模型被用于预测所述目标对象随呼吸周期的动态变化。A breathing model control module, configured to construct a breathing motion model of the target object based on the breathing monitoring signal and the three-dimensional dynamic model, and the breathing motion model is used to predict the dynamic change of the target object with the breathing cycle.
根据本发明的第四方面，提供了一种电子设备，包括处理器与存储器，According to a fourth aspect of the present invention, there is provided an electronic device including a processor and a memory,
所述存储器，用于存储代码；the memory for storing code;
所述处理器，用于执行所述存储器中的代码用以实现第一方面及其可选方案涉及的数据处理方法。The processor is configured to execute the code in the memory to implement the data processing method involved in the first aspect and its optional solutions.
根据本发明的第五方面，提供了一种存储介质，其上存储有计算机程序，该程序被处理器执行时实现第一方面及其可选方案涉及的数据处理方法。According to a fifth aspect of the present invention, there is provided a storage medium on which a computer program is stored, and when the program is executed by a processor, implements the data processing method involved in the first aspect and its optional solutions.
根据本发明的第六方面，提供了一种手术系统，包括：数据处理部与呼吸监测设备，所述数据处理部用于执行第一方面及其可选方案涉及的数据处理方法，所述呼吸监测信号是所述呼吸监测设备监测到的。According to a sixth aspect of the present invention, a surgical system is provided, comprising: a data processing part and a respiratory monitoring device, wherein the data processing part is configured to execute the data processing method involved in the first aspect and its optional solutions, the respiratory The monitoring signal is detected by the respiration monitoring device.
本发明提供的手术辅助的数据处理方法、装置、设备、介质与系统中，在确定表征目标对象动态变化的三维动态模型之后，可基于目标对象的呼吸监测信号，构建目标对象的呼吸运动模型，进而，该呼吸运动模型可预测目标对象随呼吸周期的动态变化，相较于仅能描绘出目标对象的大体位置和大体形状的方案，本发明所提供的信息可以更细致描绘出目标对象随呼吸的动态变化，提供更直观准确的辅助依据，更多样的信息。In the surgical assistance data processing method, device, equipment, medium and system provided by the present invention, after the three-dimensional dynamic model representing the dynamic change of the target object is determined, the breathing motion model of the target object can be constructed based on the breathing monitoring signal of the target object, Furthermore, the breathing motion model can predict the dynamic changes of the target object with the breathing cycle. Compared with the solution that can only depict the general position and general shape of the target object, the information provided by the present invention can more accurately depict the target object with the breathing cycle. dynamic changes, provide more intuitive and accurate auxiliary basis, and more diverse information.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.
图1是本发明一示例性的实施例中手术系统的构造示意图；FIG. 1 is a schematic structural diagram of a surgical system in an exemplary embodiment of the present invention;
图2是本发明另一示例性的实施例中手术系统的构造示意图；FIG. 2 is a schematic structural diagram of a surgical system in another exemplary embodiment of the present invention;
图3是本发明一示例性的实施例中手术辅助的数据处理方法的流程示意图；3 is a schematic flowchart of a data processing method for surgical assistance in an exemplary embodiment of the present invention;
图4是本发明一示例性的实施例中构建三维动态模型的流程示意图；4 is a schematic flow chart of constructing a three-dimensional dynamic model in an exemplary embodiment of the present invention;
图5是本发明一示例性的实施例中呼吸运动模型的构建、控制的流程示意图；5 is a schematic flowchart of the construction and control of a breathing motion model in an exemplary embodiment of the present invention;
图6是本发明另一示例性的实施例中呼吸运动模型的构建、控制流程示意图；6 is a schematic diagram of the construction and control flow of a breathing motion model in another exemplary embodiment of the present invention;
图7是本发明另一示例性的实施例中中手术辅助的数据处理方法的流程示意图；7 is a schematic flowchart of a data processing method for surgical assistance in another exemplary embodiment of the present invention;
图8是本发明一示例性的实施例中手术辅助的数据处理装置的程序模块示意图；8 is a schematic diagram of a program module of a data processing apparatus for surgical assistance in an exemplary embodiment of the present invention;
图9是本发明另一示例性的实施例中手术辅助的数据处理装置的程序模块示意图；9 is a schematic diagram of a program module of a data processing apparatus for surgical assistance in another exemplary embodiment of the present invention;
图10是本发明一示例性的实施例中电子设备的构造示意图。FIG. 10 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本发明的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to Describe a particular order or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.
下面以具体地实施例对本发明的技术方案进行详细说明。下面这几个具体的实施例可以相互结合，对于相同或相似的概念或过程可能在某些实施例不再赘述。The technical solutions of the present invention will be described in detail below with specific examples. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
请参考图1，本发明实施例提供了一种手术系统，包括：数据处理部 101与呼吸监测设备102。Referring to FIG. 1 , an embodiment of the present invention provides a surgical system, including: a
其中的呼吸监测设备102，可理解为能够对目标人体的呼吸状况进行监测的任意设备，其可以为专用于监测呼吸状况的设备，也可以为监测其他特定信息，进而，可基于该特定信息推断出呼吸状况的设备，例如可以为用于监测胸部起伏状态的设备，通过所监测出的胸部起伏状态，可推断出呼吸状况。其中的呼吸状况可例如包括以下至少之一：开始吸气的时间、开始呼气的时间、吸气时长、呼气时长、呼吸周期、吸气量、呼气量等；此外，其中的呼吸周期可以指单次吸气的吸气时长与单次呼气的呼气时长之和。The
其中的数据处理部101，可以为具备数据处理能力的任意设备或设备的组合，数据处理部101可与呼吸监测设备102直接或间接通信。一种举例中，数据处理部101可以为服务器，也可以为终端、计算机等。此外，所述数据处理部101可用于执行本发明实施例提供的手术辅助的数据处理方法。The
图2所示实施例中的数据处理部201、呼吸监测设备202，相同或相似于图1所示实施例中的数据处理部101、呼吸监测设备102，对于相同或相似的内容，在此不再赘述。The
其中一种实施方式中，请参考图2，手术系统，可以包括：In one embodiment, please refer to FIG. 2, a surgical system may include:
CT扫描设备203，用于扫描目标人体(例如目标对象)的CT数据。其中的CT数据可理解为CT扫描而得到的任意数据，例如可以包括CT图像。通过CT数据的获取，可以为数据处理部201的进一步处理(例如虚拟对象的构建)提供依据。The
其中一种实施方式中，请参考图2，手术系统可以包括：显示设备 204，可理解为能够实现信息显示的任意设备，具体的，显示设备204可直接或间接与数据处理部201通信，接收数据处理部201发送而来的数据，从而实现显示。In one of the embodiments, please refer to FIG. 2, the surgical system may include: a
一种举例中，显示设备204可以为混合现实头戴式显示设备，其中的混合现实，也可理解为MR，即Mixed Reality，进而，混合现实头戴式显示设备可设有摄像部，通过摄像部可采集手术真实画面，显示设备204的显示视野中，可显示手术真实画面，在显示画面的基础上，还可叠加显示其他虚拟对象。In one example, the
其中一种实施方式中，请参考图2，手术系统还可以包括手术器械，例如穿刺针、支气管镜等，若采用支气管镜，其也可直接或间接与数据处理部 201通信，从而将支气管镜采集到的术中图像反馈至数据处理部201。In one of the embodiments, please refer to FIG. 2 , the surgical system may further include surgical instruments, such as puncture needles, bronchoscopes, etc. If a bronchoscope is used, it can also communicate with the
其中一种实施方式中，请参考图2，手术系统还可包括一套或多套定位追踪组件205。In one embodiment, please refer to FIG. 2 , the surgical system may further include one or more sets of positioning and tracking
一套定位追踪组件205可用于对显示设备(例如混合现实头戴式显示设备)进行追踪定位，从而确定显示设备(例如混合现实头戴式显示设备)的位姿。另一套定位追踪组件205可用于对患者(或患者的衣物、配饰等设于患者的部件)进行追踪定位，从而确定对应的位姿。再一套定位追踪组件 205可用于对医疗器械206(例如支气管镜、穿刺针)进行追踪定位，从而确定医疗器械的位姿。A set of positioning tracking
定位追踪组件205的一种举例中，可包括定位追踪设备与定位工具，定位工具可设于定位追踪对象(例如显示设备、患者、医疗器械)，定位追踪设备可产生用于实现定位追踪的媒介(例如磁场、红外线、超声波等)，进而，基于该媒介，定位追踪设备或定位工具可确定定位工具的位姿，并以此位姿作为定位追踪对象的位姿。一种具体的举例中，定位工具可例如为磁场传感器，定位追踪设备可例如为磁场发生器。In an example of the positioning and
追踪定位组件可以是基于光学、磁或者电磁原理的追踪定位组件，具体举例中，可将定位工具固定或粘贴在患者、相关手术器械、混合现实头戴式显示设备上，基于各定位追踪组件的坐标系之间的转换，可以使得各设备同时渲染显示在同一坐标系空间，便于混合现实头戴式显示设备的统一展示。The tracking and positioning component can be a tracking and positioning component based on optical, magnetic or electromagnetic principles. In a specific example, the positioning tool can be fixed or pasted on the patient, related surgical instruments, and mixed reality head-mounted display devices. The conversion between coordinate systems can make each device render and display in the same coordinate system space at the same time, which is convenient for the unified display of mixed reality head-mounted display devices.
此外，手术系统还可包括实现以上设备之间互相通信的通信设备。In addition, the surgical system may also include a communication device that enables the above devices to communicate with each other.
请参考图3，本发明实施例提供了一种手术辅助的数据处理方法，包括：Referring to FIG. 3 , an embodiment of the present invention provides a method for data processing for surgical assistance, including:
S301：确定目标对象的三维动态模型；S301: Determine the three-dimensional dynamic model of the target object;
所述目标对象包括：目标人体内的器官和/或组织，具体的可以指目标人体内会随着呼吸过程而发生位置、尺寸、形状中至少之一变化的器官、组织，一种举例中，目标对象可以包括以下至少之一：皮肤、支气管树、胸膜、肋骨、心脏、血管等，此外，目标对象可以是器官、组织的完整的整体，也可以是器官、组织的一部分；The target object includes: organs and/or tissues in the target human body, and may specifically refer to the organs and tissues in the target human body whose position, size, and shape change with the respiration process. In one example, The target object may include at least one of the following: skin, bronchial tree, pleura, ribs, heart, blood vessels, etc. In addition, the target object may be the complete whole of an organ or tissue, or a part of an organ or tissue;
所述三维动态模型被用于表征目标对象随时间的动态变化，具体的，三维动态模型的位置、尺寸、形状中至少之一能够发生指定动态变化；该三维动态模型可视作一种可动态变化的虚拟三维模型，且指定动态变化可在一个指定周期内完成，通过多个指定周期多次重复执行指定动态变化，可实现三维动态模型持续的动态变化；此外，指定周期的长度可以是变化的，对应的即意味着：指定动态变化的速度可以是变化的；The three-dimensional dynamic model is used to characterize the dynamic changes of the target object over time. Specifically, at least one of the position, size, and shape of the three-dimensional dynamic model can undergo specified dynamic changes; the three-dimensional dynamic model can be regarded as a dynamic The virtual 3D model that changes, and the specified dynamic change can be completed in a specified cycle, and the specified dynamic change can be repeatedly performed through multiple specified cycles to achieve continuous dynamic changes of the 3D dynamic model; in addition, the length of the specified cycle can be changed. , the corresponding means: the speed of the specified dynamic change can be changed;
本领域任意可构建、控制动态变化的虚拟三维模型的手段，均可应用于本发明实施例而作为一种可选方案；Any means in the art that can construct and control a dynamically changing virtual three-dimensional model can be applied to the embodiments of the present invention as an optional solution;
S302：获取所述目标对象对应的至少覆盖一个呼吸周期的呼吸监测信号；S302: Acquire a respiratory monitoring signal corresponding to the target object covering at least one respiratory cycle;
该呼吸监测信号可以是呼吸监测设备监测到的，其定义和具体内容可参照图1所示实施例中的相关描述理解；The respiration monitoring signal may be monitored by a respiration monitoring device, and its definition and specific content can be understood with reference to the relevant description in the embodiment shown in FIG. 1 ;
S303：基于所述呼吸监测信号，构建所述目标对象的呼吸运动模型。S303: Build a breathing motion model of the target object based on the breathing monitoring signal.
其中的呼吸运动模型，可理解为：其形状、位置、尺寸等至少之一可随目标人体的呼吸而动态变化；The breathing motion model can be understood as: at least one of its shape, position, size, etc. can change dynamically with the breathing of the target human body;
呼吸运动模型与三维动态模型的关系可理解为：三维动态模型表征了目标对象随时间的动态变化，但该动态变化并未与目标人体的呼吸相关联，呼吸运动模型可理解为：通过将三维动态模型的动态变化过程的时间线与呼吸过程建立关联性，可以使三维动态模型的动态变化的过程与目标人体的呼吸相关联，具备这种关联性的三维动态模型即为呼吸运动模型；The relationship between the breathing motion model and the three-dimensional dynamic model can be understood as: the three-dimensional dynamic model represents the dynamic change of the target object over time, but the dynamic change is not related to the breathing of the target human body. The breathing motion model can be understood as: The timeline of the dynamic change process of the dynamic model is related to the breathing process, so that the dynamic change process of the three-dimensional dynamic model can be related to the breathing of the target human body, and the three-dimensional dynamic model with this correlation is the respiratory motion model;
可例如通过对以下至少之一内容的标定而建立三维动态模型的动态变化过程的时间线与呼吸过程的关联性：The correlation between the time line of the dynamic changing process of the three-dimensional dynamic model and the breathing process can be established, for example, by calibrating at least one of the following contents:
何时开始指定动态变化；when to start specifying dynamic changes;
指定动态变化的速度；Specify the speed of dynamic change;
何时结束指定动态变化；When to end the specified dynamic change;
是否重复指定动态变化；Whether to specify dynamic changes repeatedly;
何时开始指定动态变化中呼气阶段的变化；When to start specifying changes in the expiratory phase in dynamic changes;
何时结束指定动态变化中呼气阶段的变化；When to end the change in the expiratory phase in the specified dynamics;
呼气阶段的变化的速度；The rate of change in the exhalation phase;
何时开始吸气阶段的变化；Changes in when to start the inspiratory phase;
何时结束吸气阶段的变化；Changes in when to end the inspiratory phase;
吸气阶段的变化的速度。The rate of change in the inspiratory phase.
只要能令时间线与呼吸相关联，都不脱离本发明实施例的范围。As long as the timeline can be associated with breathing, it does not depart from the scope of the embodiments of the present invention.
以上方案中，在确定表征目标对象动态变化的三维动态模型之后，可基于目标对象的呼吸监测信号，构建目标对象的呼吸运动模型，进而，该呼吸运动模型可预测目标对象随呼吸周期的动态变化，相较于仅能描绘出目标对象的大体位置和大体形状的方案，本发明所提供的信息可以更细致描绘出目标对象随呼吸的动态变化，提供更直观准确的辅助依据，更多样的信息。In the above scheme, after determining the three-dimensional dynamic model representing the dynamic change of the target object, a breathing motion model of the target object can be constructed based on the breathing monitoring signal of the target object, and further, the breathing motion model can predict the dynamic change of the target object with the breathing cycle , compared with the scheme that can only describe the general position and general shape of the target object, the information provided by the present invention can describe the dynamic changes of the target object with breathing in more detail, provide more intuitive and accurate auxiliary basis, and more diverse information.
此外，以TTNA手术为例，呼吸运动对胸部相关器官与组织会进行不同方向的缩放或平移，对TTNA手术造成很大误差，使用本发明实施例的方案后，可准确细致地反应出器官或组织的变化，降低TTNA手术的误差。In addition, taking TTNA surgery as an example, the breathing movement will zoom or translate in different directions on the chest-related organs and tissues, which will cause great errors in TTNA surgery. After using the solution of the embodiment of the present invention, the organs and tissues can be accurately and meticulously reflected. Tissue changes reduce errors in TTNA surgery.
其中一种实施方式中，请参考图4，确定目标对象的三维动态模型的过程，可以包括：In one embodiment, please refer to FIG. 4 , the process of determining the three-dimensional dynamic model of the target object may include:
S401：获取所述目标对象的CT数据；S401: Obtain CT data of the target object;
S402：基于所述CT数据，在虚拟坐标系中，构建出所述三维动态模型。S402: Based on the CT data, in a virtual coordinate system, construct the three-dimensional dynamic model.
所述CT数据包括所述目标对象的多个时刻的CT图像；所述多个时刻的跨度覆盖了采集所述CT数据时所述目标对象的至少一个呼吸周期；进而，通过CT数据，可体现出一个呼吸周期内CT图像的变化，而一个周期内CT图像的变化，可体现出一个呼吸周期内目标对象位置、尺寸、形状中至少之一的变化；一种举例中，CT数据可以是CT扫描设备扫描并上传至数据处理部(例如服务器)。The CT data includes CT images of the target object at multiple times; the span of the multiple times covers at least one breathing cycle of the target object when the CT data is collected; further, through the CT data, it can be reflected The change of CT image in one breathing cycle can reflect the change of at least one of the position, size and shape of the target object in one breathing cycle; in an example, the CT data can be CT data. The scanning device scans and uploads to the data processing unit (eg server).
一种实施方式中，步骤S402可以包括：将所述CT数据输入经训练的第一生成模型，并获取所述第一生成模型生成的所述三维动态模型。In one embodiment, step S402 may include: inputting the CT data into a trained first generative model, and acquiring the three-dimensional dynamic model generated by the first generative model.
在训练第一生成模型时，可取日常医疗或实验过程中所采集到的CT数据作为训练用CT数据，同时，可人为(或采用其他建模算法)主动构建出训练用动态模型，然后，可利用所述训练用CT数据与训练用动态模型对第一生成模型进行训练，例如，将训练用CT数据输入第一生成模型后，第一生成模型可输出预测动态模型，然后，通过训练用动态模型与预测动态模型的差异，调整第一生成模型，训练后，第一生成模型能够针对输入的CT数据生成对应的三维动态模型。When training the first generated model, the CT data collected during daily medical treatment or experiments can be taken as the CT data for training. At the same time, the dynamic model for training can be constructed manually (or by using other modeling algorithms). The first generative model is trained by using the CT data for training and the dynamic model for training. For example, after the CT data for training is input into the first generative model, the first generative model can output a predictive dynamic model, and then the dynamic model for training can be For the difference between the model and the predicted dynamic model, the first generation model is adjusted. After training, the first generation model can generate a corresponding three-dimensional dynamic model for the input CT data.
另一实施方式中，步骤S402可以包括：基于所述CT数据，构建每个时刻所述目标对象的静态模型；并基于所述多个时刻的静态模型，构建所述三维动态模型。In another embodiment, step S402 may include: constructing a static model of the target object at each moment based on the CT data; and constructing the three-dimensional dynamic model based on the static models at multiple moments.
所述静态模型在所述虚拟坐标系中的位置被用于表征：对应时刻所述目标对象在所述目标人体内的位置，所述静态模型的形状被用于表征：对应时刻所述目标对象的形状；该静态模型可以为本领域现有相关技术中基于CT 数据而构建的模型。The position of the static model in the virtual coordinate system is used to characterize: the position of the target object in the target body at the corresponding moment, and the shape of the static model is used to characterize: the target object at the corresponding moment The static model can be a model constructed based on CT data in the related art in the field.
所述多个时刻的静态模型可体现出目标对象的位置、形状随时间的变化，所取的静态模型足够多的情况下，可充分体现出这种变化的发展趋势，进而，可以为动态模型的构建提供充分的依据。The static models at multiple moments can reflect the changes in the position and shape of the target object over time, and when enough static models are taken, the development trend of such changes can be fully reflected, and further, a dynamic model can be used. provide sufficient basis for the construction.
本领域任意基于静态模型而构建出动态模型的方案，均可应用于此而作为一种可选方案。Any solution in the art for constructing a dynamic model based on a static model can be applied to this as an optional solution.
一种举例中，基于所述多个时刻的静态模型，构建所述动态模型的过程可例如：可将所述多个时刻的静态模型输入经训练的第二生成模型，并获取所述第二生成模型生成的所述动态模型。In an example, based on the static models at the multiple times, the process of constructing the dynamic model may, for example, input the static models at the multiple times into the trained second generative model, and obtain the second generation model. The dynamic model generated by the generative model is generated.
在训练该第二生成模型时，可先取人为构建的训练用动态模型(或以其他方式构建的训练用动态模型)作为原始素材，再从中截取出多个时刻的训练用静态模型，然后利用训练用动态模型与训练用静态模型，训练该第二生成模型，从而使其具备针对输入的静态模型生成动态模型的能力。When training the second generative model, the artificially constructed dynamic model for training (or a dynamic model for training constructed in other ways) can be taken as the original material, and then the static models for training at multiple moments are extracted from it, and then the training The second generative model is trained by using the dynamic model and the static model for training, so that it has the ability to generate a dynamic model for the input static model.
另一举例中，基于所述多个时刻的静态模型，构建所述三维动态模型的过程可例如：基于任意两个相邻时刻的静态模型，拟合出静态模型在所述任意两个相邻时刻间发生的变化，并基于拟合出的变化与所述静态模型，构建所述三维动态模型。In another example, the process of constructing the three-dimensional dynamic model based on the static models at the multiple moments may be, for example, based on the static models at any two adjacent moments, fitting a static model at any two adjacent moments. The changes occurring from time to time, and based on the fitted changes and the static model, the three-dimensional dynamic model is constructed.
以上方案中，通过以CT数据而构建动态模型，可保障动态模型可准确反应出目标对象的真实形状和/或位置。In the above solution, by constructing a dynamic model based on CT data, it can be ensured that the dynamic model can accurately reflect the real shape and/or position of the target object.
其中一种实施方式中，请参考图5，基于所述呼吸监测信号以及所述三维动态模型，构建所述目标对象的呼吸运动模型的过程，可以包括：In one embodiment, please refer to FIG. 5 , based on the breathing monitoring signal and the three-dimensional dynamic model, the process of constructing the breathing motion model of the target object may include:
S501：为所述呼吸监测信号中的每种特定呼吸监测信号，匹配所述三维动态模型对应的一个指定状态，得到基于所述特定呼吸监测信号而动态变化的呼吸运动模型；S501: For each specific respiratory monitoring signal in the respiratory monitoring signal, match a specified state corresponding to the three-dimensional dynamic model to obtain a dynamic breathing motion model based on the specific respiratory monitoring signal;
与之相对应的，手术辅助的数据处理方法，还可包括：Correspondingly, the data processing method for surgical assistance may further include:
S502：基于获取到的所述目标对象的至少一个特定呼吸监测信号，控制所述呼吸运动模型调整至指定状态。S502: Control the breathing motion model to adjust to a specified state based on the acquired at least one specific breathing monitoring signal of the target object.
其中的特定呼吸监测信号，可以为基于呼吸周期而定义的任意信号，例如可以为呼吸周期中的呼气开始时刻的呼吸监测信号、吸气开始时刻的呼吸监测信号、呼气结束时刻的呼吸监测信号、吸气结束时刻的呼吸监测信号等。The specific breathing monitoring signal may be any signal defined based on the breathing cycle, for example, the breathing monitoring signal at the beginning of exhalation, the monitoring signal at the beginning of inspiration, and the monitoring signal at the end of expiration in the breathing cycle Signal, respiratory monitoring signal at the end of inspiration, etc.
其中的指定状态，可以指表征了目标对象的一种位置、尺寸、形状；The specified state can refer to a position, size, and shape that characterizes the target object;
一种举例中，指定状态可以包括第一指定状态，该第一指定状态可以表征出：呼气开始时刻目标对象的位置、尺寸、形状中至少之一，进而，可基于步骤S501的处理结果，在步骤S502中，在监测到呼气开始时刻的特定呼吸监测信号时，呼吸运动模型会进入第一指定状态；然后，呼吸运动模型可在进入第一指定状态之后，自第一指定状态开始继续后续的动态变化，直至监测到下一个特定呼吸监测信号；In an example, the specified state may include a first specified state, and the first specified state may represent: at least one of the position, size, and shape of the target object at the start of exhalation, and further, based on the processing result of step S501, In step S502, when the specific breathing monitoring signal at the expiratory start time is monitored, the breathing motion model will enter the first specified state; then, the breathing motion model may continue from the first specified state after entering the first specified state Subsequent dynamic changes until the next specific respiratory monitoring signal is monitored;
另一举例中，指定状态可以包括第二指定状态，该第二指定状态可以表征出：呼气结束时刻目标对象的位置、尺寸、形状中至少之一，进而，可基于步骤S501的处理结果，在步骤S502中，在监测到呼气结束时刻的特定呼吸监测信号时，呼吸运动模型会进入第二指定状态；然后，呼吸运动模型可在进入第二指定状态之后，自第二指定状态开始继续后续的动态变化，直至监测到下一个特定呼吸监测信号；In another example, the specified state may include a second specified state, and the second specified state may represent: at least one of the position, size, and shape of the target object at the end of exhalation, and further, based on the processing result of step S501, In step S502, when the specific breathing monitoring signal at the end of expiration is monitored, the breathing motion model will enter the second specified state; then, the breathing motion model may continue from the second specified state after entering the second specified state Subsequent dynamic changes until the next specific respiratory monitoring signal is monitored;
再一举例中，指定状态可以包括第三指定状态，该第三指定状态可以表征出：吸气开始时刻目标对象的位置、尺寸、形状中至少之一，进而，可基于步骤S501的处理结果，在步骤S502中，在监测到吸气开始时刻的特定呼吸监测信号时，呼吸运动模型会进入第三指定状态；然后，呼吸运动模型可在进入第三指定状态之后，自第三指定状态开始继续后续的动态变化，直至监测到下一个特定呼吸监测信号；In another example, the specified state may include a third specified state, and the third specified state may represent: at least one of the position, size, and shape of the target object at the start of inhalation, and further, based on the processing result of step S501, In step S502, when the specific breathing monitoring signal at the inhalation start time is monitored, the breathing motion model will enter a third specified state; then, the breathing motion model may continue from the third specified state after entering the third specified state Subsequent dynamic changes until the next specific respiratory monitoring signal is monitored;
又一举例中，指定状态可以包括第四指定状态，该第四指定状态可以表征出：吸气结束时刻目标对象的位置、尺寸、形状中至少之一，进而，可基于步骤S501的处理结果，在步骤S502中，在监测到吸气结束时刻的特定呼吸监测信号时，呼吸运动模型会进入第三指定状态；然后，呼吸运动模型可在进入第四指定状态之后，自第四指定状态开始继续后续的动态变化，直至监测到下一个特定呼吸监测信号。In another example, the designated state may include a fourth designated state, and the fourth designated state may represent: at least one of the position, size, and shape of the target object at the end of inhalation, and further, based on the processing result of step S501, In step S502, when the specific breathing monitoring signal at the end of inspiration is monitored, the breathing motion model will enter a third specified state; then, the breathing motion model may continue from the fourth specified state after entering the fourth specified state Subsequent dynamic changes until the next specific respiratory monitoring signal is monitored.
此外，特定呼吸监测信号也可不限于以上的举例，对应的，指定状态也可不限于以上举例。In addition, the specific respiration monitoring signal may not be limited to the above examples, and correspondingly, the specified state may not be limited to the above examples.
以上方案中，通过对指定状态的触发，可将呼吸运动模型的动态变化的时间线与呼吸的时间线匹配起来，实现一定程度上的时间对齐。In the above solution, by triggering the specified state, the time line of the dynamic change of the breathing motion model can be matched with the time line of the breathing, so as to achieve a certain degree of time alignment.
其中一种实施方式中，请参考图6，基于所述呼吸监测信号以及所述三维动态模型，构建所述目标对象的呼吸运动模型，还包括：In one embodiment, please refer to FIG. 6 , based on the breathing monitoring signal and the three-dimensional dynamic model, constructing a breathing motion model of the target object, further comprising:
S601：基于所述呼吸监测信号，确定所述目标人体的呼吸周期的时间长度与呼吸运动模型的动态变化速度的关联关系；S601: Based on the respiration monitoring signal, determine the correlation between the time length of the respiration cycle of the target human body and the dynamic change speed of the respiration motion model;
该关联关系可例如表征为时间长度的不同时间长度范围与不同动态变化速度的对应关系，再例如时间长度与动态变化速度的函数关系(例如线性变化的函数关系)；The association relationship can be represented, for example, as a corresponding relationship between different time length ranges of time lengths and different dynamic change speeds, and for example, a functional relationship between time lengths and dynamic change speeds (for example, a linear change function relationship);
对应的，手术辅助的数据处理方法，还包括：Correspondingly, the data processing method for surgical assistance further includes:
S602：基于所述目标人体的呼吸周期的当前长度，以及所述关联关系，确定所述呼吸运动模型的运动速度。S602: Determine the movement speed of the breathing movement model based on the current length of the breathing cycle of the target human body and the association relationship.
其中，当目标人体的呼吸周期较长时，表示目标人体呼吸较慢，此时，以上方案中，呼吸运动模型的运动速度也会随之变慢，反之，目标人体呼吸变快时，呼吸运动模型的运动速度也会变快。Among them, when the breathing cycle of the target human body is long, it means that the target human body breathes slowly. At this time, in the above scheme, the movement speed of the breathing motion model will also slow down. On the contrary, when the target human body breathes faster, the breathing movement The model will also move faster.
其中，呼吸周期的当前长度，可理解为表征了一次呼气的时长与一次吸气的时长之和，一种举例中，可以采用最近一次完整的呼吸周期的时长作为当前长度，另一举例中，可以取最近多次完整呼吸周期的时长的统计值(例如平均值)作为当前长度。The current length of the breathing cycle can be understood as representing the sum of the duration of one exhalation and the duration of one inhalation. In one example, the duration of the last complete breathing cycle can be used as the current length, and in another example , the statistical value (eg, the average value) of the durations of the most recent complete breathing cycles can be taken as the current length.
通过以上方案，可实现呼吸速度与呼吸运动模型的变化速度之间的匹配，保障了呼吸运动模型的变化速度能适配与呼吸的过程，准确反映出一个呼吸周期下目标对象的位置、尺寸、形状中至少之一变化。Through the above scheme, the matching between the breathing speed and the changing speed of the breathing motion model can be realized, which ensures that the changing speed of the breathing motion model can be adapted to the breathing process, and accurately reflects the position, size, At least one of the shapes changes.
图8所示的实施例中，步骤S701、S702与S703相同或相似于图3所示实施例中的步骤S301、S302与S303，对于相同或相似的内容，在此不再赘述。In the embodiment shown in FIG. 8 , steps S701 , S702 and S703 are the same or similar to steps S301 , S302 and S303 in the embodiment shown in FIG. 3 , and the same or similar content will not be repeated here.
所述的手术辅助的数据处理方法，还包括：The data processing method for surgical assistance further includes:
S704：在显示设备的显示视野中显示出虚拟对象；S704: Display the virtual object in the display field of view of the display device;
所述虚拟对象包括所述目标人体的模型组合，所述模型组合中的多个模型包括所述呼吸运动模型，其为动态模型；除了动态模型，也可包括其他非动态的模型。The virtual object includes a model combination of the target human body, and multiple models in the model combination include the breathing motion model, which is a dynamic model; in addition to the dynamic model, other non-dynamic models may also be included.
进一步举例中，所述虚拟对象还包括以下至少之一：In a further example, the virtual object further includes at least one of the following:
设于所述模型组合的通往病灶点的导航路径；A navigation path leading to the lesion point set on the model combination;
设于所述模型组合的匹配于病灶点的模拟穿刺点；A simulated puncture point matching the lesion point set in the model combination;
设于所述模型组合的虚拟器械，所述虚拟器械在所述模型组合的位置和 /或姿态适配于真实器械。A virtual instrument provided in the model assembly, the position and/or attitude of the virtual instrument in the model assembly is adapted to the real instrument.
通过虚拟对象的显示，可以为手术过程提供直观可靠的辅助信息。Through the display of virtual objects, intuitive and reliable auxiliary information can be provided for the surgical procedure.
在规划导航路径、模拟穿刺点灯虚拟对象时，可以自动或者交互式选取模型组合中的病变区域，自动计算皮肤入针点或交互式选择入针点作为模拟穿刺点，据此规划穿刺路径作为所述导航路径，避开血管等组织；When planning the navigation path and simulating the puncture point of the virtual object, you can automatically or interactively select the lesion area in the model combination, automatically calculate the skin needle entry point or interactively select the needle entry point as the simulated puncture point, and plan the puncture path as the Describe the navigation path and avoid tissues such as blood vessels;
若所述显示设备为混合现实头戴式显示设备，混合现实头戴式显示设备可穿戴于医疗人员(例如执行手术的医生)；If the display device is a mixed reality head-mounted display device, the mixed reality head-mounted display device can be worn by medical personnel (eg, a doctor performing surgery);
所述数据处理方法，还包括：The data processing method further includes:
S705：在所述显示视野中显示出对应于所述目标人体的术中图像；S705: Display an intraoperative image corresponding to the target human body in the display field of view;
进而，所述虚拟对象叠加显示于所述术中图像上，从而实现混合现实的显示方式。Furthermore, the virtual object is superimposed and displayed on the intraoperative image, thereby realizing a mixed reality display mode.
其中一种实施方式中，所述模型组合在所述显示视野中的显示位置适配于所述显示设备(即混合现实头戴式显示设备)与患者的相对位姿。In one embodiment, the display position of the model combination in the display field of view is adapted to the relative pose of the display device (ie, the mixed reality head-mounted display device) and the patient.
例如，数据处理方法可以包括：For example, data processing methods may include:
通过针对显示设备的追踪定位组件获取显示设备的第一位姿，通过针对患者的追踪定位组件获取患者的第二位姿，然后，基于此计算出显示设备与患者的相对位姿。The first pose of the display device is obtained through the tracking and positioning component for the display device, the second pose of the patient is obtained through the tracking and positioning component for the patient, and then the relative pose between the display device and the patient is calculated based on this.
其中显示位置与相对位姿的适配，可理解为：避免模型组合的显示位置阻挡于患者与医疗人员(穿戴混合现实头戴式显示设备的医疗人员)的眼睛之间。The adaptation of the display position and the relative pose can be understood as: preventing the display position of the model combination from being blocked between the eyes of the patient and the medical personnel (medical personnel wearing the mixed reality head-mounted display device).
在具体举例中，医生佩戴混合现实头戴式显示设备时，既可以看到实际患者的手术区域图像(即手术真实画面，亦即术中图像)，又可以看到叠加在术中图像的模型组合、导航路径等，该模型组合中的呼吸运动模型已添加对应的动态变化(例如可以为仿真运动轨迹动画)，医生可以更直观地进行手术，可参考已经规划好地路径进行穿刺手术，也可以自主选择更合适地进针点与穿刺路径。在手术过程中，混合现实头戴式显示设备会实时监测数据处理部(例如服务器)的数据变更，并同步进行显示画面的渲染与更新，即在数据处理部(例如服务器)的控制下实现以上所提及内容的显示。In a specific example, when a doctor wears a mixed reality head-mounted display device, he can not only see the image of the actual patient's surgical area (ie, the real picture of the operation, that is, the intraoperative image), but also the model superimposed on the intraoperative image. combination, navigation path, etc. Corresponding dynamic changes have been added to the breathing motion model in this model combination (for example, it can be an animation of a simulated motion trajectory), so that doctors can perform surgery more intuitively, and can refer to the planned path for puncture surgery. You can choose a more suitable needle entry point and puncture path independently. During the operation, the mixed reality head-mounted display device will monitor the data changes of the data processing unit (such as the server) in real time, and synchronously render and update the display screen, that is, realize the above under the control of the data processing unit (such as the server). Display of the mentioned content.
可见，采用混合现实头戴式显示设备的情况下，可展示由摄像机采集到的现场手术区域图像(即术中图像)；也可叠加展示数据处理部(例如服务器)输出的对应于现场手术区域图像的动态的虚拟对象；It can be seen that in the case of using the mixed reality head-mounted display device, the image of the field operation area (ie, the intraoperative image) collected by the camera can be displayed; it can also be superimposed and displayed corresponding to the field operation area output by the data processing unit (for example, the server). Dynamic virtual objects of images;
其中为了将虚拟对象与是手术中图像更好地融合渲染，可借助追踪定位组件，实时追踪固定或粘贴在患者和混合现实头戴式显示设备上的追踪定位工具，根据所计算的坐标系转换关系，实时计算患者与混合现实头戴式显示设备的相对位姿，根据相对位姿，将模型组合(例如胸部相关器官与组织的动态模型、非动态模型)渲染到特定的视野位置。Among them, in order to better integrate and render the virtual object and the intraoperative image, the tracking and positioning component can be used to track the tracking and positioning tool fixed or pasted on the patient and the mixed reality head-mounted display device in real time, and transform according to the calculated coordinate system. The relationship between the patient and the mixed reality head-mounted display device is calculated in real time, and the model combination (such as the dynamic model and non-dynamic model of the chest-related organs and tissues) is rendered to a specific visual field position according to the relative pose.
请参考图8，手术辅助的数据处理装置800，包括：Please refer to FIG. 8 , a surgical-assisted
动态模型确定模块801，用于确定目标对象的三维动态模型，所述三维动态模型被用于表征所述目标对象随时间的动态变化；所述目标对象包括：目标人体内的器官和/或组织；A dynamic
呼吸监测模块802，用于获取所述目标对象对应的至少覆盖一个呼吸周期的呼吸监测信号；A
呼吸模型构建模块803，用于基于所述呼吸监测信号以及所述三维动态模型，构建所述目标对象的呼吸运动模型，所述呼吸运动模型被用于预测所述目标对象随呼吸周期的动态变化。A breathing
可选的，动态模型确定模块801，具体用于：Optionally, the dynamic
获取所述目标对象的CT数据，所述CT数据包括所述目标对象的多个时刻的CT图像；所述多个时刻的跨度覆盖了采集所述CT数据时所述目标对象的至少一个呼吸周期；Acquiring CT data of the target object, the CT data includes CT images of the target object at multiple times; the span of the multiple times covers at least one breathing cycle of the target object when the CT data is collected ;
基于所述CT数据，在虚拟坐标系中，构建出所述三维动态模型。Based on the CT data, in a virtual coordinate system, the three-dimensional dynamic model is constructed.
可选的，动态模型确定模块801，具体用于：Optionally, the dynamic
基于所述CT数据，构建每个时刻所述目标对象的静态模型；所述静态模型在所述虚拟坐标系中的位置被用于表征：对应时刻所述目标对象在所述目标人体内的位置，所述静态模型的形状被用于表征：对应时刻所述目标对象的形状；Based on the CT data, a static model of the target object is constructed at each moment; the position of the static model in the virtual coordinate system is used to represent: the position of the target object in the target body at the corresponding moment , the shape of the static model is used to characterize: the shape of the target object at the corresponding moment;
基于所述多个时刻的静态模型，构建所述三维动态模型。The three-dimensional dynamic model is constructed based on the static models at the multiple times.
可选的，动态模型确定模块801，具体用于：Optionally, the dynamic
基于任意两个相邻时刻的静态模型，拟合出静态模型在所述任意两个相邻时刻间发生的变化，并基于拟合出的变化与所述静态模型，构建所述三维动态模型。Based on the static models at any two adjacent moments, the changes of the static model between the any two adjacent moments are fitted, and the three-dimensional dynamic model is constructed based on the fitted changes and the static model.
可选的，动态模型确定模块801，具体用于：Optionally, the dynamic
将所述CT数据输入经训练的生成模型，并获取所述生成模型生成的所述三维动态模型。The CT data is input into a trained generative model, and the three-dimensional dynamic model generated by the generative model is obtained.
图9所示的实施例中，动态模型确定模块901、呼吸监测模块902、呼吸模型构建模块903相同或相似于图8所示的动态模型确定模块801、呼吸监测模块802、呼吸模型构建模块803，对于相同或相似的内容，在此不再赘述。In the embodiment shown in FIG. 9 , the dynamic
所述的手术辅助的数据处理装置900，还包括：The surgically assisted
控制模块904，用于基于获取到的所述目标对象的至少一个特定呼吸监测信号，控制所述呼吸运动模型调整至指定状态。The
可选的，所述的手术辅助的数据处理装置900，还包括：Optionally, the surgically assisted
显示模块905，用于在显示设备的显示视野中显示出虚拟对象，所述虚拟对象包括所述目标人体的模型组合，所述模型组合中的多个模型包括受所述呼吸运动模型。The
可选的，所述显示设备为混合现实头戴式显示设备；Optionally, the display device is a mixed reality head-mounted display device;
所述显示模块905还用于：The
在所述显示视野中显示出对应于所述目标人体的术中图像，所述虚拟对象叠加显示于所述术中图像上。An intraoperative image corresponding to the target human body is displayed in the display field of view, and the virtual object is superimposed and displayed on the intraoperative image.
可选的，所述模型组合在所述显示视野中的显示位置适配于所述显示设备与所述目标人体的相对位姿。Optionally, the display position of the model combination in the display field of view is adapted to the relative pose of the display device and the target human body.
可选的，所述虚拟对象还包括以下至少之一：Optionally, the virtual object further includes at least one of the following:
设于所述模型组合的通往病灶点的导航路径；A navigation path leading to the lesion point set on the model combination;
设于所述模型组合的匹配于病灶点的模拟穿刺点；A simulated puncture point matching the lesion point set in the model combination;
设于所述模型组合的虚拟器械，所述虚拟器械在所述模型组合的位置和 /或姿态适配于真实器械。A virtual instrument provided in the model assembly, the position and/or attitude of the virtual instrument in the model assembly is adapted to the real instrument.
请参考图10，提供了一种电子设备1000，包括：Referring to FIG. 10, an
处理器1001；以及，
存储器1002，用于存储所述处理器的可执行指令；a
其中，所述处理器1001配置为经由执行所述可执行指令来执行以上所涉及的方法。Wherein, the
处理器1001能够通过总线1003与存储器1002通讯。The
本发明实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现以上所涉及的方法。Embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method is implemented.
本领域普通技术人员可以理解：实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by program instructions related to hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps including the above method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.
最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.
Claims (15)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111630612.5ACN114565741A (en) | 2021-12-28 | 2021-12-28 | Operation-assisted data processing method, device, equipment, medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111630612.5ACN114565741A (en) | 2021-12-28 | 2021-12-28 | Operation-assisted data processing method, device, equipment, medium and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114565741Atrue CN114565741A (en) | 2022-05-31 |
Family
ID=81711992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111630612.5APendingCN114565741A (en) | 2021-12-28 | 2021-12-28 | Operation-assisted data processing method, device, equipment, medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114565741A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115105205A (en)* | 2022-06-08 | 2022-09-27 | 复旦大学附属中山医院 | In vitro pulmonary nodule localization system and method under electromagnetic navigation under general anesthesia |
| CN115239768A (en)* | 2022-06-06 | 2022-10-25 | 中日友好医院(中日友好临床医学研究所) | Image processing method and device, electronic device and computer storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090156951A1 (en)* | 2007-07-09 | 2009-06-18 | Superdimension, Ltd. | Patient breathing modeling |
| CN104055520A (en)* | 2014-06-11 | 2014-09-24 | 清华大学 | Human organ motion monitoring method and human body navigation system |
| CN107182200A (en)* | 2015-12-24 | 2017-09-19 | 中国科学院深圳先进技术研究院 | Minimally Invasive Surgery Navigation System |
| CN109389669A (en)* | 2017-08-04 | 2019-02-26 | 阿里健康信息技术有限公司 | Human 3d model construction method and system in virtual environment |
| CN111009024A (en)* | 2019-12-09 | 2020-04-14 | 咪咕视讯科技有限公司 | Method for generating dynamic image, electronic equipment and storage medium |
| CN111161333A (en)* | 2019-12-12 | 2020-05-15 | 中国科学院深圳先进技术研究院 | Prediction method, device and storage medium for liver respiration motion model |
| CN111386076A (en)* | 2017-12-11 | 2020-07-07 | 柯惠有限合伙公司 | System, method and computer readable medium for estimating chest movement during respiration |
| CN111465364A (en)* | 2017-12-15 | 2020-07-28 | 美敦力公司 | Augmented Reality Solutions for Interrupting, Converting and Augmenting Cardiovascular Surgery and/or Procedure Mapping Navigation and Procedure Diagnosis |
| CN113100935A (en)* | 2021-04-13 | 2021-07-13 | 上海大学 | A preoperative puncture path planning method and training system for lung puncture surgery |
| CN113674393A (en)* | 2021-07-12 | 2021-11-19 | 中国科学院深圳先进技术研究院 | Construction method of respiratory motion model and unmarked respiratory motion prediction method |
- 2021
- 2021-12-28CNCN202111630612.5Apatent/CN114565741A/enactivePending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090156951A1 (en)* | 2007-07-09 | 2009-06-18 | Superdimension, Ltd. | Patient breathing modeling |
| CN104055520A (en)* | 2014-06-11 | 2014-09-24 | 清华大学 | Human organ motion monitoring method and human body navigation system |
| CN107182200A (en)* | 2015-12-24 | 2017-09-19 | 中国科学院深圳先进技术研究院 | Minimally Invasive Surgery Navigation System |
| CN109389669A (en)* | 2017-08-04 | 2019-02-26 | 阿里健康信息技术有限公司 | Human 3d model construction method and system in virtual environment |
| CN111386076A (en)* | 2017-12-11 | 2020-07-07 | 柯惠有限合伙公司 | System, method and computer readable medium for estimating chest movement during respiration |
| CN111465364A (en)* | 2017-12-15 | 2020-07-28 | 美敦力公司 | Augmented Reality Solutions for Interrupting, Converting and Augmenting Cardiovascular Surgery and/or Procedure Mapping Navigation and Procedure Diagnosis |
| CN111009024A (en)* | 2019-12-09 | 2020-04-14 | 咪咕视讯科技有限公司 | Method for generating dynamic image, electronic equipment and storage medium |
| CN111161333A (en)* | 2019-12-12 | 2020-05-15 | 中国科学院深圳先进技术研究院 | Prediction method, device and storage medium for liver respiration motion model |
| CN113100935A (en)* | 2021-04-13 | 2021-07-13 | 上海大学 | A preoperative puncture path planning method and training system for lung puncture surgery |
| CN113674393A (en)* | 2021-07-12 | 2021-11-19 | 中国科学院深圳先进技术研究院 | Construction method of respiratory motion model and unmarked respiratory motion prediction method |
Non-Patent Citations (5)
| Title |
|---|
| 吴巨海;徐子海;陈超敏;万伟权;: "实时跟踪放疗中关联模型和预测算法", 中国医学物理学杂志, no. 02, 25 March 2015 (2015-03-25), pages 248 - 250* |
| 崔学理: "基于4D肺部CT图像的呼吸运动模型的构建", 中国优秀硕士学位论文全文数据库，信息科技, no. 2015, 15 May 2015 (2015-05-15), pages 138 - 937* |
| 文丽丽;: "4D-CT在生物医学领域的进展研究", 科技展望, no. 11, 20 April 2016 (2016-04-20), pages 309* |
| 梁睿;张建勋;冯昌利;赵汝琛;: "基于4D CT的肺部运动呼吸模型", 天津大学学报(自然科学与工程技术版), no. 04, 15 April 2015 (2015-04-15), pages 298 - 304* |
| 罗洪艳;张雨雯;黄维;张德燕;文丽丽;: "基于四维CT图像的人体胸腔动态仿真模型研究进展", 中国介入影像与治疗学, no. 09, 10 September 2013 (2013-09-10), pages 571 - 574* |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239768A (en)* | 2022-06-06 | 2022-10-25 | 中日友好医院(中日友好临床医学研究所) | Image processing method and device, electronic device and computer storage medium |
| CN115239768B (en)* | 2022-06-06 | 2025-02-14 | 中日友好医院(中日友好临床医学研究所) | Image processing method and device, electronic device and computer storage medium |
| CN115105205A (en)* | 2022-06-08 | 2022-09-27 | 复旦大学附属中山医院 | In vitro pulmonary nodule localization system and method under electromagnetic navigation under general anesthesia |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12053144B2 (en) | Robotic systems for navigation of luminal networks that compensate for physiological noise | |
| US20230190244A1 (en) | Biopsy apparatus and system | |
| TWI828701B (en) | System and method for lung-volume-gated x-ray imaging, and non-transitory computer-readable storage medium storing instructions | |
| EP2192855B1 (en) | Patent breathing modeling | |
| JP2022017422A (en) | Augmented reality surgical navigation | |
| JP5067398B2 (en) | Ultrasound image and CT image matching system | |
| CN102727236B (en) | By the method and apparatus using the medical image of 3D model generation organ | |
| US20180055576A1 (en) | Respiration motion stabilization for lung magnetic navigation system | |
| CN111770735B (en) | Method and program for generating surgical simulation information | |
| US11779241B2 (en) | Systems, methods, and computer-readable media of estimating thoracic cavity movement during respiration | |
| WO2013141067A1 (en) | Image-generating apparatus | |
| US11723614B2 (en) | Dynamic 3-D anatomical mapping and visualization | |
| JP2015083040A (en) | Image processing apparatus, method, and program | |
| CN114565741A (en) | Operation-assisted data processing method, device, equipment, medium and system | |
| Deligianni et al. | Nonrigid 2-D/3-D registration for patient specific bronchoscopy simulation with statistical shape modeling: Phantom validation | |
| KR102298417B1 (en) | Program and method for generating surgical simulation information | |
| CN112450960A (en) | Virtual endoscope display method based on VR/AR combined digital lung technology | |
| CN115281584A (en) | Flexible endoscope robot control system and flexible endoscope robot simulation method | |
| CN115804652B (en) | Surgical operating system and method | |
| EP4181789B1 (en) | One-dimensional position indicator | |
| KR20250065077A (en) | Method for implementing object behavior based on hand gestures in mixed reality and device therefor | |
| CN120225121A (en) | Apparatus, computer program product and method for assisting in positioning at least one body part of a patient for X-ray acquisition | |
| TW202131875A (en) | System and method for augmenting and synchronizing a virtual model with a physical model | |
| WO2024003252A1 (en) | Augmented reality for ultrasound exams at the point-of-care in combination with mechanical ventilation | |
| CN115919462A (en) | Image data processing system, method and surgical navigation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |