Disclosure of Invention
The invention aims to provide a multidimensional interaction method of a laparoscopic surgery robot, which aims to solve the technical problems that the control analysis effect is limited and the optimal control cannot be realized due to the limited information quantity of a single dimension in the prior art.
In order to solve the technical problems, the invention specifically provides the following technical scheme:
a multi-dimensional interaction method of a laparoscopic surgical robot, comprising the steps of:
In a laparoscopic simulation operation standard procedure, marking robot standard motion information in visual information recorded by a camera at each procedure node, and marking instrument standard force information in force sense information recorded by a force sense sensor at each procedure node;
Combining the visual information, the force sense information, the robot standard motion information and the instrument standard force information at each flow node into a data set, and training a double-branch neural network based on the data set to construct a multidimensional interaction model for controlling the laparoscopic surgery robot through visual force sense information interaction;
real-time interaction control is carried out on the laparoscopic surgery robot by utilizing real-time motion information of the robot and real-time force information of the instrument, which are predicted by the multidimensional interaction model.
As a preferred mode of the invention, the motion information comprises the angle and position coordinates of a robot joint, and the force information comprises the depth, amplitude, speed and force of a surgical instrument loaded by the robot to pull and cut tissues.
As a preferable scheme of the invention, the method for constructing the multi-dimensional interaction model comprises the following steps:
taking the visual information as an input item of the first branch neural network, and taking the instrument standard force information as an output item of the first branch neural network;
taking the force sense information as an input item of the second branch neural network and taking the robot standard motion information as an output item of the second branch neural network;
Training a first branch neural network and a second branch neural network by utilizing the prediction loss and the reconstruction loss to obtain the multidimensional interaction model;
The multidimensional interaction model is as follows:
;
;
in the formula,Instrument standard force information output for the first branch neural network,The robot standard motion information output by the second branch neural network,In order for the visual information to be of interest,For force sense information, CNN1 is a first branch neural network, and CNN2 is a second branch neural network.
As a preferred embodiment of the present invention, the prediction loss is:
;
in the formula,In order to predict the loss of the picture,Instrument standard force information at a t-th flow node output for the first branched neural network,The true value of the instrument standard force information at the t-th flow node in the dataset,The standard motion information of the robot at the t-th flow node output by the second branch neural network,The true value of the standard motion information of the robot at the t-th flow node in the data set is given, N is the total number of the flow nodes in the standard flow of the laparoscopic surgery,AndAre all L2 Fan Shushi.
As a preferred embodiment of the present invention, the reconstruction loss is:
;
in the formula,The reconstruction is lost to the process,For force sense information at the t-th flow node in the dataset,For visual information at the t-th flow node in the dataset,Is composed ofThe force sense information obtained by the conversion is converted,Is composed ofThe visual information obtained by conversion is converted into a network, N is the total number of process nodes in the laparoscopic surgery standard process,AndAll are L2 Fan Shushi;
the conversion network exchange of the force sense information and the visual information is as follows:
;
;
in the formula,For force sense information at the t-th flow node output by the force sense information conversion network,Instrument standard force information at a t-th flow node output for the first branched neural network,Visual information at the t-th flow node output by the visual information transformation network,And for the instrument standard force information at the t-th flow node output by the first branch neural network, CNN3 and CNN4 are convolutional neural networks.
As a preferred scheme of the invention, the method for carrying out real-time interactive control on the laparoscopic surgery robot by utilizing the real-time motion information of the robot and the real-time force information of the instrument predicted by the multidimensional interactive model comprises the following steps:
Inputting real-time visual information fed back by a camera and real-time force sense information fed back by a force sense sensor into a multidimensional interaction model, outputting real-time motion information of a robot by a first branch neural network in the multidimensional interaction model, and outputting real-time force information of an instrument by a second branch neural network in the multidimensional interaction model;
And controlling the laparoscopic surgery robot to perform surgery according to the real-time motion information of the robot and the real-time force information of the instrument.
As a preferred embodiment of the present invention, the loss function of the conversion network exchange of the force sense information and the visual information is:
;
in the formula,In order to switch the predicted loss of the network,For force sense information at the t-th flow node output by the force sense information conversion network,Visual information at the t-th flow node output by the visual information transformation network,For force sense information at the t-th flow node in the dataset,Visual information at the t-th procedure node in the data set, N is the total number of procedure nodes in the laparoscopic surgery standard procedure,AndAre all L2 Fan Shushi.
As a preferable mode of the present invention, the visual information at each flow node is normalized, and the force sense information at each flow node is normalized.
As a preferred solution of the present invention, the present invention provides a multidimensional interactive system of a laparoscopic surgical robot, which is applied to a multidimensional interactive method of a laparoscopic surgical robot, the system comprising:
the data acquisition unit is used for marking robot standard motion information in visual information recorded by a camera at each process node and marking instrument standard force information in force sense information recorded by a force sense sensor at each process node in a laparoscopic simulation operation standard process;
the model building unit is used for combining the visual information, the force sense information, the robot standard motion information and the instrument standard force information at each flow node into a data set, training the double-branch neural network based on the data set, and building a multidimensional interaction model for controlling the laparoscopic surgery robot through visual force sense information interaction;
And the interaction control unit is used for carrying out real-time interaction control on the laparoscopic surgery robot by utilizing the real-time motion information of the robot and the real-time force information of the instrument predicted by the multidimensional interaction model.
As a preferred aspect of the present invention, there is provided a computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, implement a multi-dimensional interaction method such as a laparoscopic surgical robot.
Compared with the prior art, the invention has the following beneficial effects:
The invention utilizes the visual information and the force sense information to construct the multidimensional interactive model for controlling the laparoscopic surgery robot, realizes analysis of control parameters on the basis of multidimensional information interaction, improves control accuracy, and can realize automatic analysis of the control parameters of the robot by the multidimensional interactive model, improve objectivity and efficiency of control analysis, finally realize standardized control in the surgery operation process and ensure the surgery operation effect.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the present invention provides a multi-dimensional interaction method of a laparoscopic surgical robot, comprising the steps of:
In a laparoscopic simulation operation standard procedure, marking robot standard motion information in visual information recorded by a camera at each procedure node, and marking instrument standard force information in force sense information recorded by a force sense sensor at each procedure node;
Combining the visual information, the force sense information, the robot standard motion information and the instrument standard force information at each flow node into a data set, and training a double-branch neural network based on the data set to construct a multidimensional interaction model for controlling the laparoscopic surgery robot through visual force sense information interaction;
real-time interaction control is carried out on the laparoscopic surgery robot by utilizing real-time motion information of the robot and real-time force information of the instrument, which are predicted by the multidimensional interaction model.
When the control parameters of the laparoscopic surgical robot are analyzed, the information of the vision and the force sense is utilized, the multidimensional control analysis is realized, the interaction uniformity of the multidimensional information is ensured, more accurate control parameters are obtained, and the robot is more accurately controlled in a surgical mode.
In the standard flow of the laparoscopic simulation operation, the visual information and the force sense information are collected to form a data set, so that the interaction uniformity of the visual information and the force sense information can be ensured, namely, the robot motion information and the instrument force information reflected by the visual information and the force sense information at each flow node are standardized, the operation with the best operation effect can be obtained, and meanwhile, the robot motion information and the instrument force information reflected by the visual information and the force sense information at the same flow node are in one-to-one matching correspondence, and the smoothness and the accuracy of the cooperation of the operation robot and the operation instrument are ensured.
After the operation standardized control data set is constructed, the operation standardized control data set is used as a data sample for training the multidimensional interaction model, so that the multidimensional interaction model can analyze multidimensional information (namely visual information and force sense information) to accurately control the laparoscopic operation robot in real time.
The multi-dimensional interaction model constructed by the invention comprises two neural network branch structures, wherein the two neural network branch structures are used for realizing interaction supervision analysis on visual information and force sense information, the first neural network branch is used for establishing a mapping relation between the visual information and instrument standard force information so as to realize outputting the instrument standard force information according to the visual information to achieve the aim of matching the optimal instrument force information for the visual information, the second neural network branch is used for establishing a mapping relation between the force sense information and robot standard motion information so as to realize outputting the robot standard motion information according to the force sense information to achieve the aim of matching the optimal robot motion information for the force sense information, and the two neural network branches respectively establish the mapping relation of the interactivity so that the prediction results respectively output have corresponding matching property, thereby ensuring that the robot motion information and the instrument force information at the same flow node or at the same moment have unity.
When the two neural network branch structures are trained, in order to ensure that the robot motion information and the instrument force information have uniformity and standardization, a self-supervision mode is adopted, firstly, the difference between the instrument standard force information output by the first branch neural network and the true value of the instrument standard force information in the data set and the difference (namely prediction loss) between the true value of the robot standard motion information output by the second branch neural network and the true value of the instrument standard motion information in the data set are used for training the two neural network branches, so that the prediction results output by the two neural network branches have correspondence and matching performance, the prediction results output by the two neural network branches are closest to the true results, the prediction accuracy performance is higher, and the true value of the instrument standard force information in the data set and the true value of the instrument standard motion information in the data set are used as label values for training the two neural network branches, and the mutual self-supervision training of the instrument force information and the robot motion information layer is realized for the two neural network branch structures.
Secondly, training the two neural network branches by using the difference between the force sense information and the original force sense information obtained by reflection of the instrument standard force information output by the first branch neural network and the difference (namely reconstruction loss) between the visual information and the original visual information obtained by reflection of the robot standard motion information output by the second branch neural network, so that the instrument standard force information predicted and output by the first branch neural network according to the original visual information is further ensured to be in accordance with the original force sense information, and the original visual information and the original force sense information have time sequence uniformity.
Therefore, the first branch neural network is guaranteed to output the instrument force information matched with the visual information, the robot motion information predicted and output by the second branch neural network according to the original force sense information is guaranteed to be in accordance with the original force sense information, and the original visual information and the original force sense information have timing sequence uniformity, so that the second branch neural network is guaranteed to output the robot motion information matched with the force sense information, the use of reconstruction loss is guaranteed to ensure that the instrument force information and the robot motion information output by the two neural network branch structures inherit the timing sequence uniformity characteristic between the original visual information and the original force sense information, and the original visual information and the original force sense information are still utilized as the label value for training the two neural network branches in the reconstruction loss, so that the mutual self-supervision training of the visual information and the force sense information layers is realized for the two neural network branch structures.
In order to realize the force sense information obtained by reflecting the instrument standard force information and the visual information obtained by reversely mapping the robot standard motion information, a conversion network is constructed, and the mapping relation between the instrument standard force information and the force sense information and the mapping relation between the robot standard motion information and the visual information are established, so that the force sense information obtained by predicting the instrument standard force information and the visual information obtained by predicting the robot standard motion information are respectively realized.
The motion information comprises robot joint angles and position coordinates, and the force information comprises the depth, amplitude, speed and force of the surgical instrument loaded by the robot to the tissue pulling and cutting operation.
In the standard flow of the laparoscopic simulation operation, the visual information and the force sense information are collected to form a data set, so that the interaction uniformity of the visual information and the force sense information can be ensured, namely, the robot motion information and the instrument force information reflected by the visual information and the force sense information at each flow node are standardized, the operation with the best operation effect can be obtained, and meanwhile, the robot motion information and the instrument force information reflected by the visual information and the force sense information at the same flow node are in one-to-one matching correspondence, so that the smoothness and the accuracy of the cooperation of the operation robot and the operation instrument are ensured, and the method comprises the following steps of:
as shown in fig. 3, the method for constructing the multidimensional interaction model includes:
taking the visual information as an input item of the first branch neural network, and taking the instrument standard force information as an output item of the first branch neural network;
taking the force sense information as an input item of the second branch neural network and taking the robot standard motion information as an output item of the second branch neural network;
Training a first branch neural network and a second branch neural network by utilizing the prediction loss and the reconstruction loss to obtain a multidimensional interaction model;
the multidimensional interaction model is as follows:
;
;
in the formula,Instrument standard force information output for the first branch neural network,The robot standard motion information output by the second branch neural network,In order for the visual information to be of interest,For force sense information, CNN1 is a first branch neural network, and CNN2 is a second branch neural network.
The multi-dimensional interaction model constructed by the invention comprises two neural network branch structures, wherein the two neural network branch structures are used for realizing interaction supervision analysis on visual information and force sense information, the first neural network branch is used for establishing a mapping relation between the visual information and instrument standard force information so as to realize outputting the instrument standard force information according to the visual information to achieve the aim of matching the optimal instrument force information for the visual information, the second neural network branch is used for establishing a mapping relation between the force sense information and robot standard motion information so as to realize outputting the robot standard motion information according to the force sense information to achieve the aim of matching the optimal robot motion information for the force sense information, and the two neural network branches respectively establish the mapping relation of the interactivity so that the prediction results respectively output have corresponding matching property, thereby ensuring that the robot motion information and the instrument force information at the same flow node or at the same moment have unity.
The predicted loss is:
;
in the formula,In order to predict the loss of the picture,The instrument standard force information at the t-th flow node output for the first branched neural network (corresponding to that in fig. 3),True values for instrument standard force information at the t-th flow node in the dataset (corresponding to those in FIG. 3),Robot standard motion information at the t-th flow node (corresponding to that in fig. 3) output for the second branch neural network),True values for the robot standard motion information at the t-th flow node in the dataset (corresponding to those in fig. 3) N is the total number of process nodes in the laparoscopic surgery standard process,AndAre all L2 Fan Shushi.
When the two neural network branch structures are trained, in order to ensure that the robot motion information and the instrument force information have uniformity and standardization, a self-supervision mode is adopted, firstly, the difference between the instrument standard force information output by the first branch neural network and the true value of the instrument standard force information in the data set and the difference (namely prediction loss) between the true value of the robot standard motion information output by the second branch neural network and the true value of the instrument standard motion information in the data set are used for training the two neural network branches, so that the prediction results output by the two neural network branches have correspondence and matching performance, the prediction results output by the two neural network branches are closest to the true results, the prediction accuracy performance is higher, and the true value of the instrument standard force information in the data set and the true value of the instrument standard motion information in the data set are used as label values for training the two neural network branches, and the mutual self-supervision training of the instrument force information and the robot motion information layer is realized for the two neural network branch structures.
The reconstruction loss is as follows:
;
in the formula,The reconstruction is lost to the process,For force sense information at the t-th flow node in the dataset,For visual information at the t-th flow node in the dataset,Is composed ofThe force sense information obtained by the conversion is converted,Is composed ofThe visual information obtained by conversion is converted into a network, N is the total number of process nodes in the laparoscopic surgery standard process,AndAll are L2 Fan Shushi;
the conversion network exchange of the force sense information and the visual information is as follows:
;
;
in the formula,Force sense information at the t-th flow node of the output of the force sense information conversion network (corresponding to that in fig. 3),Instrument standard force information at a t-th flow node output for the first branched neural network,Visual information at the t-th flow node (corresponding to that in fig. 3) output for the visual information transformation network),And for the instrument standard force information at the t-th flow node output by the first branch neural network, CNN3 and CNN4 are convolutional neural networks.
Secondly, training the two neural network branches by using the difference between the force sense information and the original force sense information obtained by reflection of the instrument standard force information output by the first branch neural network and the difference (namely reconstruction loss) between the visual information and the original visual information obtained by reflection of the robot standard motion information output by the second branch neural network, so that the instrument standard force information predicted and output by the first branch neural network according to the original visual information is further ensured to be in accordance with the original force sense information, and the original visual information and the original force sense information have time sequence uniformity.
Therefore, the first branch neural network is guaranteed to output the instrument force information matched with the visual information, the robot motion information predicted and output by the second branch neural network according to the original force sense information is guaranteed to be in accordance with the original force sense information, and the original visual information and the original force sense information have timing sequence uniformity, so that the second branch neural network is guaranteed to output the robot motion information matched with the force sense information, the use of reconstruction loss is guaranteed to ensure that the instrument force information and the robot motion information output by the two neural network branch structures inherit the timing sequence uniformity characteristic between the original visual information and the original force sense information, and the original visual information and the original force sense information are still utilized as the label value for training the two neural network branches in the reconstruction loss, so that the mutual self-supervision training of the visual information and the force sense information layers is realized for the two neural network branch structures.
The method for carrying out real-time interaction control on the laparoscopic surgery robot by utilizing the real-time motion information of the robot and the real-time force information of the instrument predicted by the multidimensional interaction model comprises the following steps:
Inputting real-time visual information fed back by a camera and real-time force sense information fed back by a force sense sensor into a multidimensional interaction model, outputting real-time motion information of a robot by a first branch neural network in the multidimensional interaction model, and outputting real-time force information of an instrument by a second branch neural network in the multidimensional interaction model;
And controlling the laparoscopic surgery robot to perform surgery according to the real-time motion information of the robot and the real-time force information of the instrument.
The loss function of the conversion network exchange of force sense information and visual information is as follows:
;
in the formula,In order to switch the predicted loss of the network,For force sense information at the t-th flow node output by the force sense information conversion network,Visual information at the t-th flow node output by the visual information transformation network,For force sense information at the t-th flow node in the dataset,Visual information at the t-th procedure node in the data set, N is the total number of procedure nodes in the laparoscopic surgery standard procedure,AndAre all L2 Fan Shushi.
In order to realize the force sense information obtained by reflecting the instrument standard force information and the visual information obtained by reversely mapping the robot standard motion information, a conversion network is constructed, and the mapping relation between the instrument standard force information and the force sense information and the mapping relation between the robot standard motion information and the visual information are established, so that the force sense information obtained by predicting the instrument standard force information and the visual information obtained by predicting the robot standard motion information are respectively realized.
And normalizing the visual information at each flow node and normalizing the force sense information at each flow node.
As shown in fig. 2, the present invention provides a multi-dimensional interaction system of a laparoscopic surgical robot, which is applied to a multi-dimensional interaction method of a laparoscopic surgical robot, the system comprising:
the data acquisition unit is used for marking robot standard motion information in visual information recorded by a camera at each process node and marking instrument standard force information in force sense information recorded by a force sense sensor at each process node in a laparoscopic simulation operation standard process;
the model building unit is used for combining the visual information, the force sense information, the robot standard motion information and the instrument standard force information at each flow node into a data set, training the double-branch neural network based on the data set, and building a multidimensional interaction model for controlling the laparoscopic surgery robot through visual force sense information interaction;
And the interaction control unit is used for carrying out real-time interaction control on the laparoscopic surgery robot by utilizing the real-time motion information of the robot and the real-time force information of the instrument predicted by the multidimensional interaction model.
The invention provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and when a processor executes the computer execution instructions, a multidimensional interaction method such as a laparoscopic surgical robot is realized.
The invention utilizes the visual information and the force sense information to construct the multidimensional interactive model for controlling the laparoscopic surgery robot, realizes analysis of control parameters on the basis of multidimensional information interaction, improves control accuracy, and can realize automatic analysis of the control parameters of the robot by the multidimensional interactive model, improve objectivity and efficiency of control analysis, finally realize standardized control in the surgery operation process and ensure the surgery operation effect.
The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application, the scope of which is defined by the claims. Various modifications and equivalent arrangements of this application will occur to those skilled in the art, and are intended to be within the spirit and scope of the application.