Background
Previous studies have demonstrated that surgical robots are superior to endoscopic surgery in efficiency and surgical efficiency in confined spaces. However, in the robotic surgery, the installation time of the surgical robot increases the total time of the surgery, and there is a temporal disadvantage with respect to the general endoscopic surgery, which cannot be completely removed after the installation time of the robot is shortened by means of repeated training. Therefore, it is imperative that the working efficiency of the surgical robot be improved by other methods.
Such procedures as radical gastric cancer, radical colorectal cancer, and the like include both large-scale and relatively coarse procedures, and small-scale and fine procedures. This type of procedure may represent a hybrid procedure in robotic surgery over a range of motion, as compared to robotic surgery such as traditional radical prostatectomy. Motion scaling is a function that balances the speed and accuracy of surgical tasks. The asynchronous manual adjustment system which depends on real-time dynamic scaling is expected to improve the efficiency of the robot operation in various operations and expand the application range of the operation robot.
In the case of gastric resections using a surgical robot, finding an improper master-slave mapping ratio may slow the surgeon's surgical speed. In order to increase the rate of movement in the workspace while ensuring the safety and effectiveness of the robotic surgical system, it is necessary to adjust the motion scaling of the surgical robot according to the intraoperative conditions.
Among other things, the inconsistent scaling of the motion of the two hands of the surgical robot may be one of the concerns of many researchers. Prior to using the system, many researchers thought that this inconsistent motion scaling could lead to surgeon accommodation problems and physical symptoms. However, after system application, real-time adjustment of motion scaling has proven to be of some practical significance. From the conclusion of the previous experiments, the primary function of the surgeon's left hand is dragging and uncovering, especially active in the suturing task. The main function of the right hand is free movement and excision, so the left hand movement is faster, coarser, the right hand movement is slower, softer, so as not to damage the important structures. The inconsistency of the motion of the hands results in a certain rationality in theory for such "hybrid mapping".
This was found from surgical video primarily because the two hands were primarily functionally different, the right hand was primarily used for dissection and resection and the left hand was primarily used for traction. The right hand with a larger mapping ratio allows fine dissection and resection near the blood vessels and lymph nodes, while the left hand with a smaller mapping ratio allows rapid movement and rapid traction of the mass tissue. When more accurate operation is required, too small a mapping ratio can lead to easy runaway of the execution instrument, excessive operation also needs repeated adjustment, secondary injury is caused, additional hemostasis measures are required, and more duration time is consumed; an excessive master-slave mapping ratio will lack the ability for the implement to quickly pull.
It can be seen that the correct master-slave mapping ratio is mainly affected by the current surgical task. Typically, a procedure has a number of separate tasks, and different tasks have different master-slave mapping ratio requirements. In order to achieve higher operating efficiency, the master-slave mapping ratio needs to be adjusted repeatedly. However, for the operator, the iterative adjustment interrupts the surgical progress and rhythm. If the master-slave mapping proportion can be automatically and dynamically adjusted in real time on the basis of the current operation task, the operation efficiency can be effectively improved. Because the current surgical robot cannot actively identify the surgical task, the surgical task is manually identified, so that additional labor is required, and the reliability is poor.
It should be noted that the foregoing description of the background art is only for the purpose of facilitating a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. The above-described solutions are not considered to be known to the person skilled in the art simply because they are set forth in the background section of the present application.
Disclosure of Invention
The purpose of the invention is that: aiming at the defects in the background art, the technical scheme for controlling the master-slave mapping proportion of the surgical robot based on the characteristics of the surgical task is provided, so that the master-slave mapping proportion is automatically and dynamically adjusted on the basis of the current surgical task, and the surgical efficiency and reliability of the surgical robot are effectively improved.
In order to achieve the above purpose, the invention provides a master-slave mapping proportion control system of a surgical robot, which comprises a master-slave motion controller, a surgical task identification module and an endoscope imaging module; the surgical task identification module automatically identifies the surgical task according to the parameters of each joint driver of the master-slave motion controller and the surgical real-time image data of the endoscope imaging module, and realizes the adjustment of the master-slave motion controller on the mapping proportion.
Further, the joint driver parameters include a current position, a real-time motion rate, a motor real-time current or voltage fed back by the joint driver encoder.
Further, the joint driver parameters comprise the current driving force, torque and motor real-time current or voltage fed back by the brushless motor controller.
Further, the surgical live image data includes two-dimensional and/or three-dimensional video data obtained by the endoscopic imaging module.
Further, the surgical task identification module includes a processor, a memory, and an image processing unit.
Further, the surgical task identification module employs a machine learning based algorithm to identify the surgical task.
Further, the machine learning algorithm comprises one or more of ensemble learning, random forest, convolutional neural network or cyclic neural network algorithms.
Further, the surgical task identification module identifies the organ, the surgical instrument and the relative positional relationship between the organ and the surgical instrument corresponding to each pixel in the video data to identify the surgical task.
Furthermore, the surgical task identification module can realize consistent or inconsistent mapping proportion control on different degrees of freedom according to the requirements of surgical tasks.
The invention also provides a master-slave mapping proportion control method of the surgical robot, which comprises the following steps:
dividing the whole surgical process into a plurality of sub-surgical tasks according to a preset type of surgical operation, and manually adjusting and adapting the master-slave mapping proportion of the sub-surgical tasks according to the requirements of the sub-surgical tasks;
step two, recording parameters of each joint driver based on a time axis;
thirdly, recording the real-time images of the operation based on a time axis;
step four, performing time axis division based on sub-operation tasks on the operation real-time image, and simultaneously performing time axis division of joint driver parameters to obtain operation task marking labels of the joint driver parameters;
step five, training a model, namely inputting operation real-time image data and joint driver parameters through an input interface of an operation task identification module, inputting operation task marking labels simultaneously, storing the operation task marking labels in the operation task identification module, dividing the stored data into a training set and a test set, inputting the training set into a model and training the model based on a machine learning model, training the model and completing preset times of iteration, substituting the model into the test set, and observing the fitting degree of the model; if the model fitting degree is larger than the preset proportion, model training is completed; if the model fitting degree is smaller than the preset proportion, model training is not completed, and training is continued;
step six, if the fitting degree of the iterative model of the next round rises, continuing iteration; if the fitting degree of the iterative model of the latter round is reduced, the iteration is terminated, and more training model data are required to be input;
step seven, when the surgical task recognition module is used for recognizing the surgical task, inputting real-time image data and joint driver parameters into the surgical task recognition module, importing the real-time image data and joint driver parameters into a model which is modeled and trained, obtaining a surgical task recognition result through model calculation, and dynamically adjusting the mapping proportion according to the recognition result;
and step eight, after a preset time interval, performing step seven again.
The scheme of the invention has the following beneficial effects:
according to the master-slave mapping proportion control system and method of the surgical robot, based on the surgical task identification system, surgical tasks are automatically identified according to the joint driver parameters of the master-slave motion controllers and the surgical real-time image data of the endoscope imaging module, so that the adjustment of the master-slave motion controllers on the mapping proportion is realized, the uncertainty of master-slave mapping proportion adjustment based on manual intervention can be reduced, an operator can be helped to adjust to a proper master-slave mapping proportion without manual repeated intervention, the surgical operation speed is improved, the surgical process is accelerated, and the disadvantage of the surgical robot on the surgical time is reduced on the premise of ensuring the surgical operation safety;
other advantageous effects of the present invention will be described in detail in the detailed description section which follows.
Detailed Description
In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. In addition, the technical features of the different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
In the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless explicitly stated and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, a locked connection, a removable connection, or an integral connection; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Example 1:
as shown in fig. 1, embodiment 1 of the present invention provides a master-slave mapping ratio control system for a surgical robot, which includes a master-slave motion controller, a surgical task recognition module, an endoscope imaging module, a surgical robot main console, and a patient-side robot (master robot). The control system is based on a surgical task identification system, and automatically identifies surgical tasks according to the parameters of each joint driver of the master-slave motion controller and the surgical real-time image data of the endoscope imaging module, so that the adjustment of the master-slave motion controller on the mapping proportion is realized. Therefore, uncertainty of master-slave mapping proportion adjustment based on manual intervention is reduced, operators can be helped to adjust to proper master-slave mapping proportion under the condition that manual repeated intervention is not needed, the operation speed is improved, the operation progress is accelerated, and the disadvantage of an operation robot in operation time is reduced on the premise of ensuring operation safety.
In this embodiment, the joint driver parameters include the current position, real-time motion rate, motor real-time current or voltage fed back by the joint driver encoder. Meanwhile, the joint driver parameters also comprise the current driving force, moment and motor real-time current or voltage fed back by the brushless motor controller. Correspondingly, the motion scaling corresponding to the mapping scale may include position motion scaling, velocity motion scaling, acceleration motion scaling, force motion scaling, and the like.
In this embodiment, the surgical live image data according to which the surgical task identification module is based includes two-dimensional and/or three-dimensional video data obtained by the endoscopic imaging module.
In this embodiment, the surgical task recognition module includes a processor, a memory, and an image processing unit (Graphic process unit), where the surgical real-time image data, the joint driver parameters, etc. are stored in the memory, and then passed through an acceleration operation training model of the processor and the image processor, a fitness judgment, etc.
Specifically, the surgical task identification module employs a machine learning based algorithm to identify surgical tasks. Preferably, the machine learning algorithm comprises one or more of ensemble learning, random forest, convolutional neural network or cyclic neural network algorithms, and optimization algorithms thereof.
In this embodiment, the surgical task identification module identifies the surgical task by identifying the organ, the surgical instrument, and the relative positional relationship of the organ and the surgical instrument corresponding to each pixel in the video data.
In this embodiment, the surgical task identification module can implement consistent or inconsistent mapping ratio control in different degrees of freedom, such as forward, backward, left, right, up, down, deflection along the instrument tip axis, and rotation, according to the requirements of the surgical task, to achieve the technical goal of "hybrid mapping".
By adopting the control system provided by the embodiment, through operation design and clinical data verification, three most commonly used motion scaling ratios are 1: 3. 1:6 and 1:10.
Example 2:
meanwhile, as shown in fig. 2, embodiment 2 of the present invention provides a master-slave mapping ratio control method for a surgical robot, and the control system provided in embodiment 1 is adopted, which specifically includes the following steps:
step one, dividing the whole operation process into a plurality of sub-operation tasks, such as free large omentum, cleaning left vascular lymph node of the omentum, free small gastric curvature side, manually adjusting and adapting the master-slave mapping proportion of the sub-operation tasks by a robot operator with abundant experience according to the requirements of the sub-operation tasks, determining that the mapping proportion of the free large omentum adopts a left hand 1:3, the mapping proportion of the right hand 1:3 adopts a left hand 1:6, the mapping proportion of the right hand 1:6 adopts a left hand 1:3 and the mapping proportion of the right hand 1:6 at the left vascular lymph node of the omentum.
And step two, recording parameters of each joint driver based on a time axis, wherein the parameters comprise the current position, the real-time motion speed, the real-time current or voltage of the motor and the like fed back by the encoder of each joint driver, or the current driving force, the moment, the real-time current or voltage of the motor and the like fed back by the brushless motor controller.
And thirdly, recording the real-time images of the operation based on a time axis.
And fourthly, performing time axis division based on sub-operation tasks on the operation real-time image by an expert, and performing time axis division of joint driver parameters at the same time to obtain operation task marking labels of the joint driver parameters.
When the surgical task identification module is used for training a model, surgical real-time image data from the endoscope imaging module and joint driver parameters from the master-slave motion controller are input through an input interface of the surgical task identification module, surgical task mark labels are input simultaneously and stored in the surgical task identification module comprising a processor, a memory and an image processing unit, the stored data are divided into a training set and a test set, the training set is input into a model and trained based on an integrated learning+convolutional neural network model, the model is trained and 100 iterations are completed through acceleration operation of the processor and the image processor, the model is substituted into the test set, and the degree of fitting of the model is observed; if the model fitting degree is greater than a preset proportion, such as 95%, model training is completed; if the model fitting degree is less than 95%, model training is not completed, and training is continued.
Step six, if the fitting degree of the iterative model of the next round rises, continuing iteration; if the fitting degree of the iterative model of the latter round is reduced, the iteration is terminated, and more training model data is required to be input.
Step seven, when the surgical task identification module is used for surgical task identification, the surgical real-time image data from the endoscope imaging module and the joint driver parameters from the master-slave motion controller are input through an input interface of the surgical task identification module, are imported into a model which is modeled and trained, and are subjected to acceleration operation of the processor and the image processor to obtain a surgical task identification result, if the surgical task identification result is identified as a process of cleaning the left-vessel lymph node of the omentum, the surgical robot dynamically adjusts the mapping proportion to be 1:6 of the left hand and 1:6 of the right hand according to the identification result.
And step eight, after a preset time interval, for example, 15 seconds, the identification process of step seven is performed again, if the identification is that the process is at the small curve side of the free stomach, the surgical robot dynamically adjusts the mapping proportion to the mapping proportion of 1:3 of the left hand and 1:6 of the right hand according to the identification result. The master-slave mapping proportion of the two hands can be inconsistent, and only the aim of maximizing the efficiency is achieved.
In addition, if the process of the free large omentum is identified, the surgical robot dynamically adjusts the mapping proportion to the mapping proportion of the left hand 1:3 and the right hand 1:3 according to the identification result.
While the foregoing is directed to the preferred embodiments of the present invention, it will be appreciated by those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.