Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/313—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
  • G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
  • G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine

Definitions

• This inventionrelates to methods and systems for simulating medical procedures for training and for remote control .
• Endoscopic and laparoscopic interventionis being fervently applied to procedures (such as cholecys- tectomies, hernia repair, nephrectomies, appendectomies, and numerous gynecological procedures) being praised for fast patient recovery time and decreased hospital and surgery costs.
• proceduressuch as cholecys- tectomies, hernia repair, nephrectomies, appendectomies, and numerous gynecological procedures
• U.S. Patent No. 5,149,270issued to McKeown, discloses an apparatus for practicing endoscopic surgi ⁇ cal procedures.
• the apparatushas a cavity in which an object simulating a human organ is mounted for perform- ing a practice procedure.
• the cavityis closeable to outside view, thereby forcing the individual practicing the procedure to use and manipulate the instruments relying only upon a viewing scope in an attempt to mimic the actual diagnostic and operating conditions.
• This configurationby relying upon physical linkages and organ materials, lacks the versatility, robustness, and repeatability of a programmable simulation device.
• a further object of the above inventionis to provide a method for providing a virtual reality in response to a position and orientation of a member representative of a medical instrument.
• an object of the above inventionis to provide a method and system for controlling a medical instrument.
• An additional object of the above inventionis to provide a four-axis controller for a medical instru ⁇ ment.
• the present inventionprovides a method for simulating actual medical procedures which utilize at least one medical instrument.
• the actions of the medical instrumentare measured during an actual medical procedure. These measured actions of the medical instrument are generated on a member represen ⁇ tative of the medical instrument so as to simulate the actual medical procedure.
• the present inventionfurther provides a system for simulating actual medical procedures which utilize at least one medical instrument.
• the systemcomprises means for measuring the actions of the medical instrument during an actual medical procedure.
• Meansare also provided for generating the measured actions on a member repre- sentative of the medical instrument whereby the actions of the member simulate the actual medical procedure.
• the present inventionalso provides a method for providing a virtual reality in response to a position and orientation of a member representative of a medical instrument.
• An electrical signalis generated for each of a plurality of degrees of freedom of the member as a function of the position and orientation of the member in three-dimen ⁇ sional space.
• At least one virtual reality force fieldis generated in response to the generated signals.
• a force signalis generated for each degree of freedom as a function of the force field.
• a force on the memberis generated for each force signal, thus providing the virtual reality.
• the present inventionadditionally provides a method for controlling a medical instrument.
• the position and orientation of a member representative of the medical instrumentis measured.
• Actionsare generated on the medical instru ⁇ ment based upon the measured position and orientation of the member.
• the forces acting upon the medical instru ⁇ mentare measured. Forces are generated on the member based upon the measured forces acting upon the medical instrument.
• the present inventionalso provides a system for controlling a medical instrument.
• the systemcomprises means for measuring the position and orientation of a member representative of the medical instrument. Means are provided to generate actions on the medical instrument based upon the measured position and orientation of the member.
• the systemfurther comprises means for measur ⁇ ing the forces acting upon the medical instrument. Means are provided to generate forces on the member based upon the measured forces acting upon the medical instrument.
• the present inventionalso provides a four-axis controller for a medical instrument.
• the systemcomprises a member representative of the medical instrument.
• a four-axis interface devicewhich is responsive to the position and orientation of the member, is secured to the member. Electrical signals representative of the position and orientation of the member are generated in the four-axis interface device. Means for actuating the medical instrument in response to the electrical signals are included. Means for generating a force field based upon a force induced upon the medical instrument are provid ⁇ ed.
• a force signal for each axis of the interface deviceis generated by further means, with the actuators responsive to these force signals to provide a force to each axis of the interface device for reflecting a force on the medical instrument to the member.
• FIGURE 1shows the insertion of a scope and an instrument into a body during a laparoscopic medical procedure
• FIGURE 2is a block diagram representation of the recording apparatus
• FIGURE 3depicts the systematic scanning of an anatomical feature to generate a complete imagery record
• FIGURE 4is a block diagram representation of the virtual reality training/replay apparatus
• FIGURE 5is a block diagram representation of the incorporation of the recording apparatus with the virtual reality training/replay apparatus
• FIGURE 6is a block diagram representation of using the virtual reality replay system to provide a surgeon with preoperative imagery
• FIGURE 7is a block diagram representation of using the virtual reality system in conjunction with robotic endoscopic medical equipment
• FIGURE 8is a block diagram representation of the force/tactile subsystem control concept
• FIGURE 9is a schematic representation of the laparoscopic simulator.
• FIGURE 10shows the four-axis force reflecting module within an opening in the laparoscopic simulator.
• viteoretinal, otologic, laryngeal and stereotactic proceduresviteoretinal, otologic, laryngeal and stereotactic procedures; intubations, including neonatal procedures; and arterial and venous replacement and bypass methods used, for example, in treatment of innominate and subclavian occlusive disease.
• the apparatuscan be used to provide realistic simulations of patients on which to practice medical procedures for training and qualification purposes, to provide a means for trainees to see and feel actual (or simulated) surgeries, to display information from previously obtained medical diagnostics or image modali- ties (CT data, PET data, MRI data, etc.) to the surgeon overlaid on current image information during the sur ⁇ gery, and to provide the man-machine interface for robotic surgery.
• CT datacomputed tomography
• PET dataPET data
• MRI datamagnetic resonance imaging
• the apparatusconsists of a two or three dimensional display device, a two or three dimensional sound device, a graphics/image processing engine and storage module capable of real-time medical image generation, and programmable tactile/force reflecting mechanisms which can generate the "feel" of medical instruments and the interaction of these instruments with the anatomical simulation.
• the preferred embodi ⁇ ment of the inventionis designed for simulating endo ⁇ scopic medical procedures due to the minimally invasive nature of these techniques.
• the preferred embodiment of this inventionaddresses the problems presented by the prior art by providing means for controlled, preprogrammable simula ⁇ tion of a medical procedure (using both active tactile feel and imagery) so that standardized practice and qualification sessions can be performed by surgeons.
• the same embodimentcan also be used to augment the data normally delivered to the surgeon in actual endoscopic surgery, and can provide surrogate feels and imagery to surgeon trainees from actual surgeries being performed by expert surgeons.
• the basic medical procedure simulation systemconsists of two basic functions:
• FIG. 2the recording appara- tus 20 that allows the critical events in a medical procedure to be measured and recorded during actual surgeries performed by expert surgeons is shown.
• Medical instruments 16 having pressure and position sensors 22are connected to a data digitizer 24.
• the pressure and position data from the digitizer 24is recorded by a recording system 26 consisting of a computer and a storage subsystem.
• a viewing scope 14 having pressure and position sensors 28is connected to an image digitizer/recorder 30.
• the image digitiz- er/recorder 30is also connected to the recording system 26.
• the recording system 26can either compress and store digital image data or store analog video on a computer-controlled video cassette recorder (VCR) .
• VCRcomputer-controlled video cassette recorder
• the non-image datais stored on digital disk or tape storage.
• An accurate time base 32is also connected to the recording subsystem 26 to provide time stamps for the instrument locations, forces or pressures, and actions such as trigger actuations during the specified scope locations, forces (or pressures) , and image data captured during the operation.
• Each data item(force, position, trigger action item or image captured) is time stamped so that it is placed accurately into the time sequence of events during surgery.
• the recording apparatus 20 in Figure 2can be directly connected to a playback apparatus (in Figure 5) to provide immediate feedback to a trainee.
• Figure 3depicts the systematic scanning of anatomical features using the scope 14 to generate a complete imagery record of organ and tissue orientation surrounding the site ' of medical simulation. This supports generation of in-between data and interpolated views based on recorded data.
• the virtual reality training/replay apparatus 40 needed to playback the recorded datais incorporated in Figure 4.
• This apparatus 40generates the virtual space which displays the time stamped and interpolated data stored in the image and data storage system 42.
• a virtual reality control computer 44receives image and non-image data from the image and data storage system 42. Position, orientation, and force data are processed by the computer 44 and sent to the force/tactile pro- grammed controller and hand/instrument tracking subsys ⁇ tem 46.
• the force/tactile programmed controller and hand/instrument tracking subsystem 46is connected to a tactile/force reflecting subject simulation mechanism 48 in order to provide force feedback.
• the virtual reality control computer 44obtains image data from the image and data storage system 42 based on the position and orientation of the instruments. The control computer 44 processes this data and relays it to an image warp processor 50.
• the image warping function 50uses morphing, defined as the correspondence of two images through corresponding polygon patches, as the means for providing interpolated images from data collected during recording sessions.
• a 3D graphics generator 52is also connected to the virtual reality control computer 44 in order to generate accurate three dimensional images of endoscopic instru- ments.
• a graphics overlay unit 54is used to overlay the three dimensional images from the 3D graphics generator 52 on the morphed video derived imagery from the image warp processor 50. The output of the graphics overlay unit 54 is fed into a three dimensional display system 56.
• An optional (denoted by the dashed lines) trainee head tracking system 58can be connected to the virtual reality control computer 44 to provide informa ⁇ tion for the three dimensional display generation function to control generation of view dependent imagery assuming that the display system 56 is a head mounted system.
• future systemsare expected to replace CRT viewing of endoscope-derived image data with head- mounted three-dimensional viewing. This viewing ar- rangement will require the head position sensing subsys ⁇ tem 58.
• a head position sensing subsystem 58also allows the incorporation of a three- dimensional sound generation system 60.
• the sound generation systemreceives audio information from the virtual reality control computer 44 and sends this information to a sound system 62.
• the use of sound generationis also optional since it is not necessary in simulating endoscopic procedures.
• Figure 5shows how the recording apparatus 20 in Figure 2 can be used to acquire image and manipula ⁇ tion data 70 which is then used to compose simulation data sets 72 for replay on the virtual reality replay apparatus 40 from Figure 4.
• An experienced surgeon 74performs an actual endoscopic medical procedure using endoscopic instruments 76.
• the recording system 20is used to measure and record the sights, sounds, feels, and actions occurring during the actual medical proce- dure.
• the sights, sounds, feels, and actions dataare stored in the measured data and image storage system 70.
• Simulation data sets 72are formed off-line from the measured data 70 by a data composition system 78.
• the virtual reality training system 40uses images and data in the simulation database 72 to allow a trainee 80 to explore this data.
• the trainee 80can view, hear, and feel operations through manipulation and positioning of the simulated instruments, or based on his head loca ⁇ tion.
• Images or data not in the simulation database 72are generated through interpolation between images/data items which are in the . database 72 (interpolation between images is accomplished through morphing -- mapping predetermined image polygons between images) .
• Figure 5also shows how the recording appara ⁇ tus 20 can be used to pick-up data and imagery from an in-progress medical procedure for immediate replay on the virtual reality replay/training apparatus 40.
• the virtual reality replay apparatus 40immediately receives the output of the recording appara ⁇ tus 20 via a communications channel 82. Therefore, the trainee 80 does not explore the input data and imagery through manipulation, but rather experiences the data stream.
• This dataincludes the virtual forces generated by the force/tactile subsystem and applied to the trainee's hands through the simulated instruments.
• the trainee 80can gain intuitive understanding of the medical procedure, including the feelings, sights, and sounds of the operation, without the risks and possible complications associated with medical inter ⁇ vention.
• Figure 6shows how elements of the virtual reality replay system 40 can be used to provide the surgeon 90 with views of preoperative information and imagery 92, such as CT scans and MRI scans, to augment what is available now for endoscopic procedures (for instance, video imagery from the laparoscope) .
• a surgeon 90performs an actual medical procedure using endoscopic instruments 94.
• the recording system appara ⁇ tus 20is used to measure and record the actions and images in the instruments 94 during the procedure.
• the virtual reality training system 40incorporates pre ⁇ operative test data and images 92 with the output of the recording apparatus 20.
• Figure 7shows how the virtual reality systems described already can also be used to provide feedback from and control information to robotic endoscopic medical equipment 100.
• a remote surgeon's 102 instru ⁇ ments, viewing scope, and head positionare measured and recorded by the virtual reality training system 104. This data is transmitted to the robotic endoscopic surgery equipment 100 via a communications channel 106.
• a local surgeon's 108 instrument, scope, and head positionis also measured and recorded by a second virtual reality training system 110.
• the robotic instrumentsare controlled to perform the actions of either the remote surgeon 102 or the local surgeon 108 on the patient 112. • The resulting sights, sounds, actions, and feels are experienced by both the remote surgeon 102 and the local surgeon 108.
• the virtual reality concept described in this documentprovides the means for sharing control of such robotic equipment 100 between several surgeons or trainees.
• the most important subsystems in the virtual reality replay apparatusare the force/tactile control ⁇ ler 46 and the tactile/force reflecting mechanism 48. These systems use force reflection information generated from the simulation database (or model-base) to create the feel of the medical instruments.
• Figure 8shows a control concept for creating the feel of medical instru ⁇ ments during a simulation.
• the hands 120 of a surgeon or traineegrasp and move a member representative of an instrument.
• the position of the instrument 122is determined and fed into a geometric model of an organ 124.
• the geometric model 124includes data representing the organ size, position, surface characteristics (in terms of elasticity, resistance to searing, slipperi- ness, etc.).
• An apparatus for simulating laparoscopic medical procedures shown in Figure 9uses the servo systems concept described in Figure 8 to the provide force-feedback control.
• the apparatusconsists of a box-like enclosure 130 which provides three openings 132 for insertion of instruments 134.
• instruments 134When the instruments 134 are inserted they are clamped to four-axis devices 136 such as the force reflection mechanism 140 shown in Figure 10.
• This mechanism 140includes, in effect, a small backdrivable robot which moves in 4 degrees of freedom: 1 linear position (extension) and 3 attitudes (roll, pitch, and yaw) . This is accomplished by the extension actuator 142, roll actuator 144, pitch actua ⁇ tor 146, and yaw actuator 148.
• This mechanism 140can translate and provide forces/torques under programmed control (generated by the modeling subsystem in Figure 8) which feel to the trainee like the interactions characteristic of the actual medical procedure.
• programmed controlgenerated by the modeling subsystem in Figure 8
• These four-axis actuation systemsare reduced implementations of universal force reflection mechanisms described in co-pending application Serial No. 984,324, filed Decem ⁇ ber 2, 1992.
• each of the four-axis devicesIn operation, each of the four-axis devices
• the controller 138may be a virtual reality simulation of the environment of a medical procedure, formed either artificially or based upon data collected during an actual medical procedure.
• the controller 138could also represent one or more slave devices which respond to follow the actions of the instruments 134, measure the forces acting on the slave device, and transmit force commands 137 back to the four-axis devices 136.
• the four-axis devicecould be implemented. This device could read the position of a user's hand by means of a laser tracking system and feedback forces by urging the user's hand with jets of air. In the preferred embodiment, however, the user's hand will grip a member 134 representative of a medical instrument or another medical tool.
• the specific four-axis device shown in Figure 10is now examined in more detail. The position, orientation, velocity, and/or acceleration provided on the force reflective mechanism 140 by a member 134 is sensed and transmitted as a command to the simulation which implements a virtual reality force field.
• a force commandis generated based on the force field value for the position, orientation, velocity, and/or acceleration of the member 134 given by the simulation.
• This force commandis transmitted to force reflective mechanism 140.
• Forceis then generated on the member 134 by the force reflective mechanism 140 using four small, brushless, DC servo motors.
• the four axis force-reflection output and four-axis orientation and position controlprovides the user with direct kinesthetic feedback from the virtual environment traversed.
• a queuing button(not shown) can be incorpo ⁇ rated in order to actuate the instrument within the simulation or control.
• the inventionin its preferred embodiment is applicable to controlling a virtual or simulated medical environment.
• the inventionis also well suited to the control of a remote or physical medical instru ⁇ ment.
• the present inventionis suited for application to any number of axes. For instance, a full six-axis interface could be used for simulating or controlling medical procedures.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Human Computer Interaction (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Surgery (AREA)
• Computational Mathematics (AREA)
• Pathology (AREA)
• Pure & Applied Mathematics (AREA)
• Business, Economics & Management (AREA)
• Educational Administration (AREA)
• Educational Technology (AREA)
• Mathematical Optimization (AREA)
• Mathematical Analysis (AREA)
• Biophysics (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Optics & Photonics (AREA)
• Mathematical Physics (AREA)
• Radiology & Medical Imaging (AREA)
• Algebra (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Molecular Biology (AREA)
• Animal Behavior & Ethology (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Medicinal Chemistry (AREA)
• Chemical & Material Sciences (AREA)
• Endoscopes (AREA)
• Instructional Devices (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=22206450

Family Applications (1)

Country Status (3)

Families Citing this family (40)

Family Cites Families (6)

• 1994
  • 1994-06-27EPEP94922027Apatent/EP0658265A1/denot_activeWithdrawn
  • 1994-06-27AUAU72516/94Apatent/AU7251694A/ennot_activeAbandoned
  • 1994-06-27WOPCT/US1994/007218patent/WO1995002233A1/ennot_activeApplication Discontinuation

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events