VIRTUAL FORCE FEEDBACK FOR SYNTHETI[C ENVIRONMENTThis invention relates to operating instruments in a real environment with the assistance of a virtual reality environment.
Remotely operated instruments have been used to perform delicate operations in areas which are not readily accessible to human operators. For example, in minimally invasive surgery such as key hole surgery, instruments are inserted into the body via a relatively small opening. The surgeon manipulates the instruments from outside of the body.
Another example is the handling of hazardous substances or the assembly or disassembly of potentially hazardous equipment or equipment which has to be handled in a special environment such as vacuum or ultra-clean environment.
It is known to use the electro-mechanical or optomechanical interfaces in such applications. The lack of "feel" which would otherwise result can be compensated for by provided by mechanical feedback by means of sensors coupled to the instruments. Thus an operator "feel" resistance when he touches an object in proportion to the force which he is exerting on the object and thus is able to exert only sufficient force to handle the object without damaging it. Similarly a surgeon can "feel" the resistance presented by body tissues during surgery.
However, the body contains a number of ultra-thin membranes and other delicate tissues which are incapable of exerting enough force on a sensor to provide significant mechanical feedback. Such membranes may therefore be damaged by for example an unintentional slip by the operator. Similarly, in delicate manipulation of mechanical objects an unintentional movement by the operator could result in damage to a fragile item. In the case of hazardous substances, damage could result from the unintentional bringing-together of objects which should be kept apart.
For example, chemicals which react together violently could be unintentionally mixed, or an unintentional electrical short-circuit could be caused when performing manipulations in the vicinity of a "live" circuit.
It has been proposed to use synthetic environments, also known as virtual reality environments, to simulate real environments for training purposes. It has been proposed to create a synthetic environment simulating part of the human body to allow a surgeon to practice operations in the synthetic environment before carrying out a corresponding operation on an actual patient. Proposals have been made to create a virtual reality atlas of the human body. Such an atlas would assist a surgeon in guiding his instruments through the body of the patient in the course of an operation by providing a three-dimensional view of the patient. These proposals do not address the problem of preventing inadvertent damage to delicate or sensitive objects.
In accordance with the invention, apparatus for remotely operating a member in a real environment comprises: means to generate a virtual reality environment matching the real environment; means to cause the member to respond to the virtual reality environment; and means to define a region of the virtual reality environment to have properties different from those of the corresponding region of the real environment.
Thus although the operator is moving in the real environment he experiences the sensations which he receives from the synthetic environment. These may be in addition to or instead of the sensations he receives from the real environment.
The properties of the defined virtual reality region may be different from those of the real environment. For example the virtual environment may be made more viscous, thereby providing damping where no damping exists in the real environment. The defined region may be a virtual solid region which inhibits or prevents passage by the member into or through the corresponding real region. The virtual region could be made resilient such that it could be entered by applying sufficient force, but be such as to eject the probe therefrom when the force was removed.
The different properties may be disabled. This allows intentional entry into the defined region but only by a positive action on the part of the operator. This prevents accidental entry into the defined region and is of particular use where the corresponding region of the real environment contains an object which is liable to be damaged by the probe.
The probe may comprise proximity sensor responsive to real objects, the virtual reality environment comprising a virtual proximity sensor responsive to virtual objects.
The subjects of the real and virtual sensors may be combined such that the operator senses the same sensations on approaching either a real object of a virtual object.
The virtual reality environment may comprise means to produce a warning signal when the probe approaches the defined virtual region.
Alarm means may be provided to indicate when the probe approaches the defined virtual region. This gives the operator a positive indication that the virtual region is being approached and allows him to react accordingly.
For example in surgery the defined region may protect a delicate membrane associated with the eye which could be damaged by coming into contact with a probe. The warning would give the operator notice that he was approaching a critical region where extreme care has to be taken.
The means may be provided to "remember" the path taken by the probe on at least part of its way to the location where work is to be performed. Regions bounded by this region may then be defined as solid regions into which the probe is not allowed to go. This facilitates withdrawal of the probe after work has been completed, as it is then constrained to move only on its original route.
Automatic withdrawal may be effected by defining a virtual solid region beyond the probe and then expanding it, thereby "pushing" the probe back along its original path.
An embodiment of the invention will now be described by way of non-limiting example only with reference to the drawings in which:Figure 1 shows an embodiment of an apparatus in accordance with the invention; andFigure 2 illustrates an aspect of the operation of the invention.
Figure 1 illustrates in schemmatic form apparatus for performing minimally invasive surgery on the human body.
A Virtual Reality (VR) generator 2 is coupled via line 20 to a control and interface unit 1 whose function will be described later. A virtual reality display unit' 3 is coupled to the control unit 1 via line 30. A display unit 3 is arranged to present a visual image of the virtual reality environment to the operator. First and second hand controls 4, 5 are coupled to the control unit 1 via lines 40, 50. Signals from hand controls 4, 5 to the control unit 1 on lines 40, 50 inform the control unit of the spatial positions of the hand control units 4, 5. Feedback signals are fed from the control unit 1 to the hand control units 4, 5 on lines 42, 52 as will be described later.
First and second probes 6, 7 are caused to move by signals on lines 60, 70 from control unit 1. Respective sensors 64, 74 on probes 6, 7 are coupled to the control unit 1 via lines 62, 72. Signals from a video camera 8 or other image device are coupled to the control unit 1 via line 80.
Operation is as follows. Virtual reality generator 2 contains a three dimensional representation of at least that part of the body on which an operation is to be performed. This may for example be a standardised representation which is "stretched" to fit the dimensions of the actual body being operated on by the so called "rubber sheet" transformations in known manner. The actual dimensions of the body may be obtained by any convenient method such as Magnetic Resonance Imaging or by using XRays. An image of the actual body as seen from the location of the operator's probes is generated by video camera 8 in known manner e.g. by using an optical relay system to convey an image to a camera mounted externally of the body. The objective of the relay system is coupled to one of the probes 6, 7 so that the objective's spatial position and orientation are known.The interface unit 1 passes signals from the hand controls 4, 5 to the VR generator 2 so as to make the virtual reality image generated by the VR generator 2 of the relay system coupled to correspond to the view point seen by the objective camera 8. The actual and virtual images are viewable on the display 3. The display may show either the virtual image, the actual image, or a combination of both, either separately or superposed, according to operational requirements or convenience. The movement of the hand controls 4, 5 causes corresponding movement on probes 6, 7.
Signals generated by force sensors 64, 74, on the ends of probes 6, 7 are fed back to control unit 1. Unit 1 generates corresponding force feedback signals which are fed via lines 42, 52 to hand controls 4, 5 so as to provide tactile feedback to the operator.
In accordance with the invention, virtual reality generator 2 has virtual barriers built in to its map which cause virtual force feedback signals to be generated when a probe attempts to cross a virtual barrier. The virtual feedback signals are processed together with any actual force feedback signals which may be produced by sensors 64, 74 and the combined feedback signals are applied to hand controls 3, 4. Thus, on approaching a virtual barrier, the operator experiences the same sort of resistance that he would if he were approaching a physical barrier. Thus, referring to Figure 2, say it is required to investigate a real object A located in the vicinity of two delicate objects such as membranes B, C. Membranes B, C offer no appreciable physical resistance to probe 6 and could readily be damaged by probe 6 if it were to touch them.To prevent such damage the virtual reality generator 1 has built in to its map virtual barriers b, c which are models of physical barriers and which surround the delicate membranes B, C. The virtual barriers b, c have no physical existence and exist only in the virtual reality generator.
As a probe 4 approaches membrane B, so the control means 1 senses that the probe is approaching the virtual barrier b and causes a virtual feedback signal to be generated. This is summed with the actual feedback signal produced in response to sensor 64 and applied on line 42 to hand control 4 so as to prevent probe 6 from crossing the virtual barrier b. This also is effective to prevent damage should the operator accidentally release the hand control or make an unintended movement for any reason.
A number of modifications are possible within the scope of the invention. The virtual barrier need not be fixed in space but may be arranged to track the subject which it is protecting in real time.
For example, for membrane associated with the eye, the eye movements may be sensed and the virtual barrier surrounding the membrane moved accordingly. Movement of the virtual barrier may be such as to push a probe out of the way, thereby preventing damage which would be caused by the membrane striking the probe, as well as by the probe striking the membrane.
The virtual barrier may have any desired characteristics.
Instead of simulating a physical barrier which prevents the passage of a probe through it, it may merely provide an increased resistance to the passage of a probe, which resistance can be overcome by an application of sufficient force. For example, the virtual barrier may define a region having a higher viscosity than its surroundings, the resulting increased level of damping reducing the chance of an accidental slip or jerk by the operator. The virtual barrier may also be removable to allow intentional access to normally inhibited regions.
To provide the operator with a warning, the approaching of a virtual barrier may cause the triggering of an indicator such as an audible or visual alarm. The virtual barrier may be displayed as such on the display unit, superposed on the real visual image to allow the operator to "see" membranes or other delicate objects which would otherwise not readily be visible. While the embodiment has described the use of a television camera, any other suitable means may be employed, for example, ultrasound imaging, CAT imaging or X-rays.
Any desired combination of virtual and real images may be employed. The viewer may view only the virtual image, the real image being used only to ensure that the virtual image maps the real image. Alternatively, either image may be used, the virtual image being used to guide the probe to the region where it is to be used, switching to a visual display, possibly with superposed virtual barriers, when that region has been reached.
While the embodiment has been described in connection with an apparatus for performing medical operations, the invention may equally well be employed in any application where delicate objects or potentially hazardous situations exist. For example, where remote manipulations have to be made in the vicinity of electrically live circuits, a virtual barrier may be placed around items which are electrically live. Similarly a virtual barrier may be placed around optical surfaces which could be scratched by the accidental slipping of a manually controlled probe or tool. The teachings of the invention may also be applied to scenarios where remote control of instruments or probes is required, and where the accidental slipping of a tool or probe could cause damage or create a potentially hazardous situation.