US20200237449A1 - Redundant reciprocal tracking system - Google Patents

Abstract

The present invention relates to a redundant reciprocal tracking system composed of at least two trackers 10. A first tracker is able to sense partial or full pose data (orientation and position) of a second tracker in a first reference frame and the second tracker is able to sense partial or full pose data of the first tracker in a second reference frame. Pose data of first and second trackers are further transferred to a central processor 30, which is able to compute the transformation between first and second reference frame. Data generated by the trackers are such designed that they define an over-determined mathematical system (e.g. more than 6 degrees of freedom in a 3D setup). The over-determined information can be used to qualify and/or improve the transformation of the reference frame. In an embodiment of the invention, the tracking system is an optical one and the over-determined information defines an error metric used to check the validity of the transformation. Such setup could be used in surgical navigation system in order to reduce the risk of injury or death of the patient.

Description

CORRESPONDING APPLICATION
• The present application claims priority to US provisional application U.S. Ser. No. 62/128,585 filed on Mar. 5, 2015, the content of this earlier application being incorporated by reference in its entirety in the present application.
FIELD OF THE INVENTION
• The present invention relates generally to redundant reciprocal tracking systems and more specifically it relates to an optical tracking system where at least two trackers are sensing each other in order to compute a rigid transformation based on an over-determined system.
BACKGROUND OF THE INVENTION
• Optical tracking systems for tracking the pose (that is the position and orientation) of moving objects in a reference frame or system are known in the literature and prior art. These tracking devices usually use one or several camera sensors. For example, in U.S. Pat. No. 5,227,985 issued to Daniel F. De Menthon, and entitled “Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object” (the content of this patent being incorporated by reference in its entirety in the present application), the tracking system is composed of one camera sensor, and a pose estimation is realized after identifying four or more characteristic non-planar points (further called fiducials or sensed elements). These fiducials placed at known positions are defining a rigid body. In case of a setup of two or more cameras, triangulation is used to find the pose of a rigid body having three or more non-collinear identified fiducials. These fiducials may either be (a) reflective: they reflect light (e.g. reflective spheres or disks), (b) passive: image processing is used to detect specific patterns or (c) active: they transmit light (e.g. visible LEDs, Infrared LEDs, bulbs and more generally light sources). A typical active system is presented in U.S. Pat. No. 4,649,504 issued to Nicholas Krouglicof & al, and entitled “Optical position and orientation measurement techniques”, the content of this patent being incorporated by reference in its entirety in the present application.
• Redundancy can be integrated in conventional tracking systems by adding more fiducials on objects to track. A rigid registration error (e.g. “Least-Squares Fitting of Two 3-D Point Sets”, author Arun & al, 1987 IEEE, the content of this document being incorporated by reference in its entirety in the present application) enables to compute the pose between the theoretical and the measured 3D positions of the fiducials. With such a technique, it is possible to get an error metric related to the quality of the correspondences between the two sets of 3D points. This error give the possibility to provide a rejection threshold on the pose measurement. For example, partial occlusion of the fiducials, irregularities of the reflective material or even dirt or blood on the fiducial can be detected through this technique. The measurement can thus be partly or entirely rejected which is appreciable in application requiring a high level of safety.
• A complete characterization of rigid bodies in a stereo camera setup can be found in “Accuracy Assessment and Interpretation for Optical Tracking Systems”, chapter 4 and 5, authors Andrew D. Wiles & al, published in SPIE Medical Imaging, Proc. 5367, 2004, the content of this document being incorporated by reference in its entirety in the present application).
• Stereo cameras usually comprise two sensors with a known orientation and position, each of them basically being able to detect horizontal and vertical angles. When determining a 3D position of a point in space, the system is over-determined as a minimum of 3 DoF (Degrees of Freedom) are necessary and 4 DoF are actually used to compute the point in space. The fourth DoF is therefore redundant and can be used to evaluate if the system is decalibrated by using either (a) the distance between the centroid of the fiducials in the images and their respective left and right projections of the 3D point, (b) the distance between the 3D lines defined by the fiducials in the images and the optical centers, or (c) the distance of a centroid with respect to the epipolar line related to the centroid in the other image. A complete description of the different error metrics and triangulation techniques can be found either in the computer vision stereo systems literature or in “Triangulation”, authors Richard I. Hartley & al, published in Computer Vision and Image Understanding, Vol. 68, No. 2, pp. 146-157, 1997, the content of this document being incorporated by reference in its entirety in the present application.
• Other types of optical sensors can be used to determine 3D position of fiducials in space. For example in “spaceCoder: a Nanometric 3D Position Sensing Device”, author E. Grenet & al, published in CSEM Scientific & Technical Report 2011 (the content of this document being incorporated by reference in its entirety in the present application), the “spaceCoder” system comprises three parts: (a) a source of illumination face to (b) an image sensor at a given distance, and in-between (c) a transparent scale with a double marking (absolute and regular). The emitted light projects the shadow of the pattern onto the sensor. The high-resolution absolute position of the scale regarding the sensor is obtained combining the coarse absolute position (absolute code) with its fine relative position (regular pattern phase). This principle has been extended to 2-DOF, the pattern on the code-plate being composed of an absolute 2D binary code and a regular 2D pattern. The high robustness, precision and accuracy are essentially due to the fact that the information is distributed over the whole image: each pixel gives its contribution to the fine positioning. Given two 2-DOF spaceCoders placed at a known relative rigid transformation, it is possible to get the 3D position of the light source by means of triangulation. Other arrangement of the scale enable to triangulate directly on a unique 2D sensor and to output the 3D position of the light source. The 3-DOF principle of the spaceCoder technology can easily be extended to 6-DOF. For this, the spaceCoder sensor has to detect the 3D position of at least three independent light sources, which are fixed on the target and define its reference frame. In the present specification, the mention of “spaceCoder” or “spaceCoders” will refer to the “spaceCoder” described in this publication, unless otherwise stated.
• Reciprocal optical tracking systems are not commonly found in prior art literature. Such systems requires at least two sensors. Fiducials should be fixed on the sensors' assembly at a known positions. In the frame of the present application, we define a “tracker” as the assembly comprising at least one sensor and at least one fiducial, the position and orientation of all the elements being known. If a system comprises two trackers each of them able to detect 3D fiducials, a total of three fiducials on both trackers is theoretically sufficient to compute the pose between them. The lack of redundancy of such a setup is unfortunately not appropriate in applications requiring a high level of safety.
• Computer assisted surgical systems comprising a tracking system must have redundancy in order to reduce the risk of injury or death of the patient undergoing surgery. As existing systems are not reciprocal, they are using a classical optical tracking system combined with at least two rigid bodies. Rigid bodies are most often redundant and comprise four or more fiducials. One rigid body is fixed on the patient, other ones are fastened on surgical equipment and/or surgical instruments. In the workflow of a classical computer assisted surgery operation, it is necessary to put in correspondence the pre-operative dataset of the patient (e.g. MRI, CT series) with the actual patient. This phase is known as rigid registration. To proceed, the surgeon is usually palpating with a pointer three or more anatomical landmarks already identified and marked in the pre-operative dataset. A rigid body is fixed on the patient. The pointer is equipped with another rigid body, where the pointer tip position and direction is known with respect to the frame of reference of the rigid body. After registration, the computer is able to put in relation the reference frame of the pre-operative dataset/planning with the reference frame of the tracking system enabling to guide the surgeon during critical steps of the procedure (e.g. inserting a biopsy needle at the good position, placing an orthopaedic implant at the good position and orientation, etc.). Sometimes, the patient's rigid body has to be removed and/or exchanged. It may occur when (a) reflective spheres (such as markers or fiducials) are dirty or (b) a battery powering active fiducials is low. If the system is not designed to have a reproducible fixation, the rigid registration step should be redone after reinstalling a new rigid body.
• Another example using a reproducible fixation can be found in “Apex Robotic Technology: APEX Knee Surgical Navigation with the PRAXIM Robotic Cutting Guide”, copyright by OMNIlife science, 2011, Rev 07/11 at page 46: “For easier access, remove the “G” (rigid body) from the NanoBlock using the push-button on the reproducible fixation.” In this case the reproducible fixation is used for ergonomic reasons to improve accessibility of the patient.
• It is important to point out that not only technical aspects but also safety and workflow constraints should be taken into account when integrating a surgical optical tracking system. As such, reciprocity, redundancy and a reproducible fixation are three key aspects that this invention improves compared to state-of-the art systems.
BRIEF SUMMARY OF THE INVENTION
• The invention generally relates to a redundant reciprocal tracking system which includes at least two Trackers and a Central Processor.
• There has thus been outlined, rather broadly, some of the features of the invention in order that the detailed description thereof may be better understood, and in order that the present contribution to the art may be better appreciated. There are additional features of the invention that will be described hereinafter.
• In this respect, before describing embodiments of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction or to the arrangements of the components or to the embodiments set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of the description and should not be regarded as limiting. The present invention thus covers additional elements as described, for example means and technical elements equivalent to those described herein, all within the scope of the present invention.
• An aim and object of the present invention is to improve the known devices, methods a systems used in tracking applications.
• A further object of the present invention is to provide a redundant reciprocal tracking system for simultaneously enabling a first tracker to sense at least a second tracker in the reference frame of the first tracker, the second tracker being able to sense the first tracker in the reference frame of the second one. Both sensing data are transferred to a central processor (typically a computer with appropriate programs and memory capabilities) that will compute the pose (position and orientation) between the reference frames. Even if the minimal requirements are to solve a 6 DoF (Degrees of Freedom) problem—three translations (x,y,z) and three orientations (alpha, beta, gamma)—the overall tracking system is designed to be over-determined. This over-determination is used either to improve the quality of the measure or to evaluate its performances (i.e. pertinence of the measure, decalibration of the system).
• Another object of the present invention is to provide a redundant reciprocal tracking system with one or more reproducible fixations that can be used within a computer assisted surgery (CAS) system. The reproducible fixations enabe the replacement of the trackers during the intervention without requiring to recalibrate instruments or to reregister the patient, a clear improvement over known devices and systems.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used within a computer assisted surgery (CAS) system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used within a robotic assisted surgery system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used within an intelligent tool assisted surgery system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used within a navigated minimal invasive surgery system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used within a physical rehabilitation/therapy system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used in a motion capture system.
• Another object is of the present invention to provide a redundant reciprocal tracking system that can be used in a natural user interface system.
• Another object of the present invention is to provide a redundant reciprocal tracking system that can be used for general metrology/spatial measurement applications.
• Other objects and advantages of the present invention will become obvious to the reader and it is intended that these objects and advantages are within the scope of the present invention. To the accomplishment of the above and related objects, this invention may be embodied in the form illustrated in the accompanying drawings, attention being called to the fact, however, that the drawings are illustrative only, and that changes may be made in the specific construction illustrated and described within the scope of this application.
• In an embodiment, the invention concerns a tracking system and method comprising at least two trackers wherein a first tracker is able to sense positional information of a second tracker in a first reference frame and the second tracker is able to sense positional information of the first tracker in a second reference frame. The positional information of both first and second trackers enables then to calculate the pose of one tracker with respect to the reference frame of the other tracker, the computation being over-determined with more than 6 DoF. The tracking system further comprises means, such as communication means, for the trackers to send their positional information to either a central processor or to a distributed setup of processors that may be integrated in the trackers and means for the processor or setup of processors to combine the positional information of at least two of said trackers and to compute the rigid transformation of one reference frame with respect to the other. In this system the over determination of the mathematical system enables to estimate the quality of this transformation and/or to reduce the measurement noise and/or to improve the quality of the computed transformation.
• In an embodiment, the positional information is the pose of the trackers and the tracking system comprises means for the processor(s) to evaluate the pose of one of said trackers with respect to the other of said trackers, to combine these poses in a transformation and/or estimate the quality of this transformation and/or improve the transformation computation by taking into account a reciprocal measure between trackers as redundant data.
• In an embodiment of the system, each tracker of said at least two trackers may comprises at least an optical sensing element and three or more light sources that are at known position with respect to the optical sensing element, the optical sensing element of one tracker being able to identify and detect the position in 3D of at least three individual light sources emitted by another tracker. The known position may be a fixed position or a moving position which has for example a known pattern.
• In an embodiment of the tracking system the sensing element may be a 3-DoF spaceCoder and the sensed elements may be light sources.
• In an embodiment of the tracking system, the optical sensing element may comprise at least two 2-DoF spaceCoders, the sensed elements being light sources and the position and orientation between the spaceCoders is known which enable to compute the 3D position of the light sources by means of triangulation. The extra DoF when computing the 3D position may be used to evaluate the decalibration of the Tracker.
• In an embodiment of the tracking system, the light sources are preferably emitting in a spectrum not visible by the human eye (e.g. near infrared). Other spectrum (visible) is also possible in the scope of the present invention.
• In an embodiment of the tracking system the processing is distributed and done in the trackers which comprise an appropriate processor.
• In an embodiment of the tracking system, said trackers comprise a reproducible fixation localized at a known rigid transformation in the reference frame of the tracker such that said trackers may be attached to an object having a part complementary with said reproducible fixation, or released, or reattached or even exchanged while delivering the same pose data to the processor(s).
• In an embodiment of the tracking system, said at least one tracker is integrated or fixed on a medical tool and/or on a medical equipment.
• In an embodiment of the invention, the tracker is disposable and may be conditioned in a sterile package. It preferably comprises means to turn on the energy of the tracker automatically when required.
• In an embodiment of the tracking system, the means to turn on the tracker may be a sensor (e.g. a photo-receptor) which detects a change in the environment to wake-up the electronics when said tracker is taken out from a package, for example the sterile package.
• In an embodiment of the tracking system, the mechanism to turn on the tracker may be an electrical contact that is conductive when the tracker is attached to its fixation.
• In an embodiment, the tracking system comprises at least three trackers, wherein the first tracker senses the second tracker and vice-versa, the second tracker senses a third tracker and vice-versa, wherein the rigid transformation of the third tracker with respect to first reference frame can be indirectly determined by a combination of transformations including the poses of the second tracker.
• In an embodiment, the tracking system may be used in a medical, surgical, radio-interventional or rehabilitation context.
• In an embodiment of the tracking system, the redundancy of the system is used to reduce the risk of injury or death of the patient.
• In an embodiment, the present invention concerns a tracker as defined herein. Such trackers may be disposable, sterilisable or partly sterilisable.
• In an embodiment the invention, it relates to an apparatus using a tracking system and trackers as described and detailed herein.
• The apparatus may be a medical, surgical, radio-interventional or rehabilitation apparatus.
• The present invention also relates to a tracking method using the tracking system or trackers as described and detailed herein.
• The tracking method may be used in a medical, a surgical, a radio-interventional or a rehabilitation application, for example.
• The method according to the present invention is defined in the appended independent claims and the dependent appended method claims define further embodiments of the method according to the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS
• Various other objects, features and attendant advantages of the present invention will become fully appreciated as the same becomes better understood when considered in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the several views, and wherein:
• FIG. 1 is a perspective view of an embodiment of the present invention;
• FIG. 2 is a perspective view of another embodiment of the present invention;
• FIG. 3 is a perspective view of another embodiment of the present invention;
• FIG. 4 is a perspective view of another embodiment of the present invention;
• FIG. 5 is a perspective view of another embodiment of the present invention;
• FIG. 6 is a perspective view of another embodiment of the present invention and
• FIG. 7 is a block-diagram of an embodiment of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
• FIG. 1 presents as an embodiment of the invention an optical variation of a Redundant Reciprocal Tracking System, which is comprises twoTrackers10. They are facing each other in the figure. Four Light Sources—or SensedElements11—are rigidly fixed to each Trackers. Each Trackers comprises asensor sub-assembly20 fixed with respect to the Light Source(s). The Sensor sub-assembly comprises two 2DOptical Sensor Modules21. This Sensor sub-assembly is able to detect the 3D positions (with 4 DoF) of the distinct Light Sources located on the other Tracker via triangulation of the two Optical Sensors Modules. The over-determination of the 3D measure enabling to get information either on Tracker decalibration or measurement problem. If three or more Light Sources present on the other Tracker are detected, it is also possible to calculate the pose (position+orientation) of the other Tracker. The other Tracker can symmetrically or reciprocally perform the same type of measurement by mean of its own Sensor sub-assembly. Data of one and/or two Trackers are further transferred to theCentral Processor30 via a wired orwireless Link40. Note that this Central Processor could alternatively be part of a Tracker. The reciprocal pose measure can be used.
• FIG. 2 presents as an embodiment of the present invention: an optical version of the proposed Redundant Reciprocal Tracking System in a Computer-Assisted Surgery (CAS) setup. OneTracker10 is rigidly fixed on the skull of the patient. The other is located on a Biopsy Needle50 and operated by a Surgeon/Radiologist60. The setup further comprises aTablet30 and a Touch Screen31. Both Trackers are continuously sending their respective pose via awireless Link40. After a registration of the preoperative images (CT, MRI, etc) with respect to the patient and the calibration of the biopsy needle—the pose of the Tracker with respect to needle's tip and axis should be known—, it is possible to display the position and trajectory of theneedle33 within the pre-operative images and/or within acorresponding 3D representation32.
• FIG. 3 illustrates a similar setup as the one ofFIG. 2. This configuration further comprises a reproducible fixation81 on theTracker10 located on the Patient. The reproducible fixation counterpart82 is integrated in the mechanical structure which is rigidly fixed on thePatient70. The reproducible fixation is designed such that both parts can be clipped and released. When clipped, the fixation should be as tight as possible. Left drawing illustrates when the Tracker is detached from the reproducible fixation, right drawing when it is attached. The reference frame of a Tracker having a reproducible fixation is defined with respect to the fixation itself in order for the Tracker to be exchanged without requiring a re-registration or re-calibration. The reproducible fixation can use any means to reach this aim: corresponding shapes (as illustrated as an example inFIG. 3), markings, couplings, etc.
• FIG. 4 illustrates an alternative embodiment of the present invention. It shows a configuration with threeTrackers10 named (a), (b) and (c). Trackers (a) and (b) are ideally not seeing each other, Tracker (b) is facing both Trackers (a) and (c) and is able to sense them. Trackers (a) and (b) are able to sense each other and to compute a pose (POSE_ab). Trackers (b) and (c) are able to sense each other and to compute a pose (POSE_bc). Pose between Trackers (a) and (c) (POSE_ac) is a combination of POSE_ac and POSE_bc.
• FIG. 5 presents as an embodiment of the present invention: an optical version of the proposed Redundant Reciprocal Tracking System in a Robotic-Assisted Surgery setup. OneTracker10 is rigidly fixed on the skull of the patient. The other is located on the Robot90 or robot arm, the Biopsy Needle50 being at a known position with respect to the end-effector of the robot. Alternatively, the Robot50 can just replace the human60 ofFIG. 2 and directly hold the biopsy needle50. The setup further comprises aTablet30 and a Touch Screen31. Both Trackers are continuously sending their respective pose via awireless Link40. In this embodiment, thetracker10 on the robot could alternatively send poses or positional information through a wired link integrated in the robotic structure. After a registration of the preoperative images (CT, MRI, etc.) with respect to the patient and the calibration of the biopsy needle—the pose of the Tracker with respect to needle's tip and axis should be known—, it is possible to display the position and trajectory of theneedle33 within the pre-operative images and/or within acorresponding 3D representation32.
• FIG. 6 presents as an embodiment of the present invention: an optical version of the proposed Redundant Reciprocal Tracking System in an endoscopic setup. OneTracker10 is rigidly fixed on the skull of thepatient70. The other is located on the endoscope91 and operated by aSurgeon60. The orientation and position of the endoscope tip with respect to the tracker is known. Alternatively, the intrinsic parameters (focal distances, optical centre) of the endoscopic camera are known in order to provide a more precise augmentedreality37. Ideally, the tracker on the endoscope is fixed with a reproducible fixation (not shown here). The setup further comprises aTablet30 and a Touch Screen31. BothTrackers10 are continuously sending their respective pose via awireless Link40. In this case, the tracker on the endoscope91 could send poses through a wired link integrated in the endoscope. After a registration of the preoperative images (CT, MRI, etc.) with respect to thepatient70 and the calibration of the endoscope91, it is possible to display the position and trajectory of theendoscope34 within the pre-operative images and/or within acorresponding 3D representation32. An overlay of the endoscopic image is visible in the application. If trajectories and/or anatomical landmarks36 (e.g. a tumour) are segmented (and/or specified) in the preoperative data, it is possible to display them in theendoscopic image37 providing augmented reality to the surgeon.
• FIG. 7 illustrates a block diagram of an embodiment of the method according to the present invention with arrows showing the exchange of information between the elements of the system as described herein in reference to the several embodiments of the present invention. The trackers sense positional information (for example pose) and this information is used in the main application as described in detail in the present application, for example of a device using the tracking system of the present invention.
A. Overview
• Turning now descriptively to the drawings, in which similar reference characters denote similar elements throughout the several views, the figures illustrate at least two Trackers—both comprising at least one Sensor and Sensed Elements—and a Central Processor. The following description will discuss in more detailed manner the elements and features of the present invention by reference to the drawings.
B. Tracker
• ATracker10 comprises at least oneSensor20 and one or more SensedElements11.Trackers10 are intended and designed to be fixed on the objects to be tracked (e.g. surgical tools, surgical equipment, patient, etc.). SensedElements11 and the Sensor(s)20 are disposed at known position and orientation with respect to the frame of reference of theTracker10. The positioning of the elements can either be achieved by a precise manufacturing or by using an individual calibration method.Trackers10 are facing each other in a way that Sensor(s)20 from afirst Tracker10 can sense one or several SensedElements11 located on anotherTracker10 and vice-versa. The Sensor(s)20 are able to get positional information of the SensedElements11 with a number of DoF (Degrees of Freedom) that depends on theSensor20 type. In a theoretical system, a minimum of 6 DoF is required to compute both the position and orientation of afirst Tracker10 with respect to asecond Tracker10. In a configuration where theSensor20 is able to get the 3D position of a fiducial, a total of three fiducials placed on bothTrackers10 define this minimum requirement. Having more than 6 DoF is mandatory in several applications like surgical ones in order to provide a higher safety as for example has been described above. This is achieved by either adding redundancy in theSensors20 and/or in the SensedElements11. Redundancy enables to improve the quality of the measurements by mean of statistical methods. The system will moreover be more tolerant to noise as the influence of outliers will be diminished. Redundancy can also provide an error metric related to (a) the status of the Trackers10 (e.g. is a Tracker decalibrated) and/or (b) the status of the measurements (e.g. is a Sensedelement11 dirty, partly occluded, etc.). This error metric can further used by the application to decide if the measurements made are reliable or not.
• Typical Sensor20 and SensedElements11 could be optical ones. In this case, the SensedElements11 could be Light Sources (preferably near Infrared LEDs). In the optical configuration, theSensor20 may be an array of optical detectors in a combination that allows retrieving the distinct 3D positions of the photons generated by the Light Sources. A typical optical tracker would have an electronic circuitry comprising at least the sensor(s), a CPU, persistent memory, a communication link (e.g. wired cable, Bluetooth, WiFi, LiFi, or other equivalent communication technology etc.) and a power source (e.g. battery or an accumulator, rechargeable battery). At least theSensor20 and the SensedElements11 should be manufactured in a rigid housing or in a mobile configuration where their relative position and orientation is fixed and perfectly known during the acquisition. In a rigid housing, their exact position and orientation can be either determined once if the manufacturing is very precise or individually calibrated. Calibration as well as other information like the tool tip, the tool axis, etc. may be stored in a persistent memory (for example in a chip or other similar electronic element) in theTracker10 and sent to theCentral Processor30 for further pose processing of thedifferent Trackers10 used in the system.
• A Reproducible Fixation81 can be part or attached to theTracker10. The Reproducible Fixation81 is at a known and invariable position and orientation with respect to the frame of reference of theTracker10. The Reproducible Fixation81 attaches to the object to be tracked via a complementary part82. TheTracker10 is thus preferably designed such that if released, reattached or even exchanged with a different ornew Tracker10, the same measurement will be provided. The position of the tracker is thus unique and non-variable. This can be for example achieved by defining the frame of reference of all theTracker10 on the Reproductible Fixation81. Accordingly, a Tracker may be exchanged at any time without loss of the calibration.
• The link between theTrackers10 could be wired or wireless (e.g. Bluetooth, Zigbee, WiFi, LiFi, Optical Communication, IrDA, etc.). Extra information such as tracker id, serial numbers, calibration, temperature, button status and other sensor and meta-data can be exchanged through this link.
C. Sensor(s)
• In the present application, the term Sensor(s)20 refers to either a single Sensor or a subsystem comprising several Sensors where their respective orientation and position is known during the measure. They are preferably fixed but could also move.
• ASensor20 is fixed on theTracker10 at a known orientation and position with respect to its frame of reference. The role of theSensor20 is to get positional information of the SensedElements11 visible within the field of view (the notion of “visible” and “view” are used herein in their abstract definition, so that in a real application, a non-optical sensor could also be used if it has an equivalent effect) of theSensor20 and located on anotherTracker10. The distinct positions of the detected SensedElements11 of a set of twoTrackers10 are further used in theCentral Processor30 to determine at least the 6 DoF required to compute the rigid transformation between them.
• Typical Sensor20 and SensedElements11 could be optical ones. In this case, the SensedElements11 could be Light Sources (preferably near Infrared LEDs).
• An optical sensor could comprise one or more 1Doptical sensor modules21. In this case, the light detected by anoptical sensor module21 defines a single angular information (1 DoF). ASensor20 comprising three 1Doptical sensor modules21 oriented in at least two different directions provides the 3D position of a light source by mean of triangulation. In this setup, threelight sources11 are sufficient to compute a 6 DoF pose. An example of 1D optical sensor could be: (a) a camera linear sensor (e.g. CCD/CMOS), (b) a Position Sensitive Detector (PSD), (c) an optical system comprising either a combination of diaphragm with a cylindrical lens, (d) a mask pattern positioned in front of the camera sensor (e.g. 1 DoF spaceCoder as described herein).
• Another type of optical sensor could comprise one or more 2Doptical sensor modules21. In this case, the light detected by an optical sensor module defines two angular information (2 DoF). A Sensor comprising two 2D optical sensor modules provides the 3D position of a light source (by mean of triangulation) with 4 DoF. The extra DoF enables to detect the decalibration of the system using (a) the reprojection error, (b) distance to the epipolar lines or (c) 3D distance of lines defined by optical centers and the center of the fiducial in the image. These error metrics are described in the background section of the present application (see above). Examples of 2D optical sensor could be: (a) a CMOS/CCD camera array sensor with diaphragm and lens(es) (e.g. a conventional digital camera system) or (b) simply a mask pattern placed in front of the camera sensor (e.g. 2 DoF SpaceCoder as described herein).
• An alternate solution for a 2Doptical sensor module21 is to use a Dynamic Vision Sensor (DVS) instead of a CMOS/CCD. The main difference between a DVS and a normal CMOS camera is that the DVS output is a stream of events that encode changes in the brightness. Each event encodes the location of the change, whether there was a positive or negative change in brightness. A DVS has the advantage of enabling the Light Sources to simultaneously send data while being tracked. In this configuration, the Light Sources can all be turned on (without any sequential limitation) while sending identification data.
• Another type of optical sensor could comprise one 3Doptical sensor module21. A possible module could comprise (a) a CMOS/CCD array sensor, including an optical system enabling to perform triangulation on a single array sensor (e.g. 3 Dof spaceCoder as described herein) from CSEM, (b) a time of light based camera.
• Any combination of 1D, 2D and 3D optical sensor modules is of course possible within the scope of the present invention.
• Near infrared filter pass band filter may be used in combination with infrared LEDs to improve the robustness of a solution based on optical sensors.
• Other Sensors technologies are possible using other technologies like ultrasound, magnetism, or even light-field, etc.
• In case of light-field sensor, sensor enable to sense both light sources position and coming ray orientation. With this extra information (position and angles), triangulation and identification of fiducials is more efficient (time, quality, etc.).
D. Central Processor
• TheCentral Processor30 is gathering (raw, pre-processed, and/or processed) data from thedifferent Sensors20 located on theTrackers10 via a wired orwireless Link40. TheCentral Processor30 is further calculating the respective position and orientation of the Trackers based on the gathered data. Triangulation can for example be used to get the 3D position of the SensedElements11. A pose estimation algorithm (see Arun publication in the background section above) can be used to calculate the orientation and position of oneTracker10 with respect to anotherTracker10. Quality improvement of the measure as well as error metrics can be computed from the over-determined system. Both pose, error metrics and the necessary high-level tracking information are further transferred to the application for appropriate treatment.
• The system can furthermore process and/or reprocess positional data using other sensed or external parameters. Such parameters enable to define trustfulness of the measure provided by a given tracker. The measure could be pondered given its trustfulness (in real-time or during post-processing). Trustfulness could be based on the circular (resp. ellipsoid) estimation of spherical (res. Disk) fiducial, a bump sensor detecting shock that could have deformed the tracker, the use of the tracker outside of the optimal usage temperature, etc.
• Note that theCentral Processor30 may be integrated within aTracker10. The concept may de decentralized ifseveral Central Processors30 are alternatively present in the system. At minimum,Central Processors30 should be able to gather data from twodistinct Trackers10 and send the result to the application. ACentral Processor30 may be included in everyTracker10. Their respective data and/or calculations may be exchanged and the result directly transferred to the main application. In this case, all the metrological processing may be performed in theTrackers10.
• In this invention, the notion of acquiring positional information of reciprocal trackers at the “same time” is related to the sensor technology, the communication protocol and the final application. Ideally, it is simultaneous. Practically if the lag is very small, it could be neglected for the reciprocal measurement. If the lag is longer it should be compensated in the reciprocal measurement (e.g. by extrapolation of interpolation of positional information at the same timestamp).
• In case the SensedElements11 are acquired successively, the direct computation of the pose of a movingTracker10 is noisy because the acquisition of the entire SensedElements11 is not done at the same time. Noise can drastically be reduced by interpolation or extrapolation of the position of the SensedElements11 at a common timestamp.
• TheCentral Processor30 may be an electronic board with a processor, an embedded processor (SoC), a PC, a tablet, a Smartphone or directly integrated in the Tracker or any other equivalent equipment. For a tablet application, the Link is preferably Bluetooth or WiFi as non-limiting embodiments.
• The Link from theTrackers10 to theCentral Processor30 amy be wired or wireless (e.g. Bluetooth, Zigbee, WiFi, LiFi, via GSM, etc.). In case the Central Processor is directly in the Tracker a bus, i2c, or serial interface may be used).
E. Connections of Main Elements and Sub-Elements of the Invention
• SensedElements11 are emitting in a way they are not interfering each other or with theSensors20. The system is designed such as the SensedElements11 can be uniquely identified. This identification depends on the sensor technology. In a basic setup, the Sensed Elements are activated the one after the others to guarantee a unique identification. If the activation is simultaneous, one example to solve this problem is described in “Affordable Infrared-Optical Pose-Tracking for Virtual and Augmented Reality”, authors T. Pintaric & al, Proceedings of IEEE VR Workshop on Trends and Issues in Tracking for Virtual Environments, March 2007, the content of this document being incorporated by reference in its entirety in the present application. Depending on the Sensor technology used, emissions of the Sensed Elements may be simultaneous, sequential or random. The emission by the SensedElements11 and the reception by theSensors20,—both located on thedifferent Trackers10,—may also be locally or globally synchronized or random. Identification of the SensedElements11 is either implicit or explicit. In an explicit mode, theSensor Element20—, or an alternate communication means,—could generate an extra signal to identify itself. In case ofOptical Sensor Modules21, the identification of the emitting LEDs may be done by superposing a hi-frequency signal encoding its id between the acquisition phases of theSensors20. At the end, the positional information of the different SensedElements11 visible by theSensors20 are captured and identified. Partial or complete spatial 3D position can be computed on theTrackers10 or later on in theCentral Processor30. Data are transmitted via aLink40 to theCentral Processor30. TheCentral Processor30 aggregates the data. For example, if at least a total of three SensedElements11 are visible by two 3DoF Trackers10, it is possible to compute the pose of theTrackers10. If eachSensor20 detects three or more SensedElements11 of anotherTracker10, it is possible to compute the pose offirst Tracker10 in the frame of reference of thesecond Tracker10 and vice-versa. One pose is theoretically the inverse of the other. This redundancy can be used in a medical application to reduce the measurement risks. Finally, theCentral Processor30 provides the necessary high-level tracking information to the main application31.
• In a setup composed of three Trackers10 (a, b and c) as presented inFIG. 4, if Tracker a sense Tracker b (and vice-versa) and Tracker b sense Tracker c (and vice-versa), it is moreover possible to compute the pose of Tracker a in the frame of reference of Tracker c (POSE_ac) and vice-versa (POSE_ca). This will be a combination of POSE_ab and POSE_bc. This technique can be extended to any number ofTrackers10 interacting together and determine their relative poses. Such a system allows to handle a complete occultation between two trackers.
• In a setup composed of three Trackers10 (a, b and c), if Tracker a sense Tracker b (but b might not sense a) and Tracker b sense Tracker c (but c might not sense b) and Tracker c sense Tracker a (but a might not sense c), it is moreover possible to provide pose between all trackers with redundancy.
F. Alternative Embodiments of Invention
• In an embodiment of the invention, the Sensor(s)20 of aTracker10 comprise at least two 2 DoF spaceCoders as described herein. A 2 DoF spaceCoders is an Optical Sensor Module comprising a camera sensor and a reticule in front of it. The reticule is designed such that when a light source is in front of the sensor, it casts a shadow on the camera sensor through the reticule so that horizontal and vertical angular position of the light source can be retrieved. The SpaceCoders (as described above) are both placed at a known position and orientation on the reference frame of theTracker10. In an embodiment of this invention, this baseline is less than 40 mm, reasonably less than 30 mm, ideally less than 20 mm and optimally, less than 10 mm. The other geometrical parameters of the Optical Sensor Modules are well known, so that the 4 DoF position of alight source11 located on anotherTracker10 can be computed by mean of triangulation.Light sources11 are emitting sequentially. Identification of thesources11 and synchronization of the acquisition is realized with a higher speed optical communication that is not interfering with the acquisition of the SpaceCoders. Angular position and/or 3D positions of the LEDs (light sources11) are further transmitted to theCentral Processor30 via a wireless Link (e.g. Low Power Bluetooth). Pose calculation and optionally cross validation is done on theCentral Processor30. The extra DoF when extracting the position of aLED11 enable to estimate the decalibration of theTracker10 and/or the quality of the measurement. In an embodiment of the invention, at least threeLEDs11 are located on eachTracker10 allowing to compute the complete pose of aTracker10 and its reciprocal. Having both poses enables to easily and rapidly check the quality of the measurement.
• In an embodiment of the invention, theSensor20 is composed of at least one 3 DoF spaceCoder. A 3 DoF spaceCoder is an Optical Sensor Modules composed of a camera sensor and a reticule in front of it (see the above reference to the prior art publication to Grenet & al of CSEM). The reticule is designed such that when a light source is in front of the sensor, it casts two distinct shadows on the camera sensor. Instead of doing triangulation on two 2 DoF spaceCoders, triangulation is done on two distinct areas of the camera sensor where the shadow is cast. Other elements of this embodiment and processes are similar to the previous embodiment.
• In an embodiment of the invention, the proposed Tracking System is installed in a computer-assisted surgery setup (see for exampleFIG. 2). OneTracker10 is rigidly fixed on the skull of apatient70. The other is located on a medical device50 (for example a Biopsy Needle) and operated by a Surgeon/Radiologist60. The setup further comprises aTablet30 and a Touch Screen31. BothTrackers10 are continuously sending their respective poses to a Central Processor (Tablet30) via awireless Link40. After the correct registration of the preoperative images (e.g. CT, MRI, PET) with respect to thepatient70 and the calibration of the axis and tip of the biopsy needle50, it is possible to display the position and trajectory of theneedle33 within the pre-operative images or3D data32. Compared to existing surgical systems, this system offers a redundancy of the measure, less occlusion problems, improved ergonomics as well as a better overall accuracy.
• The redundancy of the system is also mandatory (a) to reduce noise and (b) to avoid wrong measurements that could cause either injuries or to sample tissues at inappropriate positions.
• In an embodiment of the invention, theTracker10 located on thepatient70 of the computer-assisted surgery embodiment is equipped with a Reproducible Fixation81,82. Removing and replacing theTracker10—for example if its battery is too low—after the registration process can easily be done and will not require to re-register the patient in the system.
• In an embodiment of the invention, theTracker10 located on the Biopsy Needle of the computer-assisted surgery embodiment as well as the Biopsy Needle50 (or any other instrument used according to the principles of the present invention) are disposable elements. They may be connected together in the sterile package. Needle tip and axis can coincide to the reference frame of theTracker10 so that a calibration of the Biopsy Needle50 is not necessary. The biopsy needle tip and orientation can alternatively be stored in the memory of theTracker10.
• In an embodiment of the invention, the proposed Tracking System is integrated in a computer/robotic assisted surgery setup (FIG. 5). In this configuration, one orseveral Trackers10 are located on the robot's end-effector and at least anotherTracker10 is fixed on thepatient70.
• In an embodiment of the invention, the proposed Tracking System is integrated in a minimal invasive surgery (and/or microscope) setup (FIG. 6). In this configuration, one orseveral Trackers10 are located on the endoscope/microscope and at least anotherTracker10 is fixed on thepatient70. Tracking information is further used to display information in the endoscope/microscope image like pre-operative planning, position of tumor, points of interest or other tracked instruments50.
• In an embodiment of the invention, the proposed Tracking System is integrated in a physical rehabilitation/therapy system, in a sport's training/learning system, or in any motion capture system. In this configuration, one ormore Trackers10 are fixed on the subject to track/analyze.Other Trackers10 can be integrated in other parts of the subject, in the room, either on a removable frame, or on a tripod or is following the person on a trolley (e.g. mobile robot, drone). The application records the location of the subject as well as his/her joints movements. These data are further analyzed in a dedicated motion capture application.
• In an embodiment of the invention, disposable Trackers can be conditioned in a sterile package. The battery can be pre-installed during the manufacturing stage and electronics is operating in a deep sleep mode. Once the sterile recipient is opened—during the intervention,—a sensor (e.g. a photo-receptor) detects a change in the environment and wake-up the electronics. It allows the system to conditioned the battery inside the Tracker and avoid using an extra power switch.
• In an embodiment of the invention, the disposable Trackers can be conditioned in a sterile package. The battery can be pre-installed during the manufacturing stage and electronics is off. Once the Tracker if attached to the Reproducible Fixation, a (conductive) contact or reed switch or is established between the battery and the electronic circuitry enabling to power it up. It allows to avoid using an extra power switch.
• What has been described and illustrated herein is a preferred embodiment of the invention along with some of its variations. Also, embodiments and features of different may be combined together according to circumstances and needs and they are not exclusive. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention in which all terms are meant in their broadest, reasonable sense unless otherwise indicated. Any headings utilized within the description are for convenience only and have no legal or limiting effect.

Claims (21)

1-32. (canceled)

33. A first tracker device, comprising:

at least one optical sensor module comprising at least one image sensor and at least one reticule disposed proximate the at least one image sensor;

memory comprising instructions stored thereon; and

at least one processor coupled to the memory and the at least one optical sensor module and configured to execute the stored instructions to:

generate positional information of a second tracker device in a first reference frame of the first tracker device based on a shadow cast by the at least one reticule on the at least one image sensor upon illumination of at least one sensed element of the second tracker device; and

outputting the positional information via at least one communication link to facilitate determination of one or more of a pose of the second tracker device with respect to the first reference frame or a rigid transformation of the first reference frame relative to a second reference frame of the second tracker device.

34. The first tracker device according toclaim 33, wherein the at least one processor is further configured to execute the stored instructions to output the positional information via the at least one communication link to one or more of:

a central processor; or

one or more third tracker devices to facilitate distributed processing of the positional information.

35. The first tracker device according toclaim 33, wherein the positional information comprises at least a horizontal angular position and a vertical angular position of the at least one sensed element of the second tracker device.

36. The first tracker device according toclaim 33, wherein the at least one sensed element of the second tracker device comprises at least three light sources and the at least one processor is further configured to execute the stored instructions to detect a position in three dimensions of the at least three light sources upon sequential illumination of the at least three light sources.

37. The first tracker device according toclaim 33, wherein the at least one processor is further configured to execute the stored instructions to generate the positional information based on a first shadow and a second shadow cast by the at least one reticule on a first portion and a second portion, respectively, of the at least one image sensor upon illumination of the at least one sensed element of the second tracker device.

38. The first tracker device according toclaim 33, wherein the at least one optical sensor module comprises two optical sensor modules and the first tracker device further comprises a reference frame on which the optical sensor modules are disposed at a known position and orientation.

39. The first tracker device according toclaim 33, further comprising at least three light sources, wherein the at least one processor is further configured to execute the stored instructions to sequentially illuminate each of the at least three light sources to facilitate generation of additional positional information of the first tracker device by the second tracker device.

40. A method for reciprocal optical tracking, the method comprising:

generating, by a first tracker device, positional information of a second tracker device in a first reference frame of the first tracker device based on a shadow cast by at least one reticule of at least one optical sensor module of the first tracker device on at least one image sensor of the at least one optical module upon illumination of at least one sensed element of the second tracker device; and

outputting, by the first tracker device, the positional information via at least one communication link to facilitate determination of one or more of a pose of the second tracker device with respect to the first reference frame or a rigid transformation of the first reference frame relative to a second reference frame of the second tracker device.

41. The method according toclaim 40, further comprising outputting, by the first tracker device, the positional information via the at least one communication link to one or more of:

a central processor; or

one or more third tracker devices to facilitate distributed processing of the positional information.

42. The method according toclaim 40, wherein the positional information comprises at least a horizontal angular position and a vertical angular position of the at least one sensed element of the second tracker device.

43. The method according toclaim 40, wherein the at least one sensed element of the second tracker device comprises at least three light sources and the method further comprises detecting, by the first tracker device, a position in three dimensions of the at least three light sources upon sequential illumination of the at least three light sources.

44. The method according toclaim 40, further comprising generating, by the first tracker device, the positional information based on a first shadow and a second shadow cast by the at least one reticule on a first portion and a second portion, respectively, of the at least one image sensor upon illumination of the at least one sensed element of the second tracker device.

45. The method according toclaim 40, wherein the at least one optical sensor module comprises two optical sensor modules and the first tracker device further comprises a reference frame on which the optical sensor modules are disposed at a known position and orientation.

46. The method according toclaim 40, further comprising sequentially illuminating, by the first tracker device, each of at least three light sources to facilitate generation of additional positional information of the first tracker device by the second tracker device.

47. A non-transitory computer readable medium having stored thereon instructions for reciprocal optical tracking and comprising executable code that, when executed by at least one processor, causes the at least one processor to:

generate positional information of a second tracker device in a first reference frame of a first tracker device based on a shadow cast by at least one reticule of at least one optical sensor module of the first tracker device on at least one image sensor of the at least one optical module upon illumination of at least one sensed element of the second tracker device; and

output the positional information via at least one communication link to facilitate determination of one or more of a pose of the second tracker device with respect to the first reference frame or a rigid transformation of the first reference frame relative to a second reference frame of the second tracker device.

48. The non-transitory computer readable medium ofclaim 47, wherein the executable code, when executed by the at least one processor, further causes the at least one processor to output the positional information via the at least one communication link to one or more of:

a central processor; or

one or more third tracker devices to facilitate distributed processing of the positional information.

49. The non-transitory computer readable medium ofclaim 47, wherein the positional information comprises at least a horizontal angular position and a vertical angular position of the at least one sensed element of the second tracker device.

50. The non-transitory computer readable medium ofclaim 47, wherein the at least one sensed element of the second tracker device comprises at least three light sources and the executable code, when executed by the at least one processor, further causes the at least one processor to detect a position in three dimensions of the at least three light sources upon sequential illumination of the at least three light sources.

51. The non-transitory computer readable medium ofclaim 47, wherein the executable code, when executed by the processors, further causes the at least one processor to generate the positional information based on a first shadow and a second shadow cast by the at least one reticule on a first portion and a second portion, respectively, of the at least one image sensor upon illumination of the at least one sensed element of the second tracker device.

52. The non-transitory computer readable medium ofclaim 47, wherein the executable code, when executed by the processors, further causes the at least one processor to sequentially illuminate each of at least three light sources to facilitate generation of additional positional information of the first tracker device by the second tracker device.

Images