CANNULA ASSEMBLY FOR ENHANCING CRITICAL ANATOMY
VISUALIZATION DURING A SURGICAL PROCEDURE
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional Application Serial No. 63/489,478, filed March 10, 2023, the complete disclosure of which is incorporated herein by reference for all purposes.
BACKGROUND
[0002] Minimally invasive surgery involves making small incisions into a body of a patient to insert surgical tools. For example, a surgeon may perform a laparoscopic procedure using multiple cannulas inserted through individual incisions that accommodate various surgical tools, including illumination devices and imaging devices. To accomplish the insertion, cannula assemblies may be used to puncture the body cavity. A cannula assembly often includes an obturator and a cannula. An obturator is a guide placed inside a cannula, the obturator having either a sharp tip (e.g., a pointed cutting blade) or a blunt tip for creating an incision or opening in the patient for the cannula to pass through. After the obturator and cannula are inserted, the obturator is removed, leaving the cannula in place for use in inserting the surgical tools into the surgical space within a patient. Typically, in addition to cannulas forming individual incisions for surgical tools, an individual incision may also be made through the patient by a cannula that is thereafter dedicated to holding an illumination and/or imaging device, e.g., a traditional endoscope or laparoscope. A surgical tool combining a cannula and an imaging device in a single unit is disclosed, for example, in U.S. Patent No. 8,834,358, the disclosure of which is herein incorporated by reference in its entirety. SUMMARY
[0003] The following presents a simplified summary of the claimed subject matter in order to provide a basic understanding of some aspects of the claimed subject matter. This summary is not an extensive overview of the claimed subject matter. It is intended to neither identify key or critical elements of the claimed subject matter nor delineate the scope of the claimed subject matter. Its sole purpose is to present some concepts of the claimed subject matter in a simplified form as a prelude to the more detailed description that is presented later.
[0004] Described hereinbelow, in accordance with various embodiments thereof, is a cannula assembly that comprises a cannula tube having a longitudinal axis, a proximal end portion and a distal end portion configured for insertion into a patient. The cannula assembly also includes a housing coupled to the cannula tube between the proximal and distal ends of the cannula tube so as to be positioned within the patient when the distal end of the cannula tube is inserted into the patient. The housing is movable relative to the cannula tube between a closed position and an open position. The housing includes a non-white light source and an image sensor configured to provide enhanced image data based on the non-white light source when the housing is in the open position within the patient. In various embodiments, the cannula assembly may also comprise a processor configured to receive the enhanced image data from the image sensor and to overlay the enhanced image data with other image data on a display device. The other image data with which the enhanced image data is overlaid may include image data received by the processor from a white-light image sensor. This other image data received by the processor from a white-light image sensor may be received via a white light image sensor that is positioned on either the same cannula tube (e.g., a separate sensor housing mounted or coupled to the same cannula tube), a second cannula assembly (e.g., a cannula assembly configured similarly to the first cannula assembly but having a white-light source rather than a non-white light source), or a secondary imaging unit (e.g., a traditional endoscope or laparoscope extending through a traditional surgical cannula).
[0005] In various embodiments, the cannula assembly may also include at least one spatial data generating component for providing data related to the relative position of the cannula assembly. The spatial data may be used by the processor to more accurately overlay the enhanced image data from the image sensor and the other image data on the display device.
[0006] In some embodiments, the cannula assembly may be configured such that the light source is configured to transmit blue light, which may be employed as an agent-free contrast enhancement of a patient’s microvasculature. Additionally or alternatively, the blue light may be employed for tumor detection in a patient’s bladder, for example in a blue light cystoscopy procedure that finds bladder cancer tumors. In other embodiments, the light source may be configured to transmit light at a wavelength outside of the spectrum visible to the human eye, such as, for example, the light source may be configured to transmit infrared or ultraviolet light, which may be employed for enhanced visualization of tumors, lymphatics and/or a patient’s nerves. In embodiments, the non-white light source may be configured to generate light at multiple wavelengths. In such an embodiment, a processor may be configured to receive from the image sensor the image data related to each of the multiple wavelengths and to perform an operation on the image data.
[0007] In these manners, in accordance with various embodiments thereof, there is provided enhanced visualization of critical patient anatomy during a surgical procedure and, advantageously, enables the same via a cannula assembly having the non-white light source coupled or mounted thereon so as to potentially reduce the number of cannulas typically employed during a surgical procedure. [0008] In further embodiments, the cannula assembly may be configured such that the housing is rotatable about an axis transverse to the longitudinal axis of the cannula tube. In this way, when in the open position, the housing may be positioned more laterally relative to the longitudinal axis of the cannula tube as compared to the closed position. Additionally, the cannula tube may have an internal lumen extending from the proximal end portion to the distal end portion, and the cannula assembly may also include an obturator configured to be removably inserted into the internal lumen of the cannula tube. The distal end of the obturator may extend beyond the distal end portion of the cannula tube, and the distal end of the obturator may be configured to penetrate a patient’ s abdominal wall to help position the distal end portion of the cannula tube inside the patient. The obturator may be configured to be removed from the internal lumen of the cannula tube when the distal end portion of the cannula tube is positioned inside the patient, thereby providing a lumen through which additional surgical instruments, e.g., surgical stapler, etc., may be inserted into the surgical site.
[0009] In still further embodiments, there is provided a surgical system that comprises a cannula assembly. The cannula assembly may include a cannula tube having a longitudinal axis, a proximal end portion and a distal end portion configured for insertion into a patient. The cannula assembly may also include a housing coupled to the cannula tube between the proximal and distal ends of the cannula tube so as to be positioned within the patient when the distal end of the cannula tube is inserted into the patient. The housing may be movable relative to the cannula tube between a closed position and an open position. The housing may include a non-white light source and an image sensor configured to provide enhanced image data based on the non-white light source when the housing is in the open position within the patient. The system may also include a display device for viewing by a user. The system may further include a processor configured to receive the enhanced image data from the image sensor and to overlay the enhanced image data with other image data on the display device.
[0010] In some embodiments, the surgical system is configured such that the other image data with which the enhanced image data is overlaid may include image data received by the processor from a white-light image sensor. The white light image sensor may be positioned on one of the cannula tube, a second cannula assembly, and a secondary imaging unit, as described above.
[0011] In further embodiments, the cannula assembly of this system may also include at least one spatial data generating component for providing data related to the relative position of the cannula assembly. This spatial data may be used by the processor to overlay the enhanced image data from the image sensor and the other image data on the display device. As mentioned above, the light source may be configured to transmit blue light, or the light source may be configured to transmit light at a wavelength outside of the spectrum visible to the human eye, for example infrared or ultraviolet light.
[0012] In still further embodiments, the cannula assembly may be configured such that the housing is rotatable about an axis transverse to the longitudinal axis of the cannula tube. In this arrangement, the cannula assembly may be configured such that, when in the open position, the housing is positioned laterally relative to the longitudinal axis of the cannula tube. Still further, the cannula assembly may be configured such that the cannula tube has an internal lumen extending from the proximal end portion to the distal end portion. The obturator may be configured to be removably inserted into the internal lumen of the cannula tube such that a distal end of the obturator extends beyond the distal end portion of the cannula tube. The distal end of the obturator may be configured to penetrate a patient’s abdominal wall so as to help position the distal end portion of the cannula tube inside the patient. The obturator may be configured to be removed from the internal lumen of the cannula tube when the distal end portion of the cannula tube is positioned inside the patient.
[0013] The non-white light source may be configured to generate light at multiple wavelengths. In such an embodiment, the cannula assembly may also include a processor configured to receive from the image sensor the image data related to each of the multiple wavelengths and to perform an operation on the image data.
[0014] In still further embodiments, there is provided a surgical system that includes first and second cannula assemblies. Each of the first and second cannula assemblies may include a cannula tube having a longitudinal axis, a proximal end portion and a distal end portion configured for insertion into a patient. Each of the first and second cannula assemblies may also include a housing coupled to the cannula tube between the proximal and distal ends of the cannula tube so as to be positioned within the patient when the distal end of the cannula tube is inserted into the patient, the housing movable relative to the cannula tube between a closed position and an open position. The housing of each of the first and second cannula assemblies may include a light source and an image sensor configured to provide image data when the housing is in the open position within the patient. The light source of the first and second cannula assemblies may be configured to generate light at different wavelengths relative to each other, wherein at least one of the wavelengths relates to non-white light. The surgical system may also include a display device for viewing by a user, and a processor configured to receive the image data from the image sensors and to perform an operation on the image data. The operation may include overlaying the images on the display device.
[0015] In various embodiments, the non-white light source may be configured to transmit blue light. Additionally or alternatively, the non-white light source may be configured to transmit light at a wavelength outside of the spectrum visible to the human eye. For example, the non-white light source may be configured to transmit infrared or ultraviolet light.
[0016] Among various other advantages provided by certain embodiments as will be evident from the Detailed Description below, there may also be the benefit that fewer punctures through a patient, e.g., through an abdominal wall or other bodily surface, are made during a surgical procedure. As set forth above, a surgeon typically performs a laparoscopic procedure using multiple cannulas inserted through individual incisions, wherein at least one such cannula and incision is occupied by an illumination/imaging device, such as a traditional endoscope and/or laparoscope. According to various embodiments thereof, there is provided a cannula assembly and/or system therefor that eliminates the need for this separate puncture by a cannula assembly for an endoscope/laparoscope, since it provides, in certain embodiments, a cannula assembly which provides both an illumination/imaging device (e.g., mounted or coupled to the cannula tube) and an internal lumen through which a separate surgical tool (e.g., a surgical stapler, etc.) may be inserted. The reduction of at least puncture during a surgical procedure, as may be enabled in certain embodiments, may improve the safety of the surgical procedure by avoiding potential complications, reducing pain and/or speeding the patient’s recovery.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] FIG. 1 shows a cannula assembly, in accordance with various embodiments.
[0018] FIG. 2 shows a system including two cannula assemblies, in accordance with various embodiments.
[0019] FIG. 3 is a system block diagram that illustrates two cannula assemblies employed in a surgical procedure, in accordance with various embodiments.
[0020] FIG. 4 shows a system block diagram that illustrates a cannula assembly and a secondary imaging unit, in accordance with various embodiments. [0021] FIG. 5 shows a block diagram illustrating an example of a device controller in accordance with various embodiments.
[0022] FIG. 6 shows a block diagram illustrating an example of an imaging controller for a system in accordance with aspects.
DETAILED DESCRIPTION
[0023] Generally, described hereinbelow are imaging systems and, more particularly, endoscopic imaging systems. Systems and methods in accordance with various embodiments provide a cannula assembly that includes a non-white light source and an imaging sensor configured to provide enhanced image data based on the non-white light source. In some embodiments, the enhanced image data based on the non-white light source that is provided by the cannula assembly’s imaging sensor is overlaid with image data from a second imaging unit - which may itself be either a second image sensor on the same cannula assembly, a second cannula assembly having an image sensor, or may be a separate imaging unit such as a traditional endoscope - into an enhanced display image.
[0024] In those systems and methods in accordance with various embodiments that overlay image streams from separate cannula assemblies, e.g., at least one from a cannula assembly having a non-white light source and an image sensor, the combination of those image streams can employ relative spatial information of the cannula assemblies to enable the image data streams to be accurately overlaid relative to each other. In some such embodiments, the spatial information can include, for example, distance, angle, and rotation of the cannula assemblies relative to one another. In some embodiments, the overlaid image stream can be, for example, a three-dimensional (“3D”) stereoscopic view. Also, in some embodiments, the system and methods can provide additional functionality and advantages, as described for example in Applicant’s co-pending U.S. Provisional Patent Application Serial No. 63/112,398, the disclosure of which is incorporated by reference herein in its entirety, such as functionality whereby the overlaid image stream can have a wider field of view than individually provided by a single one of either cannula assembly, the system can identify and characterize structures, such as surgical tools or tissues, in the images, from the overlaid image stream, and/or the system can remove obstructions between the respective views of the cannula assemblies from the overlaid image stream.
[0025] Reference will now be made in detail to specific embodiments illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that embodiments may be practiced without these specific details. In other instances, known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
[0026] FIG. 1 illustrates one example embodiment. According to this example embodiment, there is provided a cannula assembly 111 A. The cannula assembly 111 A includes a cannula tube 209 having a longitudinal axis 209a, a proximal end portion 209b, and a distal end portion 209c configured for insertion into a patient. The cannula tube 209 has an internal lumen (not visible in this view) extending from the proximal end portion 209b to the distal end portion 209c. The cannula assembly 111 A also includes a sensor housing 217 coupled to the cannula tube 209 between the proximal and distal ends 209b, 209c of the cannula tube 209 so as to be positioned within the patient when the distal end portion 209c of the cannula tube 209 is inserted into the patient. In this embodiment, the sensor housing 217 is movable relative to the cannula tube 209 between a closed position and an open position. The sensor housing 217 includes a non-white light source 235A and an image sensor 231 A configured to provide enhanced image data based on the non-white light source 235 A when the sensor housing 217 is in the open position within the patient. Additional details of various embodiments of the light source 235 A (and other lights sources, such as the light source 235B shown and described in FIG. 2 and elsewhere) are described in greater detail below.
[0027] In still further embodiments, the cannula assembly 111 A may also include a processor or device controller 201 that is configured to receive the enhanced image data from the image sensor 231 A and to overlay the enhanced image data with other image data on a separate display device. Advantageously, the cannula assembly 111 A is configured such that the sensor housing 217 is rotatable about an axis (not shown in FIG. 1) that is transverse to the longitudinal axis 209a of the cannula tube 209. In this way, when the sensor housing 217 is in the above- mentioned open position, the sensor housing 217 is moved to a position that is more lateral relative to the longitudinal axis 209a of the cannula tube 209 as compared to the position of the sensor housing 217 when in the closed position. In further embodiments, the other image data onto which the enhanced image data is overlaid includes image data received by the processor 201 from a white-light image sensor (shown and described below in connection with FIG. 2). [0028] Although not shown herein, it should be understood by those skilled in the art that the cannula assembly 111A (and other cannula assemblies shown and described herein) may include other components and features in addition to those described herein. For example, any of the herein-described cannula assemblies 111 A, 11 IB may include sealing components, such as an instrument seal for sealing around an instrument inserted therethrough, a zero seal for sealing the cannula assembly in the absence of any instrument inserted therethrough, and/or any number of different ports, e.g., insufflation or irrigation ports, for the introduction of various gases or liquids into the surgical site.
[0029] Referring again to the light source 235 A, in some embodiments, the cannula assembly 111A is configured such that the non-white light source 235A is configured to transmit blue light, for example in a blue light cystoscopy procedure. A blue light cystoscopy procedure is a surgical procedure that finds bladder cancer tumors by using a combination of ultraviolet and white light, as well as an imaging dye such as hexaminolevulinate HC1 (also known as “Cysview™”). The Cysview makes cancer cells glow bright pink under UV light while healthy tissues remain blue.
[0030] Other surgical procedures employing blue light are also contemplated, e.g., blue light imaging of colon polyps, agent-free contrast enhancement of microvasculature and others. More specifically, blue light imaging is particularly useful to enhance visibility of shallow vessels and other submucosal structures. Like the blue light cystoscopy procedure mentioned above, it is well-suited for tumor detection, since it may enable unusually vascularized, or regions with unusual presence of microvesicles, to be recognized by a surgeon as indicative of cancerous growth. Example embodiments of cancerous tissue types and other growths for which blue light could be employed include, for example, neoplastic lesions (including surface neoplasia and Barrett’s neoplasia), sessile serrated lesions, low-grade dysplasia colon polyps, diminutive colon polyps, hyperplastic colon polyps, cancer cells in the bladder, intestinal metaplasia, identification of atrophic gastritis, etc.
[0031] In other embodiments, the light source 235 A is configured to transmit light at a wavelength outside of the spectrum visible to the human eye, such as for example, infrared or ultraviolet light. For example, infrared light can be employed for cancer detection, and nearinfrared (NIR) fluorescence imaging is a technique that can be employed to visualize tumors, vital structures, lymphatic channels, and lymph nodes. Near-infrared imaging could also be employed, in various embodiments, for perfusion assessment, e.g., during anastomosis, to determine the degree to which blood passes into tissues having been stapled or otherwise sealed, fastened, etc. [0032] Although the light source 235 A is described above as providing a single wavelength, in other embodiments, the light source may employ multiple wavelengths, also referred to as multispectral or hyperspectral imaging. In some embodiments, these multiple wavelengths may be generated by a single light source, such as the light source 235 A shown in FIG. 1 by, e.g., employing different wavelengths at different intervals of time, e.g., alternatingly. Incorporating multiple wavelengths, some of which lie in the near-infrared spectrum, can provide additional sensing capabilities.
[0033] For example, by employing multispectral or hyperspectral imaging, the cannula assembly 111A (e.g., via its processor 201 as described below, or by an imaging processor, such as imaging processor 105 shown and described below in connection with FIGS. 3 or 4) can algorithmically compare an image generated using a first wavelength to an image generated using a second wavelength. For example, in an embodiment, the light source 235 A may alternatingly generate data using a first wavelength corresponding to oxygenated blood and then generate data relating to a second wavelength corresponding to deoxygenated blood so as to perform an operation therebetween, e.g., compare the data at the two different wavelengths. Such an operation could be an operation that thereby establishes a characteristic related thereto. For example, in the embodiment described above related to oxygenated and deoxygenated blood, such an operation could establish the rate of flow of oxygenated blood (e.g., to enable the perfusion assessment mentioned above).
[0034] Alternatively, in still another embodiment, a first wavelength may be employed that is strongly absorbed by bile present in the gall bladder, while a second wavelength may be employed that is not strongly absorbed by bile present in the gall bladder. In this embodiment, image data that is generated using the first and second wavelengths may be employed in an operation, e.g., subtracting the first image from the second image data, to, e.g., highlight the location of the gall bladder, thereby potentially improving a surgeon’s ability to navigate. In still further embodiments, lights sources employing more than two different wavelengths may be provided, and the device controller 201 and/or the imaging processor 105 may be configured to employ multivariate analysis or deep learning techniques to differentiate cancerous tissue from healthy tissue and/or to characterize multiple organs or vital structures in the body.
[0035] FIG. 2 shows a system illustrating another example embodiment. In this embodiment, there is shown a system in which there are two cannula assemblies 111A, 11 IB. Although FIG. 2 illustrates two such cannula assemblies, it should be understood that certain advantages may be obtained with a single such cannula assembly, as shown for example in FIG. 1. This embodiment having two cannula assemblies will have additional advantages as shown and described below.
[0036] In the embodiment shown in FIG. 2, the cannula assemblies 111 A, 11 IB include a housing 200, a device controller 201, a body 203, an actuator handle 205, a cannula tube 209, an obturator 211, and a sensor housing 217. In the embodiment shown, the cannula assemblies 111A, 11 IB may also include spatial data components, in this case antennas 221 A, 221B, 221C. The cannula tube 209, the obturator 211, and the sensor housing 217 of the individual cannula assemblies 111A, 11 IB can be inserted into the body of a patient (e.g., patient 117) and positioned relative to each other, e.g., such as at an angle 137 with respect to each other, so as to provide differing fields-of-view from the sensor housing 217.
[0037] The device controller 201 can be one or more devices that process signals and data to generate respective image streams 127A, 127B (see also FIG. 3) and, in this embodiment, spatial information 129A, 129B (see also FIG. 3) of the cannula assemblies 111A, 11 IB. In some embodiments, the device controller 201 can determine the spatial information 129 A, 129B by processing data from spatial sensors (e.g., accelerometers) to determine the relative position, angle, and rotation of the cannula assemblies 111A, 11 IB. In some embodiments, the device controller 201 can also determine the spatial information 129 A, 129B by processing range information received from sensors (e.g., image sensor 231 and LiDAR device 233) in the sensor housing 217. Additionally, in some embodiments, the device controller 201 can process the spatial information 129A, 129B by processing signals received via the antennas 221 A, 221B, 221C to determine relative distances of the cannula assemblies 111A, 11 IB. It is understood that, in some embodiments, less than all, e.g., only one or none, of the cannula assemblies 111A, 11 IB provides spatial information 129. The spatial information, as shown and described herein, is advantageous to ensure that image streams are accurately overlaid relative to each other.
[0038] The cannula tube 209 may be formed of a variety of cross-sectional shapes. For example, the cannulas 209 can have a generally round or cylindrical, ellipsoidal, triangular, square, rectangular, and D-shaped (in which one side is flat). In some embodiments, the cannula tube 209 includes an internal lumen 202 into which the obturator 211 is inserted. The obturator 211 can be retractable and/or removable from the cannula tube 209. In some embodiments, the obturator 211 is made of solid, non-transparent material. In another embodiment, all or parts of the obturator 21 lare made of optically transparent or transmissive material such that the obturator 211 does not obstruct the view through the camera (discussed below). The obturator 211 may have a tip shape that is configured to penetrate, either via incision or via insertion between tissue planes, through the abdominal wall of the patient.
[0039] The sensor housing 217 can be integral with the cannula tube 209 or it may be formed as a separate component that is coupled to the cannula tube 209. In either case, the sensor housing 217 can be disposed on or coupled to the cannula tube 209 at a position proximal to the distalmost end of the cannula tube 209 such that it is positioned within the patient’s body when the distal end portion of the cannula tube 209 has been inserted into the patient. In some embodiments, the sensor housing 217 can be actuated by the actuator handle 205 to open, for example, after inserted into the patient’s 117 body cavity. The sensor housing 217 can reside along cannula tube 209 in the distal direction such that it is positioned within the body cavity of a patient (e.g., patient 117) during a surgical procedure. At the same time, sensor housing 217 can be positioned proximal to the distal end such that it does not interfere with the insertion of the distal end of the cannula tube 209 as it is inserted into a patient (e.g., patient 117). In addition, the sensor housing 217 can be positioned proximally from distal end to protect the electronic components therein as the distal end is inserted into the patient.
[0040] In some embodiments, the sensor housings 217 can include one or more image sensors 231 and a light source 235. In the embodiment shown, the sensor housing 217 of cannula assembly 111 A includes a non-white light source 235 A, while the sensor housing 217 of cannula assembly 11 IB includes a white light source 235B. The non-white light source 235A is configured to provide, via the image sensor 231 A, enhanced image data based on the non-white light source 235 A, while in some embodiments, the white light source 235B is configured to provide, via the image sensors 231, standard image data based on the non-white light source 235B.
[0041] In the embodiment shown, wherein there are two cannula assemblies 111A, 11 IB, one having a non-white light source 235 A and one having a white light source 235B, the white light source 235B can be dimmable light-emitting device, such as a LED, a halogen bulb, an incandescent bulb, or other suitable light emitter. Regardless, the light source 235B is configured to provide, via its image sensor 23 IB, standard image data based on the white light source 235B. The standard image data based on white light source 235B can be overlaid onto the enhanced image data based on the non-white light source 235 A, or alternatively, the enhanced image data based on the non-white light source 235 A can be overlaid onto the standard image data based on white light source 235B, so as to provide an overlaid image stream, as will be described more fully below. [0042] Generally, the image sensors 231 A, 23 IB can be devices configured to detect light reflected from the light source 235 and output an image signal. The image sensors 231 A, 23 IB can be, for example, a charged coupled device (“CCD”) or other suitable imaging sensor. In some embodiments, the image sensors 231 A, 23 IB includes at least two lenses providing stereo imaging. In some embodiments, the image sensors 231 A, 23 IB can be an omnidirectional camera.
[0043] In some embodiments, where spatial information is employed, the sensor housing 217 can include a LiDAR device 233. The LiDAR device 233 can include one or more devices that illuminate a region with light beams, such as lasers, and determine distance by measuring reflected light with a photosensor. The distance can be determined based a time difference between the transmission of the beam and detection of backscattered light. For example, using the LiDAR device 233, the device controller 201 can determine spatial information 129 by sensing the relative distance and rotation of the cannulas 209 or the sensor housing 217 inside a body cavity.
[0044] Additionally, the antennas 221A, 221B, 221C can be disposed along the long axis of the cannula assemblies 111A, 11 IB. In some embodiments, the antennas 221 A, 221B, 221C can be placed in a substantially straight line on one or more sides of the cannula assemblies 111 A, 11 IB. For example, two or more lines of the antennas 221 A, 221B, 221C can be located on opposing sides of the housing 203 and the cannula tube 209. Although FIG. 2 shows a single line of the antennas 221 A, 221B, 221C on one side of the cannula assemblies 111A, 11 IB, it is understood that the additional lines of the antennas 221 A, 221B, 221C can be placed in opposing halves, thirds, or quadrants of the cannula assemblies 111 A, 11 IB.
[0045] As illustrated in FIG. 2, in some embodiments, the device controllers 201 can transmit a ranging signal 223. In some embodiments, the location signals are ultra-wideband (“UWB”) radio signal usable to determine a distance between the cannula assemblies 111A, 11 IB less than or equal to 1 centimeter based on signal phase and amplitude of the radio signals, as described in IEEE 802.15.4Z. The device controller 201 can determine the distances between the cannula assemblies 111 A, 11 IB based on the different arrival times of the ranging signals 223 A and 223B at their respective antennas 221 A, 221B, 221C. For example, referring to FIG. 4, the ranging signal 223 A emitted by cannula assembly 111A can be received by cannula assembly 11 IB at antenna 221C and an amount of time (T) after arriving at antenna 22 IB. By making a comparison of the varying times of arrival of the ranging signal 223 A at two or more of the antennas 221 A, 221B, 221C, the device controller 201 of cannula assembly 11 IB can determine its distance and angle from cannula assembly 111A. It is understood that the transmitters can be placed at various suitable locations within the cannula assemblies 111A, 11 IB. For example, in some embodiment, the transmitters can be located in the cannulas 209 or in the sensor housings 217.
[0046] As set forth above, in various embodiments, the light sources may employ multiple wavelengths, also referred to as multispectral or hyperspectral imaging. In the embodiment described above in connection with FIG. 1, these multiple wavelengths may be generated by a single light source, such as the light source 235 A shown in FIG. 1. However, in still other embodiments, these multiple wavelengths may be generated by more than one light source, such as the light sources 235 A, 235B shown in FIG. 2, so as to provide additional sensing capabilities.
[0047] For example, by employing multispectral or hyperspectral imaging, the cannula assemblies 111A, 11 IB (e.g., via their respective processors 201 and/or by the imaging processor 105) can algorithmically compare an image generated using a first wavelength to an image generated using a second wavelength. For example, in an embodiment, the light source 235 A may generate data using a first wavelength corresponding to oxygenated blood, and the light source 235B may generate data relating to a second wavelength corresponding to deoxygenated blood, so as to perform an operation therebetween, e.g., compare the data at the two different wavelengths. As set forth previously, such an operation could include an operation that thereby establishes a characteristic related thereto. For example, in the embodiment described above related to oxygenated and deoxygenated blood, such an operation could establish the rate of flow of oxygenated blood (e.g., to enable the perfusion assessment mentioned above). The other example embodiments set forth above in connection with FIG. 1 (using a single light source) are also contemplated herein for the embodiment of FIG. 2 (using multiple different light sources).
[0048] FIG. 3 shows a block diagram illustrating an example of an environment 100 for implementing systems and methods in accordance with aspects. This particular embodiment, as set forth previously, includes two cannula assemblies 111 A, 11 IB, one having a non-white light source 235 A and one having a white light source 235B. Alternate embodiments, e.g., in which a cannula assembly 111 A is employed with a secondary imaging unit - not a cannula assembly - that provides image data based on a white light source, will be described separately below. Also envisioned are embodiments in which a single cannula assembly, such as cannula assembly 111A, has multiple sensor housings 217 mounted or coupled thereto, e.g., on opposing lateral sides thereof, whereby one such sensor housing 217 includes a non-white light source and an imaging sensor configured to generate enhanced image data based on the non- white light source, and the other sensor housing 217 includes a white light source and an image sensor configured to generate image data based on the white light source.
[0049] In the embodiment shown in FIG. 3, the environment 100 can include an imaging controller 105, a display device 107, cannula assemblies 111A, 11 IB, and a patient 117. The imaging controller 105 can be a computing device connected to the display device 107 and the cannula assemblies 111A, 11 IB through one or more wired or wireless communication channels 123 A, 123B, 123D. The communication channels 123 A, 123B may use various serial, parallel, video transmission protocols suitable for their respective signals such as enhanced image streams 127A, standard image stream 127B, and overlaid image stream 133, and data signals, such as spatial information 129A, 129B.
[0050] The imaging controller 105 can include hardware, software, or a combination thereof for performing operations. The operations can include receiving the enhanced image stream 127 A, standard image stream 127B and the spatial information 129 A, 129B from the cannula assemblies 111A, 11 IB. The operations can also include processing the spatial information 129 A, 129B to determine relative positions, angles, and rotations of the cannula assemblies 111A, 11 IB. In some embodiments, the enhanced image stream 127A, the standard image stream 127B and the spatial information 129A, 129B can be substantially synchronous, realtime information captured by the cannula assemblies 111A, 11 IB. In some embodiments, determining the relative positions, angles, and rotations includes determining respective fields- of-view of the cannula assemblies 111A, 11 IB. For example, the relative visual perspective can include a relative distance, angle and rotation of the cannula assemblies’ 111A, 11 IB fields-of-view.
[0051] The operations of the imaging controller 105 can also include overlaying the enhanced image stream 127 A and the standard image stream 127B into the overlaid image stream 133 based on the spatial information 129 A, 129B. In some embodiments, overlaying the enhanced image stream 127A and the standard image stream 127B includes registering and overlaying the images in the fields-of-view of the cannula assemblies 111A, 11 IB based on the spatial information 129 A, 129B. The overlaid image stream 133 can provide the enhanced image stream 127 A as an overlay of the standard image stream 127B (or vice versa) so as to provide a user, e.g., a surgeon viewing the enhanced display 145, with enhanced visualization of critical anatomy within the patient’s surgical space. In some embodiments, the enhanced display 145 may be an enhanced stereoscopic 3D view from the perspective of one of the cannula assemblies 111A, 11 IB.
[0052] The display device 107 can be one or more devices that can display the enhanced display 145 for an operator of the cannula assemblies 111A, 11 IB. As described above, the display device 107 can receive the overlaid image stream 133 and display the enhanced image 145, which can include the overlay of the enhanced image stream 127A and the standard image stream 127B from the cannula assemblies 111 A, 11 IB. The display device 107 can be a liquid crystal display (LCD) display, organic light emitting diode displays (OLED), cathode ray tube display, or other suitable display device. In some embodiments, the display device 107 can be a stereoscopic head-mounted display, such as a virtual reality headset.
[0053] FIG. 4 shows a block diagram illustrating an alternate example of an environment 100 for implementing systems and methods in accordance with various embodiments. This particular embodiment includes one cannula assembly 111A having a non-white light source 235 A, and a secondary imaging unit 115. According to various embodiments, the secondary imaging unit 115 can be a traditional endoscope or laparoscope device that is separate from and selectively insertable through a traditional cannula assembly (not shown). Alternatively, the secondary imaging unit 115 can be a CT scanner, an X-ray imager, an ultrasound system, or a fluorescence imaging system, for example.
[0054] In the embodiment shown in FIG. 4, the environment 100 may include similar components relative to the embodiment shown in FIG. 3, but having the alternate source of image data. For example, as shown in FIG. 4, the environment 100 may include an imaging controller 105, a display device 107, cannula assembly 111 A, secondary imaging unit 115, and a patient 117. Like above, the imaging controller 105 can be a computing device connected to the display device 107, the cannula assembly 111A, and the secondary imaging unit 115 through one or more wired or wireless communication channels 123 A, 123C and 123D. The communication channels 123A, 123C and 123D may use various serial, parallel, video transmission protocols suitable for their respective signals such as enhanced image stream 127A, alternate image stream 131, and, in some embodiments, data signals such as spatial information 129 A.
[0055] The imaging controller 105 can again include hardware, software, or a combination thereof for performing operations. The operations can include receiving the enhanced image stream 127 A and spatial information 129 A from the cannula assembly 111 A, and an alternate image stream 131 from the secondary imaging unit 115. The operations can also include processing the spatial information 129 A to determine the relative position of the cannula assembly 111 A as compared to other surgical tools or anatomical structures within the patient. In some embodiments, the enhanced image stream 127A, the alternate image stream 131 and the spatial information 129A can be substantially synchronous, real-time information captured by the cannula assembly 111 A and the secondary imaging unit 115.
[0056] The operations of the imaging controller 105 can also include overlaying the enhanced image stream 127A and the alternate image stream 131 into the overlaid image stream 133 based on, e.g., the spatial information 129A. In some embodiments, overlaying the enhanced image stream 127A and the alternate image stream 131 includes overlaying the enhanced image stream 127 A images from the cannula assembly 111 A onto the image stream 131 of the secondary imaging unit 115, based on the spatial information 129A. Specifically, the overlaid image stream 133 can provide the enhanced image stream 127 A as an overlay of the alternate image stream 131 (or vice versa) so as to provide a user, e.g., a surgeon viewing the enhanced display 145, with enhanced visualization of critical anatomy within the patient’s surgical space.
[0057] The display device 107 can be one or more devices that can display the enhanced display 145 for a user. As described above, the display device 107 can receive the overlaid image stream 133 and display the enhanced image 145, which can include the enhanced image stream 127 A from cannula assembly 111 A overlaid onto the alternate image stream 131 from the secondary imaging unit 115, or vice versa.
[0058] FIG. 5 shows a functional block diagram illustrating an example of a device controller 201 in accordance with aspects. The device controller 201 can be the same or similar to that described above. In the embodiment shown, the device controller 201 can include a processor 305, a memory device 307, a storage device 309, a communication interface 311, a transmitter/receiver 313, an image processor 315, spatial sensors 317, and a data bus 319.
[0059] In some embodiments, the processor 305 can include one or more microprocessors, microchips, or application-specific integrated circuits. The memory device 307 can include one or more types of random-access memory (RAM), read-only memory (ROM) and cache memory employed during execution of program instructions. The processor 305 can use the data buses 319 to communicate with the memory device 307, the storage device 309, the communication interface 311, the image processor 315, and the spatial sensors 317. The storage device 309 can comprise a computer-readable, non-volatile hardware storage device that stores information and program instructions. For example, the storage device 309 can be one or more, flash drives and/or hard disk drives. The transmitter/receiver 313 can be one or more devices that encodes/decodes data into wireless signals, such as the ranging signal 223.
[0060] The processor 305 executes program instructions (e.g., an operating system and/or application programs), which can be stored in the memory device 307 and/or the storage device 309. The processor 305 can also execute program instructions of a spatial processing module 355 and an image processing module 359. The spatial processing module 335 can include program instructions that determine the spatial information 129 by combining spatial data provided from the transmitter/receiver 313 and the spatial sensors 317. The image processing module 359 can include program instructions that, using the enhanced image signal 127 from an imaging sensor (e.g., image sensor 231 A), register and overlay the images to generate the image stream 127. The image processor 315 can be a device configured to receive an image signal 365 from an image sensor (e.g., image sensor 231 A) and condition images included in the image signal 365. In accordance with aspects, conditioning the image signal 365 can include normalizing the size, exposure, and brightness of the images. Also, conditioning the image signal 365 can include removing visual artifacts and stabilizing the images to reduce blurring due to motion. Additionally, the image processing module 359 can identify and characterize structures in the images.
[0061] In some embodiments, the spatial sensors 317 can include one or more of, piezoelectric sensors, mechanical sensors (e.g., a microelectronic mechanical system (“MEMS”), or other suitable sensors for detecting the location, velocity, acceleration, and rotation of the cannula assemblies (e.g., cannula assemblies 111 A, 11 IB).
[0062] It is noted that the device controller 201 is only representative of various possible equivalent-computing devices that can perform the processes and functions described herein. To this extent, in some embodiments, the functionality provided by the device controller 201 can be any combination of general and/or specific purpose hardware and/or program instructions. In each embodiment, the program instructions and hardware can be created using standard programming and engineering techniques.
[0063] FIG. 6 shows a functional block diagram illustrating an imaging controller 105 in accordance with aspects. The imaging controller 105 can be the same or similar to that previously described herein. The imaging controller 105 can include a processor 405, a memory device 407, a storage device 409, a network interface 413, an image processor 421, an I/O processor 425, and a data bus 431. Also, the imaging controller 105 can include image input connections 461A, 461B, 461C, image output connection 463 that receive and transmit image signals from the image processor 421. Further, the imaging controller 105 can include input/output connections 469A, 439B that receive/transmit data signals from I/O processor 425.
[0064] In embodiments, the imaging controller 105 can include one or more microprocessors, microchips, or application-specific integrated circuits. The memory device 407 can include one or more types of random-access memory (RAM), read-only memory (ROM) and cache memory employed during execution of program instructions. Additionally, the imaging controller 105 can include one or more data buses 431 by which it communicates with the memory device 407, the storage device 409, the network interface 413, the image processor 421, and the I/O processor 425. The storage device 409 can comprise a computer- readable, non-volatile hardware storage device that stores information and program instructions. For example, the storage device 409 can be one or more, flash drives and/or hard disk drives.
[0065] The VO processor 425 can be connected the processor 405 and can include any device that enables an individual to interact with the processor 405 (e.g., a user interface) and/or any device that enables the processor 405 to communicate with one or more other computing devices using any type of communications link. The I/O processor 425 can generate and receive, for example, digital and analog inputs/outputs according to various data transmission protocols. [0066] The processor 405 executes program instructions (e.g., an operating system and/or application programs), which can be stored in the memory device 407 and/or the storage device 409. The processor 405 can also execute program instructions of an image processing module 455 and an image overlay module 459. The image processing module 455 can be configured to stabilize the images to reduce the blurring, compensate for differences in tilt and rotation, remove reflections and other visual artifacts from the images, and normalize the images. Additionally, the image processing module 455 can be configured to identify and characterize structures, such as tools or tissues, in the images. Further, the imaging processing module can be configured to determine obstructions in the overlapping fields of view and process the images streams 127A, 127B to remove the obstructions.
[0067] The image overlay module 459 can be configured to analyze images received in enhanced image stream 127 A and standard image stream 127B from the cannula assemblies and overlay them into a single, overlaid image stream 133 based on the spatial information. In some embodiments, the image combination module 459 generates the overlaid image stream 133 by registering and overlaying the enhanced image stream 127A and the standard image stream 127B based on the respective fields-of-view of the cannula assemblies. In some embodiments, either of the cannula assemblies can be selected by an operator as a primary cannula assembly (e.g., cannula assembly 111A), and the image overlay module 459 can generate the overlaid image stream 133 by using the image stream 127B of the secondary cannula assembly to augment the image stream 127A. The overlaid image stream 133 can also provide a 3D view from the perspective of the primary cannula assembly. In some embodiments, the overlaid image stream 133 lacks the obstructions removed by the image processing module 455. In some embodiments, the overlaid image stream 133 also includes images provided by a secondary imaging system (e.g., secondary imaging system 115). [0068] It is noted that the imaging controller 105 is only representative of various possible equivalent-computing devices that can perform the processes and functions described herein. To this extent, in some embodiments, the functionality provided by the imaging controller 105 can be any combination of general and/or specific purpose hardware and/or program instructions. In each embodiment, the program instructions and hardware can be created using standard programming and engineering techniques.
[0069] The particular embodiments described in this application are not limiting, as they are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims. Only the terms of the appended claims are intended to be limiting, along with the full scope of equivalents to which such claims are entitled. It is also to be understood that the terminology used herein, e.g., “and”, “or”, “including”, “at least” as well as the use of plural or singular forms, etc., is for the purpose of describing examples of embodiments and is not intended to be limiting.