Hand held device and method for capturing images of skin portions
Field of the invention
[0001] The present invention concerns a method and a device for capturing images of skin portions, in particular for detection of skin lesions and skin cancer, such as melanomas. Description of related art
[0001] The number of skin cancer has more than double for the last 20 years. This is a cancer that has 95% chance to recover if detected at early stage. Melanoma and skin cancer are the only cancer that can be detected visually and with non invasive methods. [0002] The ABCD rules (Asymmetry, Border, Colour, Diameter) using naked eyes or macro photography has a low sensitivity (-80%) and a low specificity (-75%) depending of the various studies done on this subject. Also this method is even much harder to use for the very early stage detection. For early stage melanoma ABCD gives only -50% sensitivity and -60% specificity.
[0003] Dermatoscopy is a rather a recent method (really used at large scale only after the year 2000) used to detect skin cancer at early stage. This method can reach a sensitivity of detection of > 95 %in sensitivity with a well trained dermatologist. This method is used more and more often by the specialist but requires some good training to be efficient. But the sensitivity level of detection is such higher versus the other method that this shouldbe much broadly used.
[0004] Today, electronic dermatoscopes are targeting the professional market and would be in average very expensive for a non professional. [0005] Thanks to the new technologies of information (tele-video, web etc) that have so much improved for the lastl 0 years that tele-medicine starts to be a real subject of actuality. However the implementation of such system tends to be really not as common as it could be expected. Many doctors are reticent even though there is a clear market need from the patient. [0006] For the detection of skin cancer there are today few solutions in the market to get a first level of information to get a skin lesion controlled, but usually it is only by using standard macro photography (with a smartphone, a digital camera, i.e.). Even if this can be used for an auto control and observe variation of the lesion over time, this is definitively not sensitive enough to have a secure pre diagnostic based on a picture only. Those solutions are definitively not suitable for regular and full screening check having many spots to control.
[0007] One way to improve the quality of the images and the quantity of information that can be retrieved from each image is to spread o gel or liquid on the captured skin portion. For example it is known to apply a gel, such as a dermoscopic gels or liquids (water, alcohol 70%, etc) , that will reveal structures under the surface of the skin that won't be visible without the gel. This technology is known as epiluminescence microscopy. However, the gel or liquid must be carefully spread over the skin, in an adequate height, and without any bubbles. This is difficult to achieve for non specialists, and especially time-consuming even for specialists, especially when a large number of moles or skin lesions need to be examined. If the amount of gel or liquid varies, the quality of the images decreases, and a comparison between successive images of the same skin portion is difficult. [0008] Another technique is based on the use of polarized light to achieve similar type of images. Both techniques can provide complementary information, but the trend and according to some studies is to use wet dermoscopy to cover more type of pattern recognition while and better for final diagnosis, while polarized can be enough for first level of screening. [0009] US2004092802 discloses a hand held device and a method for determining the variation in concentration of chromophores within an epithelial tissue. The proposed device comprises lights sources, image sensor, data processing and displaying means. The device could comprise a patient-contacting film that is coated with index matching oil. Main drawbacks of this solution derive from the limited quantity of index matching oil that could be coated on the film. Actually, this quantity limits the number of manipulations on the patient's skin during both the preparation and the images acquisition phases.
[0010] US20120172685 discloses a skin analysis system. This system is in form of additional equipment for consumer electronic devices. The proposed solution allows periodically image acquisitions of moles, images comparisons and calculation of moles development rates. The device could generate alerts, inviting user to an urging consultation to medical specialists. The system provides consumables in form of a sheet with self- adhesive elements to enter information in the image, in particular ID numbers, dates, names and image calibration elements. These consumables are to be stuck upon the skin around the region of interest. The location of the skin lesions images is provided by taking a large body portion image where multiple skin lesions are captured together with said consumables.
[0011] WO02080771 discloses a disposable calibrated camera for skin lesion examination. This camera assembly is equipped with a transparent window that, during the use, could be coated with an index matching liquid. In one embodiment, a liquid reservoir equips the camera assembly.
However, this camera assembly focuses on a single-use mode. Actually, camera assembly components are specifically designed for a single use, limiting the employ of skin anomalies evolutions analysis and survey methods that are indispensable to support skin lesions and melanomas detections.
[0012] It is an aim of the present invention to provide a new and improved method and a new and improved device for the detection of skin lesions and melanomas. Brief summary of the invention
[0013] According to the invention, these aims are achieved by means of methods and devices comprising the claimed features. [0014] In one aspect, the invention is related to a hand held device for capturing images of skin portions for dermatoscopy analysis, comprising:
an image sensor for capturing said image;
a lens arranged for projecting a focused image of the skin on the sensor when the device is in direct contact with the skin;
a memory for storing the captured digital image;
a tank for storing a plurality of portions of contact gel and/or liquid.
[0015] The tank thus stores a plurality of portions of contact gel and/or liquid that can be used for capturing a plurality of images of successive skin portions. Therefore, the application of dermatoscopic gel or liquid is easier since the gel or liquid used for a plurality of successive image captures may be provided directly from a tank comprised in the device, without needing an external recipient for the gel or liquid.
[0016] In one embodiment, the device further includes a distribution nozzle able to spread it on the skin. The type of distribution nozzle could be a jet nozzle, a spray nozzle and/or flat stream nozzle in order to control the flow and quantity of contact gel and/or liquid that is to spread.
Multiple nozzle of the same type or combination of various types could also be employed to improve the spreading efficacy and regularity. Therefore, since the gel or liquid is distributed from a tank and with a nozzle comprised in the device, a regular, controlled amount of gel or liquid can be spread over various portions of skin.
[0017] The invention thus provides a new simple type of dermatoscopy device, that anyone could use at home and send good quality dermatoscopy pictures to a medical service centre for advice, could open a revolution in the early skin cancer detection. Health cost reduction and decease of death is the final outcome of such solution.
[0018] The device allows patients and physicists to take quickly and many dermatoscopy pictures, ideally in less than few seconds, preferably less than 3 seconds per spot.
[0019] The device could be a full electronic autonomous offline dermatoscope. The inspection process could be done offline, meaning without any direct view on a computer screen. Alternatively, the device could also work with a connected computer when required.
[0020] In one embodiment, the tank for storing a plurality of portions of contact gel and/or liquid is located above or under the surface of the body device. This tank could be entirely or partially made of elastic materials so that a portion of contact gel and/or liquid could be delivered by a simple pressure with a finger or with a mechanical pressure element on the tank to deliver the gel or liquid through an aperture.
[0021] In another embodiment, a portion of contact gel and/or liquid could be delivered by actuating a lever acting against the surface of the tank for expelling a controlled amount or gel or liquid from the tank when the lever is actuated.
[0022] In one embodiment, the device includes a tank located within the device body. A portion of contact gel and/or liquid could be delivered by actuating a mechanically or an electro-mechanically system, controlled by an external switch, button, roller and/or levers. [0023] In one embodiment, the tank is made of a disposable gel and/or liquid cartridge that can be inserted within the device
[0024] In one embodiment, the device further includes a disposable contact glass. This contact glass could be intended for direct contact with the skin. The contact glass could be made of glass or any other transparent material, including plastic. The single use contact lens could be a single use contact lens. The single use contact lens could be delivered with a contact gel and/or liquid already on the contact lens. One purpose of the contact glass is to maintain flat and thin the liquid or gel on the skin for wet dermoscopy. In an embodiment, the contact glass could also be a lens that refocuses the ray light onto the sensor.
[0025] In one embodiment, the device includes a distribution nozzle able to spread contact gel and/or liquid onto the outside surface of this contact glass and/or onto the skin. The type of distribution nozzle could be a jet nozzle, a spray nozzle and/or flat stream nozzle in order to control the flow and quantity of contact gel and/or liquid that is to spread. Multiple nozzle of the same type or combination of various types could also be employed to improve the spreading efficacy. [0026] In one embodiment, a small canal and/or a groove is provided in the contact glass that will help to bring the gel or liquid to the skin portion. This embodiment is more comfortable for the patient because of less liquid spreading and flowing all over.
[0027] Wet dermoscopy requires to apply the right pressure. In one embodiment, the hand held device is further equipped with a small pressure sensor that will indicate to the user by a beep and / or small led and / or other visual indicator that the pressure is right or too high or too low. Those indications can also be used in combination with an interactive web base platform to drive the user to the best and optimum image capture process.
[0028] In one embodiment, the device further includes at least one reference marker on the contact glass, or on any other glass or lens in the optical path, in order to include a reference mark in the captured digital image. The marker could be a colour marker. [0029] In one embodiment, the device also includes software for calibrating the colour of the captured image, based on the known colour of the markers.
[0030] In one embodiment, the device further comprises a lighting system for lighting the portion of skin during the capture of the image.
[0031] In one embodiment, the device further comprises a light pointer for projecting a light beam on the centre of portion of skin from which an image should be captured. The device is then equipped with means for automatically switching this light off during the capture of the image. [0032] In one embodiment, the device further comprises data selection means for indicating the body area from which an image is taken. The data selection means could be arranged for allowing a user selection of one body area among predefined body areas.
[0033] In one embodiment, the device further comprises a computer product having a software code arranged for causing a microprocessor that runs said software code to include an indication of a body area as metadata in corresponding captured images.
[0034] In another embodiment, the device further comprises software means for automatic identification of image area containing skin lesions. [0035] In one embodiment, the device further includes computer vision modules for recognizing features in the capture images, for example features representatives of particular lesions or melanomas. The device could also include software modules for recognizing changes between successive images of a same spot, in particular changes representatives of a melanoma.
[0036] In one embodiment, the device further comprises a computer in charge of the execution of at least one of the calibration, image processing and/or analysis software modules. The device could be connected to this computer by one or multiple communication means, as, for example, an USB cable.
Brief Description of the Drawings
[0037] The invention will be better understood with the aid of the description of an embodiment given by way of example and illustrated by the figures, in which:
Fig. 1 shows a view over a hand held device.
Fig. 2 shows markers on the skin contact glass.
Fig. 3 shows a single use contact glass with the contact gel already on the contact glass.
Fig. 4 shows a hand held device with contact gel and/or liquid tank according to the invention.
Fig. 5 shows details of a hand held device contact glass with contact gel and/or liquid distribution canal or groove according to the invention.
Fig. 6shows a device equipped with contact glass provided by contact gel already on the contact glass.
Fig. 7shows a hand held device with a light pointer.
Fig. 8 shows a view of a spot with markers clearly visible.
Fig. 9 shows successive views of the same spot taken at different time.
Fig. 10 shows two successive pictures of a same spot where similar areas are circled. Fig. 1 1 shows an example of a reference picture of a skin portion used for automatic or manual determination of skin type.
Fig. 12shows a skin image capturing system comprising a hand held device with contact gel and/or liquid tank and a connected computer.
Fig. 13shows a skin image capturing system comprising a hand held device and a computer.
Detailed Description of possible embodiments of the Invention
[0038] Figure 1 shows a device, or dermoscope, comprising an image sensor and a contact glass 2 that will be in contact with the skin for dermatoscopy picture. It is also possible to have a device with a lens or glass maintained at a predefined distance of the skin but without direct contact. The captured images of skin portions can be associated with metadata which can be used by a software in a remote computer 66 for analysing the pictures as showed in Figs.12-13.
[0039] The focus of the device is set at factory so that a lens 70 projects a focused image of the skin portion on the image sensor. The user has not to worry about adjusting the focus.
[0040] The skin contact glass 2 may have some markers 6 on it (Fig. 2) that will be used for some software treatment. There might be some white and black reference dots. There might be some reference colour dots.
[0041] The markers are clearly visible on each image captured with the device, as shown on Figs. 8-1 1 . The colour and/or brightness of those reference dots are known and might be used for any image treatment during inspection by the user or the dermatologist, for calibrating the colour and brightness and saturation of the stored and captured image, and/or the colour of the displayed and/or printed image. The expected colour and brightness of the reference marks can be compared with their colours in the captured images, to adjust the colour and brightness of the whole image.
[0042] Furthermore, the orientation of successive images of a same spot, for example successive views taken at different times can be adjusted by software, by comparing matching patterns in different orientations, as shown on Figure 9. A software module in the device 1 , or in a computer 66 receiving the images captured with the device, can be used for adapting the orientation of various images of a same portion of skin and facilitating their comparison. [0043] This is a way to calibrate the colours and to have systematically the same results independently to the screen or computer platform used. This allows making sure that comparing pictures of the same lesion at different time, a reference is available for a true comparison in order to detect correctly any colour changes. [0044] The software used to visualize the lesion will automatically adjust the colour and brightness based on the reference dot. This might also be used for automatic or human detection of significant patterns.
[0045] The device 1 may include a single use contact glass 2 with the contact gel and or Iiquid8 already on the glass. An example of single-use glass 2 with gel 8 on it is illustrated on Figure 3. In the example, a thin protection foil 1 1 is used to protect this contact gel and/or liquid layer 8 during transportation and storage of this contact glass in a packaging blister 10. The contact glass can be removed from the blister and mounted at the extremity of the device 1 , as illustrated on Figures 6, 12 and 13. The contact glass with the gel on it may be replaced after each image, or after each series of images captured from a patient. If the amount of liquid or gel on the contact glass is not sufficient, it might be possible to add more gel from a tank 22 comprised in the device. [0046] Alternatively, the device 1 might comprise a contact glass that will be used substantially during the whole life of the device 1 , without need to be replaced after each use.
[0047] In both embodiments, some gel or water needs to be applied on the skin for every new spot to check. In order to make the process faster, easy to use, and more reproducible, the device is equipped with a tank 22 for storing a plurality of portions of contact gel and/or liquid as shown on Fig. 4. When the device 20 is on contact with the skin, this tank could be actuated in order to dispense the right quantity of gel and/or water through a distribution canal 23 equipped with an aperture to make the dermatoscopy picture.
[0048] This tank 22 could be located on the external surface of the device body 5 in order to be manipulated by the device's user. The tank could be, for example, entirely or partially made of elastic materials. A simple pressure of the tank with at least one finger will distribute the required portion of contact gel and/or liquid from the tank on the skin portion.
[0049] Alternatively, the device could comprise a lever (not shown) whose movement actuates in a mechanical way the tank, delivering a more precisely defined portion of contact gel and/or liquid onto the skin and/or onto the contact glass.
[0050] Alternatively, the device could comprise an electro-mechanical actuator activated by a switch, a button and/or a roller to deliver a precise portion of gel or liquid onto the skin and/or onto the contact glass. [0051] The tank could be located within the device body 5.
[0052] A part of distribution canal 23 could be located on or within the hand held device body 5. [0053] A part of the distribution canal 23 could be located on or within the hand held device extender 3.
[0054] The tank may be provided with an overture and a lid for contact gel and/or liquid recharge purpose. [0055] In case of need, the tank could be made of a disposable contact gel and/or liquid cartridge for a safe, clean and rapid recharge.
[0056] The device could be equipped with distribution nozzle 21 to control the direction, the flow and the quantity of contact gel and/or liquid that is to spread on the skin and/or on the contact glass 2. [0057] Various types of nozzle could be employed, as for example, jet nozzle, spray nozzle and flat stream nozzle. Multiple nozzles of the same type or combination of various types could be used to improve the efficacy and regularity of the spreading operation.
[0058] In one embodiment, the gel or the liquid is applied first on the skin or on the contact glass of the dermatoscope, and then the device is applied and pressed on the lesion. This requires some practice and fast handling operation when the gel or liquid is applied on vertical skin portion.
[0059] Therefore, in another embodiment, a small canal is provided in the glass, or a groove 24 along the body of the device, that will help to bring the gel or liquid to the skin portion 33 (Fig. 5). In such case, the device is pressed on the skin lesion and then we push the liquid through. This embodiment is also more comfortable for the patient because of less liquid spreading and flowing all over. [0060] Using wet dermoscopy requires to apply the right pressure. If the pressure is too low, the epiluminescence is not appearing and more air bubbles might be seen. If the pressure is too high, the liquid might be faster flowing outside the area. If the pressure is really above a normal and reasonable pressure it can influence the blood vessel close by and change some aspect of the lesion. But usually after few shots only, it is easy to get it right. Nevertheless, one feature of the invention is a small pressure sensor 69 that will indicate to the user by a beep and / or small led and / or other visual indicator that the pressure is right or too high or too low.
[0061] Those indications can also be used in combination with an interactive web base platform to drive the user to the best and optimum image capture process.
[0062] The small pressure sensor could be mounted in order to measure the pressure between the contact glass 2 and the extender 3, or
alternatively, between the extender 3 and the body 5 of the hand held device.
[0063] The device may comprise a light pointer 31 shown on Figure7that projects a focused beam on the skin 30 where the centre of the image is. This help to centre small lesions 32 of few millimetres diameter only. The ray of light pointer can be with LED or laser. The power must be very low and should not risk any injuries to the eyes or the skin.
[0064] Just before the snap shot, the device 1 automatically switches off the ray of light pointer 32 and turns on a white LED, or a plurality of LEDS producing in combination a white light. The LED remains on only few seconds. The power of the LED is automatically adjusted to have the reference spots on the skin contact glass correctly lighting.
[0065] The device further comprises indication retrieving means of body area from which the acquired image is or will be captured. The purpose is to code the picture with the body area to make easier some offline software treatment at later stage.
[0066] This means could be automatically executed and/or manually driven. [0067] The indication retrieving means could comprise means for automatically matching the captured image with a previously captured image stored in the memory 68, and automatically or manually
determining the body area to which said previously captured image belongs. A signature from the captured image could be retrieved and compared with signatures already available and associated with a body area. This signature could depend on skin natural markers such as nevus 36, skin pores, skin lesions 33, skin grains, hairs34, buttons and/or veins (Fig. 9).
[0068] The indication retrieving means could further comprise means for stitching the acquired image with other stored images, and building an image of a larger body area.
[0069] The indication retrieving means could further comprise an inertial sensor in the hand-held device for determining the position of said hand-held device. [0070] Before to start shooting on some skin lesion, the patient could furthermore select manually the body area on which he or she has the lesion to be controlled.
[0071] An example of device with a menu based selection, body area selection roller 4 and display 40 is shown on Figures 4, 6, 7, 12 and 13. [0072] The device could further comprise a code bar reader exploiting the device sensor 61 . The body area could so manually selected capturing a code bar image indicating a particular body area. The code-bar could be a conventional code-bar, a QR-code or any other type of visual coding that can be used as identification with image analyzes. [0073] The body area can be identified such as right forearm, left forearm, neck, top hand right, etc. [0074] With detailed body map and multiple code bar or QR code or other, the localisation can be much more precisely identified (for instance with a number, such as for example ten, different zone on the left arm)
[0075] All the pictures captured are saved on an onboard memory 68, which could be a FRAM, and/or a removable SD cards for instance. Pictures stored in the memory 68 may be downloaded to a computer when the device 1 or the portable memory (SD cards) is plugged into a computer 66.A USB interface 63 could be used for this purpose. Pictures can also be transferred over a wireless interface, for example over wifi or Bluetooth or any wireless technology.
[0076] The device contains at least one battery64 that is rechargeable via USB port 63 or via a battery loader or loading station.
[0077] The device contains a real time clock 67 that allows to clearly identifying the time of capture of each picture. [0078] The device 1 contains all the necessary code to automatically open and connect to the web application, in order to transfer the images to a remote server for storage, comparison with previously captured images of the same skin portion, and/or analysis by a dermatologist or
automatically with a computer vision system for example. [0079] A coding of each picture is used throughout the web application. The device associates a unique code similar to what is described below to the picture:
ID: 1 12395657824 - 001 - 00002 - 0000001 - 2501 122102 - xxxxxx - xxxxxx - xxxxxx (Device - user - location - Picture number - date / time picture - flag diagnose - date diagnose - Visa). [0080] This code could be transferred along with the picture to a remote computer 66 or server where is can be used for classifying and retrieving the picture, and for facilitating the comparison with other images of the same spot at different times. The user identification part of the code can be an anonymous identification or an alias.
[0081] The images captured with the device can be processed locally in the device, for example in order to adjust their colour, contrast and brightness, possibly based on the reference marks. Moreover, a software module in the device might carry out computer vision steps for
automatically identifying images areas which might with some probability match a skin cancer or another skin lesion. An indication might be displayed to the user, for example a visual or audible indication, when a suspicious spot has been found on the skin.
[0082] The images might then be transferred to a further computer, for example over a wireless interface, a USB cable, a memory card and so on. The images might also be transferred to a remote computer over an
Internet connection. The computer receives the images with associated metadata, such as a picture code, and carries out further methods for processing the images, classifying them and for assisting the user or the dermatologist in detecting skin lesions and melanomas and/or automatic detection of suspicious images.
[0083] The device can be used with a real time interactive web base application for guidance, real time information and training and feedback.
[0084] The software in the device and/or in the computer might carry out an automatic detection of skin cancer with digital image analysis. The software might also assist the user or dermatologist and help him in the fast and reliable detection of melanomas. The specialist operator while checking and scrolling through hundreds of pictures for a pre diagnostics, is thus driven through all the steps with software assistance to highlight some area or characteristics when required, and to help him in verifying the images and detecting more quickly and in a more reliable way the most critical images.
[0085] For example, the software executed on the computer can detect that several pictures taken at different time correspond to a same spot. The comparison might be used on the date - it is unlikely that several pictures of same spot taken on the same day need to be compared -, on the body part identification, and on computer vision algorithms for matching successive views of a same spot when those views include similar group of pixels, or possible transformation from one view to the next. Fig. 10 shows two views of a same spot taken at two months intervals; the circled area 35 corresponds to similar group of pixels which can be used for automatic matching of views.
[0086] The software can also be used for facilitating manual matching of successive images, for example by presenting to the user a selection of images likely to correspond and that the user can manually match.
[0087] The software can also be used for automatic detection of areas that have changed between successive pictures, and automatic marking of those areas, or automatic marking of series of pictures in which significant changes were observed. [0088] The software might detect patterns or image elements
representative of different types of lesions. Skin cancer such as melanomas can have various levels of the development characterized by changing aspects and various patterns. The software might comprise routines or modules for detecting image portions representative of some of those levels. For example, a module can be executed for detecting and
highlighting pictures having blue gray globules which are sometimes hard to see but very specific to carcinoma at level 2. The software can highlight the pictures having those areas, and in those pictures highlight the area that matches the characteristics of the blue gray globules. At level 6, the software might highlight the group of pixels that have similar colour of blood vessels and organized like blood vessel. [0089] The process might also comprise a step of indicating of detecting the skin type of the patient. A skin type might be associated to a series of pictures taken during a session. The skin type might be included in the image coding. The skin type might be determined from a picture taken at a specific location, for example a location without melanomas. An example of image which can be used for automatic and/or manual determination of skin type is shown on Figure 10. The skin type might be indicated as a skin index type and might depend on the skin colour.
List of elements used in the drawings
1 : Hand held device for capturing images of skin portion
2: Contact glass
3: Extender
4: Roller for data selection
5: Body of the hand held device
6: Image calibration marker
7: Acquiring button
8: Contact gel and/or liquid
9: Field-of-view of the image sensor
10: Contact glass packaging blister
1 1 : Thin protection foil
20: Hand held device for capturing images of skin portion
21 : Contact gel and/or liquid distribution nozzle
22: Contact gel and/or liquid tank
23: Contact gel and/or liquid distribution canal
24: Contact gel and/or liquid distribution canal or groove within the contact glass
30: Skin surface
31 : Ray light focusing point
32: Skin portion of interest
33: Skin anomalies
34: Hair
35: Similar areas between two successive pictures of a same spot
36: Nevus
40: LCD display
60: Lighting system
61 : Image sensor
62: Microprocessor
63: USB interface
64: Battery pack
65: USB cable : Personal Computer: Real time clock: Memory
: Pressure sensor: Lens