CN108198175B

Movatterモバイル変換

Info

Publication number: CN108198175B
Application number: CN201711465117.7A
Authority: CN
Inventors: 吴安平
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2017-12-28
Filing date: 2017-12-28
Publication date: 2021-09-10
Anticipated expiration: 2037-12-28
Also published as: WO2019129004A1; CN108198175A

Abstract

Translated fromChinese

本发明公开了一种光学元件破裂的检测方法。光学元件包括相机模组的相机光学元件，所述检测方法包括：获取第一图像；处理第一图像以判断第一图像是否存在裂纹图案；在第一图像存在裂纹图案时确认相机光学元件破裂。本发明还公开了一种光学元件的破裂的检测装置、计算机设备和计算机可读存储介质。本发明实施方式的检测方法、检测装置、计算机设备和计算机可读存储介质通过识别相机模组获取的第一图像中是否有裂纹图案来判断相机光学元件是否破裂。如此，计算机设备可以自行检测光学元件是否完好，提升了计算机设备的智能性。

The invention discloses a method for detecting the rupture of an optical element. The optical element includes a camera optical element of a camera module, and the detection method includes: acquiring a first image; processing the first image to determine whether a crack pattern exists in the first image; and confirming that the camera optical element is broken when a crack pattern exists in the first image. The invention also discloses a crack detection device of an optical element, a computer device and a computer-readable storage medium. The detection method, detection device, computer equipment, and computer-readable storage medium of the embodiments of the present invention determine whether the optical element of the camera is broken by identifying whether there is a crack pattern in the first image obtained by the camera module. In this way, the computer equipment can detect whether the optical element is in good condition by itself, which improves the intelligence of the computer equipment.

Description

Detection method, detection device, computer equipment and computer readable storage medium

Technical Field

The present invention relates to the field of imaging technologies, and in particular, to a detection method, a detection apparatus, a computer device, and a computer-readable storage medium.

Background

A conventional mobile phone is generally mounted with a device having an optical element such as a camera. During the use of the mobile phone, the optical element may be broken due to external force. However, the current mobile phone does not have the function of self-detecting whether the optical element is broken, and the intelligence of the mobile phone is poor.

Disclosure of Invention

The embodiment of the invention provides a detection method, a detection device, a computer and a computer-readable storage medium.

The invention provides a method for detecting breakage of an optical element. The optical element comprises a camera optical element of a camera module, and the detection method comprises the following steps:

acquiring a first image;

processing the first image to determine whether a crack pattern is present in the first image; and

confirming the camera optical element is broken when a crack pattern is present in the first image.

The invention provides a detection device for optical element breakage. The optical element comprises a camera optical element of a camera module, and the detection device comprises a first acquisition module, a first processing module and a first confirmation module. The first acquisition module is used for acquiring a first image. The first processing module is used for processing the first image to judge whether a crack pattern exists in the first image. The first confirmation module is to confirm that the camera optical element is broken when a crack pattern is present in the first image.

The invention provides a computer device, which comprises a memory and a processor, wherein the memory is stored with computer readable instructions, and the instructions are executed by the processor to cause the processor to execute the detection method.

The present invention provides one or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the detection methods described above.

The detection method, the detection device, the computer equipment and the computer readable storage medium of the embodiment of the invention judge whether the camera optical element is broken or not by identifying whether the crack pattern exists in the first image acquired by the camera module. Therefore, the computer equipment can automatically detect whether the optical element is intact or not, and the intelligence of the computer equipment is improved.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow diagram of a detection method according to some embodiments of the present invention.

FIG. 2 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 3 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 4 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 5 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 6 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 7 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 8 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 9 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 10 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 11 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 12 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 13 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 14 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 15 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 16 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 17 is a block diagram of a computer device according to some embodiments of the invention.

FIG. 18 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 19 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 20 is a schematic flow chart of a detection method according to some embodiments of the present invention.

FIG. 21 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 22 is a block diagram of a computer device in accordance with certain embodiments of the invention.

FIG. 23 is a block diagram of image processing circuitry in accordance with certain implementations of the invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

Referring to fig. 1, fig. 2 and fig. 23, the present invention provides a method for detecting a crack of an optical device. The optical element includes a cameraoptical element 411 of a camera module. The detection method comprises the following steps:

01: acquiring a first image;

02: processing the first image to determine whether the first image has a crack pattern; and

04: the cameraoptical element 411 is confirmed to be broken when the first image has a crack pattern.

Referring to fig. 2, the present invention further provides adevice 10 for detecting a broken optical element. The optical element includes a cameraoptical element 411 of a camera module. The method for detecting a break of an optical element according to the embodiment of the present invention can be realized by theapparatus 10 for detecting a break of an optical element according to the embodiment of the present invention. Thedetection device 10 comprises a first acquisition module 11, afirst processing module 12 and afirst confirmation module 15.Step 01 may be implemented by the first obtaining module 11,step 02 may be implemented by thefirst processing module 12, and step 03 may be implemented by thefirst confirming module 15. That is, the first acquiring module 11 may be used to acquire the first image. Thefirst processing module 12 may be configured to process the first image to determine whether the first image has a crack pattern. Thefirst confirmation module 15 may be used to confirm that the cameraoptical element 411 is broken when the first image has a crack pattern. Further, thefirst confirmation module 15 may be used to confirm that the cameraoptical element 411 is not broken when the first image is absent of a crack pattern.

Referring to fig. 3, the present invention further provides acomputer apparatus 100. Thecomputer apparatus 100 includes amemory 51 and aprocessor 52. Thememory 51 has stored therein computer-readable storage instructions. The computer readable instructions 511, when executed by theprocessor 52, cause theprocessor 52 to perform the operations of acquiring the first image, processing the first image to determine whether the first image has a crack pattern, and confirming that the cameraoptical element 411 is broken when the first image has a crack pattern.

In an embodiment of the present invention, the camera module is aninfrared camera 41, and the first image instep 01 is formed by theinfrared camera 41 capturing infrared light from ambient light reflected by the object. The cameraoptical element 411 of the camera module is a lens component in theinfrared camera 41, such as a filter, a focusing lens, a dustproof lens, and the like.

In some embodiments, thecomputer device 100 may be a cell phone, a tablet computer. Notebook computer, intelligent bracelet, intelligent wrist-watch, intelligent helmet, intelligent glasses etc..

Taking a mobile phone as an example, it can be understood that the existing mobile phone is usually equipped with devices with optical elements, such as a visible light camera, aninfrared camera 41, and a light supplement element. During the use of the mobile phone, the optical element may be broken due to external factors, such as dropping of the mobile phone. However, the current mobile phone does not have the function of self-detecting whether the optical element is broken, and the intelligence of the mobile phone is poor.

Since theinfrared camera 41 photographs the crack pattern on the broken cameraoptical element 411 when the cameraoptical element 411 of theinfrared camera 41 is broken, the detection method, thedetection device 10 and thecomputer apparatus 100 for detecting the broken optical element of the embodiment of the present invention determine whether the cameraoptical element 411 is broken by recognizing whether there is a crack pattern in the first image acquired by the camera module. Thus, thecomputer device 100 can detect whether the optical element is intact by itself, and the intelligence of thecomputer device 100 is improved.

Further, in some embodiments, upon confirming that the cameraoptical element 411 is broken, thedetection apparatus 10 and a prompting module (e.g., a speaker, a display screen) in thecomputer device 100 may prompt the user to replace the broken cameraoptical element 411 by voice or text.

Referring to FIG. 4, in some embodiments, processing 02 the first image to determine whether the first image has a crack pattern includes:

021: constructing a multi-stage crack pattern classifier based on the Haar-like rectangular feature set;

022: training a multi-stage crack pattern classifier by adopting a positive crack pattern sample and a negative crack pattern sample; and

023: and detecting whether the first image has a crack pattern or not by adopting the trained multistage crack pattern classifier.

Referring to fig. 5, in some embodiments, thefirst processing module 12 includes a first constructing unit 121, afirst training unit 122 and a first detecting unit 123. Step 021 may be implemented by the first construction unit 121,step 022 may be implemented by thefirst training unit 122, and step 023 may be implemented by the first detection unit 123. That is, the first construction unit 121 may be configured to construct a multi-stage crack pattern classifier based on a Haar-like rectangular feature set. Thefirst training unit 122 may be used to train a multi-stage crack pattern classifier using positive and negative crack pattern samples. The first detection unit 123 may be configured to detect whether a crack pattern exists in the first image by using the trained multi-stage crack pattern classifier.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, cause theprocessor 52 to further perform the operations of constructing a multi-stage crack pattern classifier based on the Haar-like rectangular feature set, training the multi-stage crack pattern classifier using the positive crack pattern samples and the negative crack pattern samples, and detecting whether the crack pattern exists in the first image using the trained multi-stage crack pattern classifier.

Specifically, first, a large number of positive crack pattern samples and negative crack pattern samples are selected, wherein the positive crack pattern samples are samples with crack patterns of the cameraoptical element 411, and the negative crack pattern samples are samples with crack patterns of the non-cameraoptical element 411. Secondly, the positive crack pattern sample and the negative crack pattern sample are subjected to resolution normalization calculation to reduce the size of the image, so that the subsequent rapid detection of image features is facilitated. Then, aiming at morphological characteristics of the crack patterns, a Haar-like rectangular feature set capable of detecting line features, center surrounding features, diagonal line features and edge features is adopted to construct a multistage crack pattern classifier. Specifically, for a matrix eigenvalue corresponding to each Haar-like rectangular characteristic calculator, then, a plurality of matrix eigenvalues are selected from the plurality of matrix eigenvalues to form a plurality of weak classifiers, and each weak classifier comprises one or more matrix eigenvalues. The weak classifiers form a plurality of strong classifiers, and each strong classifier comprises a plurality of weak classifiers. All strong classifiers are cascaded to form a multi-stage crack pattern classifier. And then, training the multistage crack pattern classifiers by adopting a large number of positive crack pattern samples and negative crack pattern samples with normalized resolution to correct the weight corresponding to each weak classifier, so that the loss function of the finally output image classification result is smaller than a preset loss value. And finally, detecting whether the first image has the crack pattern by using the trained multistage crack pattern classifier. The multistage crack pattern classifier is formed by cascading a plurality of strong classifiers, each strong classifier is formed by cascading a plurality of weak classifiers, the multistage structure can accurately extract image characteristics, and the accuracy of image classification is improved.

Referring to FIG. 6, in some embodiments, the first image includes a plurality of frames.Step 02 of processing the first image to determine whether the first image has a crack pattern includes:

021: processing the multiple frames of first images to judge whether crack patterns exist in the multiple frames of first images;

the method for detecting the breakage of the optical element further comprises:

031: comparing whether the crack positions of the crack patterns in any two frames of first images are consistent;

032: when the crack positions in any two frames of the first image coincide, it is confirmed that the cameraoptical element 411 is broken.

Referring to fig. 7, in some embodiments, the detectingdevice 10 further includes a first comparingmodule 13. Step 021 may be implemented by thefirst processing module 12,step 031 may be implemented by the first comparingmodule 13, and step 032 may be implemented by the first confirmingmodule 15. That is, thefirst processing module 12 may be further configured to process the plurality of frames of the first image to determine whether the crack pattern exists in each of the plurality of frames of the first image. The first comparingmodule 13 may be configured to compare whether the crack positions of the crack patterns in any two frames of the first image are consistent. Thefirst confirmation module 15 may also be used to confirm that the cameraoptical element 411 is broken when the crack positions in any two frames of the first image coincide. At this time, when the crack pattern exists in all of the plurality of frames of the first image, but the crack positions in any two frames of the first image do not coincide, it is confirmed that the cameraoptical element 411 is not broken.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, enable theprocessor 52 to further perform the operations of processing the plurality of first images to determine whether crack patterns exist in each of the plurality of first images, comparing whether crack positions of the crack patterns in any two first images are consistent, and confirming that the cameraoptical element 411 is broken when the crack positions in any two first images are consistent.

Specifically, after theinfrared camera 41 captures a plurality of frames of first images, theprocessor 52 may detect each frame of first image by using a trained multi-stage crack pattern classifier to determine whether each frame of first image has a crack pattern. Upon detecting the presence of a crack pattern in each frame of the first image, theprocessor 52 extracts the crack location in the corresponding first image according to each crack pattern identified. Subsequently, theprocessor 52 compares the crack positions in the first images of the plurality of frames two by two, and when each comparison result shows that the crack positions in the corresponding two first images are consistent, the cameraoptical element 411 is considered to be broken. The two first images are divided into two frames, wherein the two first images have the same crack position, and the difference of the distances between the crack positions in the two first images is smaller than a preset distance difference. It is understood that when the cameraoptical element 411 of theinfrared camera 41 is broken, the position of the crack pattern is unchanged in each frame of the first image captured by theinfrared camera 41. Therefore, whether the cameraoptical element 411 is broken or not can be further determined by the crack position, so that the accuracy of the detection of the breakage of the cameraoptical element 411 can be improved.

Referring to fig. 8, in some embodiments, the optical elements further include a fill-in optical element of the infrared fill-inlamp 42. The method for detecting the breakage of the optical element further comprises:

051: when theoptical element 411 of the camera is not damaged, the infrared fill-inlight 42 and theinfrared camera 41 are turned on;

052: acquiring a second image according to the infrared light emitted by the infraredlight supplement lamp 42;

053: processing the second image to determine whether the second image has a light pattern; and

054: and confirming that the light supplementing optical element is broken when the second image has the light pattern.

Referring to fig. 9, in some embodiments, the detectingdevice 10 further includes afirst opening module 21, a second obtainingmodule 22, asecond processing module 23, and a second confirmingmodule 25. Step 051 may be implemented by thefirst opening module 21,step 052 may be implemented by the second obtainingmodule 22,step 053 may be implemented by thesecond processing module 23, and step 054 may be implemented by the second confirmingmodule 25. That is, the first turning-onmodule 21 may be configured to turn on the infrared fill-inlight 42 and theinfrared camera 41 when it is determined that the cameraoptical element 411 is not damaged. The second obtainingmodule 22 may be configured to obtain a second image according to the infrared light emitted by the infrared fill-inlight 42. Thesecond processing module 23 can be used for processing the second image to determine whether the second image has the light pattern. Thesecond confirmation module 25 may be configured to confirm that the fill-in optical element is broken when the second image has the light pattern. Further, the second confirmingmodule 25 may be further configured to confirm that the light supplementing optical element is not broken when the second image does not have the light pattern.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, cause theprocessor 52 to perform: when theoptical element 411 of the camera is determined not to be damaged, the infrared fill-inlamp 42 and theinfrared camera 41 are turned on, a second image is obtained according to infrared light emitted by the infrared fill-inlamp 42, the second image is processed to determine whether a light pattern exists in the second image, and when the light pattern exists in the second image, the operation of breaking the fill-in optical element is determined.

The method for detecting the breakage of the optical element in the embodiment of the invention can be further used for detecting whether the light supplement optical element of the infraredlight supplement lamp 42 is broken or not. The light supplement optical element of the infraredlight supplement lamp 42 is a lens or a lampshade in the infraredlight supplement lamp 42. Because when the light supplementing optical element is broken, the light emitted by the light supplementing optical element cannot be uniformly diffused into a scene, and because of the existence of cracks of the light supplementing optical element, light fringes can also appear in the infrared light diffused into the scene. Therefore, when it is confirmed that the cameraoptical element 411 of theinfrared camera 41 is not broken, the infrared fill-inlamp 42 and theinfrared camera 41 can be turned on at the same time. Theinfrared camera 41 can capture light emitted by the infrared fill-inlight 42 to capture a second image. Theprocessor 52 processes the second image to detect whether there is a light pattern in the second image, and determines that the fill-in optical element is broken when there is a light pattern in the second image. Therefore, on one hand, thecomputer device 100 can automatically detect the integrity of the light supplementing optical element, and the intelligence of thecomputer device 100 is improved; on the other hand, the light that infraredlight filling lamp 42 sent is infrared laser usually, and when light filling optical element broke, the people's eye was burnt easily to the infrared laser that sends through crack department, consequently, can in time close infraredlight filling lamp 42 whencomputer equipment 100 detected that light filling optical element broke to the situation of avoiding infrared laser to burn people's eye takes place, promotes the security that the user usedcomputer equipment 100, improves user experience.

Referring to fig. 10, in some embodiments, thestep 053 of processing the second image to determine whether the second image has the light pattern includes:

0531: constructing a multi-level optical pattern classifier based on a Haar-like rectangular feature set;

0532: training a multi-stage light pattern classifier by adopting positive light pattern samples and negative light pattern samples; and

0533: and detecting whether the second image has the light pattern or not by adopting the trained multi-stage light pattern classifier.

Referring to fig. 11, in some embodiments, thesecond processing module 23 includes a second constructing unit 231, a second training unit 232, and a second detecting unit 233.Step 0531 may be implemented by the second construction unit 231,step 0532 may be implemented by the second training unit 232, andstep 0533 may be implemented by the second detection unit 233. That is, the second construction unit 231 may be configured to construct a multi-level optical grain pattern classifier based on the Haar-like rectangular feature set. The second training unit 232 may be used to train the multi-stage lightprint pattern classifier using the positive lightprint pattern samples and the negative lightprint pattern samples. The second detecting unit 233 may be configured to detect whether the second image has a light pattern by using the trained multi-stage light pattern classifier.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, cause theprocessor 52 to further perform operations of constructing a multi-level lightstripe pattern classifier based on the Haar-like rectangular feature set, training the multi-level lightstripe pattern classifier using the positive lightstripe pattern samples and the negative lightstripe pattern samples, and detecting whether a lightstripe pattern exists in the second image using the trained multi-level lightstripe pattern classifier.

Specifically, firstly, a large number of positive light pattern samples and negative light pattern samples are selected, wherein the positive light pattern samples are samples with light patterns, and the negative light pattern samples are samples with non-light patterns. Secondly, the resolution normalization calculation is carried out on the positive light pattern sample and the negative light pattern sample to reduce the size of the image, so that the subsequent rapid detection of the image characteristics is facilitated. Then, aiming at morphological characteristics of the light pattern, a Haar-like rectangular feature set capable of detecting line features, center surrounding features, diagonal features and edge features is adopted to construct a multilevel light pattern classifier. Specifically, for a matrix eigenvalue corresponding to each Haar-like rectangular characteristic calculator, then, a plurality of matrix eigenvalues are selected from the plurality of matrix eigenvalues to form a plurality of weak classifiers, and each weak classifier comprises one or more matrix eigenvalues. The weak classifiers form a plurality of strong classifiers, and each strong classifier comprises a plurality of weak classifiers. All the strong classifiers are cascaded to form a multistage light pattern classifier. And then, training the multi-stage light pattern classifiers by adopting a large number of positive light pattern samples and negative light pattern samples with normalized resolution to correct the weight corresponding to each weak classifier, so that the loss function of the finally output image classification result is smaller than a preset loss value. And finally, detecting whether the second image has the light pattern by using the trained multistage light pattern classifier. The multistage light pattern classifier is formed by cascading a plurality of strong classifiers, each strong classifier is formed by cascading a plurality of weak classifiers, the multistage structure can accurately extract image characteristics, and the accuracy of image classification is improved.

Referring to fig. 12, in some embodiments, the second image comprises a plurality of frames, and thestep 053 of processing the second image to determine whether the second image has the light pattern comprises:

0534: processing the multiple frames of second images to judge whether the multiple frames of second images all have light pattern or not;

the method for detecting the breakage of the optical element further comprises:

0541: comparing whether the positions of the light patterns in any two frames of second images are consistent; and

0542: and when the positions of the light fringes in any two frames of second images are consistent, confirming that the light supplementing optical element is broken.

Referring to fig. 13, in some embodiments, thedetection apparatus 10 further includes a second comparison module 24.Step 0534 may be implemented by thesecond processing module 23.Step 0541 may be performed by the second comparison module 24.Step 0542 may be implemented by thesecond validation module 25. That is, thesecond processing module 23 is further configured to process the plurality of frames of the second image to determine whether the light pattern exists in each of the plurality of frames of the second image. The second comparing module 24 can be used to compare whether the positions of the light stripes in any two frames of the second images are consistent. Thesecond confirmation module 25 may further be configured to confirm that the light supplement optical element is broken when the positions of the light fringes in any two frames of the second image are consistent. Further, thesecond confirmation module 25 may be further configured to confirm that the light supplementing optical element is not broken when the light pattern positions in any two frames of the second image are not consistent.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511 are executed to enable theprocessor 52 to further perform the operations of processing the plurality of frames of the second images to determine whether the light pattern exists in each of the plurality of frames of the second images, comparing whether the positions of the light patterns in any two frames of the second images are consistent, and confirming that the light-filling optical element is broken when the positions of the light patterns in any two frames of the second images are consistent.

Specifically, when it is confirmed that the cameraoptical element 411 of theinfrared camera 41 is not broken, theinfrared fill light 42 and theinfrared camera 41 are turned on at the same time. After theinfrared camera 41 captures infrared light emitted by the infrared fill-inlight 42 to obtain multiple frames of second images, theprocessor 52 may detect each frame of second image by using a trained multi-stage light pattern classifier to determine whether there is a light pattern in each frame of second image. Upon detecting the presence of a light pattern in each frame of the second image,processor 52 extracts the position of the light pattern in the corresponding second image according to each identified light pattern. Subsequently, theprocessor 52 compares the positions of the light fringes in the second images of the multiple frames two by two, and when each comparison result shows that the positions of the light fringes in the corresponding two frames of second images are consistent, the light supplementing optical element is considered to be broken. The consistency of the positions of the light stripes in the two frames of second images means that the distance difference between the positions of the light stripes in the two frames of second images is smaller than a preset distance difference. It can be understood that, when the light supplement optical element of the infraredlight supplement lamp 42 is broken, since the distance between the infraredlight supplement lamp 42 and theinfrared camera 41 is fixed, the position of the crack pattern is not changed in each frame of the second image captured by theinfrared camera 41. Therefore, whether the light supplement optical element of the infraredlight supplement lamp 42 is broken or not can be further judged through the crack position, so that the accuracy of breakage detection of the light supplement optical element can be improved.

Referring to fig. 14, in some embodiments, the optical elements further include structured light optical elements of the structuredlight projector 43, wherein the structured light optical elements refer to collimation, diffraction elements, etc. in the structuredlight projector 43. The method for detecting the breakage of the optical element further comprises:

061: upon confirming that the cameraoptical element 411 is not broken, the structuredlight projector 43 and theinfrared camera 41 are turned on;

062: acquiring a third image from the infrared light emitted from the structuredlight projector 43;

063: processing the third image to judge whether depth abnormal information exists in the third image; and

065: confirming that the structured light optical element is broken when there is depth anomaly information in the third image.

Referring to fig. 15, in some embodiments, thedetection apparatus 10 further includes asecond starting module 31, a third obtainingmodule 32, athird processing module 33, and a third confirmingmodule 35. Step 061 may be implemented by the second enablingmodule 31,step 062 may be implemented by the third obtainingmodule 32,step 063 may be implemented by thethird processing module 33, and step 065 may be implemented by the third confirmingmodule 35. That is, thesecond activation module 31 may be used to activate the structuredlight projector 43 and theinfrared camera 41 upon confirming that the cameraoptical element 411 is not broken. Thethird acquisition module 32 may be configured to acquire a third image based on the infrared light emitted by the structuredlight projector 43. Thethird processing module 33 may be configured to process the third image to determine whether depth anomaly information exists in the third image. Thethird confirmation module 35 may be used to confirm that the structured light optical element is broken when there is depth anomaly information in the third image.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, cause theprocessor 52 to further perform the operations of turning on the structuredlight projector 43 and theinfrared camera 41 when it is determined that the cameraoptical element 411 is not broken, acquiring a third image based on infrared light emitted from the structuredlight projector 43, processing the third image to determine whether depth anomaly information exists in the third image, and determining that the structured light optical element is broken when the depth anomaly information exists in the third image.

The method for detecting the breakage of an optical element according to the embodiment of the present invention can be further used for detecting whether or not the structured light optical element of the structuredlight projector 43 is broken. The structured light optical element of the structuredlight projector 43 may be a collimating lens or a Diffractive Optical Element (DOE) in the structuredlight projector 43. When the structured light optical element is broken, and theinfrared camera 41 calculates the depth information of the current scene from the captured image and the preset reference image, a certain part of the depth information of the current scene may become abnormally large. Therefore, when it is confirmed that the cameraoptical element 411 of theinfrared camera 41 is not broken, the structuredlight projector 43 and theinfrared camera 41 can be turned on at the same time. Theinfrared camera 41 may capture infrared light projected by the structured-light projector 43 to capture a third image.Processor 52 processes the third image to detect whether depth anomaly information is present in the third image and confirms that the structured light optical element is broken when depth anomaly information is present in the third image. Therefore, on one hand, thecomputer device 100 can automatically detect the integrity of the structured light optical element, and the intelligence of thecomputer device 100 is improved; on the other hand, the light emitted by the structuredlight projector 43 is usually infrared laser, and when the structured light optical element of the structuredlight projector 43 is broken, the infrared laser of the structuredlight projector 43 easily burns the human eyes, so that the structuredlight projector 43 can be closed in time when thecomputer device 100 detects that the structured light optical element is broken, thereby avoiding the situation that the human eyes are burnt by the infrared laser, improving the safety of the user using thecomputer device 100, and improving the user experience.

Referring to fig. 16, in some embodiments, theprocessing 063 of the third image to determine whether depth anomaly information exists in the third image comprises:

0631: calculating the scattered point offset of each pixel point in the third image according to the third image and a preset reference image;

0632: calculating the depth information of each pixel point in the third image according to the scatter offset;

0633: calculating the number of pixel points with adjacent depth information larger than a preset threshold; and

0634: and determining that the depth abnormal information exists in the third image when the number of the adjacent pixel points is larger than the preset number.

Referring to fig. 17, in some embodiments, thethird processing module 33 includes a first calculatingunit 331, asecond calculating unit 332, athird calculating unit 333 and a confirmingunit 334.Step 0631 may be implemented by thefirst calculation unit 331,step 0632 may be implemented by thesecond calculation unit 332,step 0633 may be implemented by thethird calculation unit 333, andstep 0634 may be implemented by thevalidation unit 334. That is to say, the first calculatingunit 331 is configured to calculate a scatter offset of each pixel point in the third image according to the third image and the preset reference image, the second calculatingunit 332 is configured to calculate depth information of each pixel point in the third image according to the scatter offset, the third calculatingunit 333 is configured to calculate the number of pixel points whose adjacent depth information is greater than the preset threshold, and the determiningunit 334 is configured to determine that depth anomaly information exists in the third image when the number of adjacent pixel points is greater than the preset number.

Referring back to fig. 3, in some embodiments, when executed by theprocessor 52, the computer readable instructions 511 enable theprocessor 52 to perform operations of calculating a scatter offset of each pixel point in the third image according to the third image and the preset reference image, calculating depth information of each pixel point in the third image according to the scatter offset, calculating the number of pixel points whose adjacent depth information is greater than a preset threshold, and determining that depth anomaly information exists in the third image when the number of adjacent pixel points is greater than a preset number.

Specifically, first, theprocessor 52 extracts a plurality of input image blocks from the third image, where center points of any two input image blocks are the same and have different sizes, and then theprocessor 52 extracts a plurality of matching search windows from the preset reference image, where the matching search windows are matching blocks extracted within a certain range around a corresponding point in the preset reference image, which is the same as the center point of the input image block, as the center point. Then, the sum SAD of the absolute values of the differences of corresponding pixels between the input image block and a matching block of the same size in the matching search window is calculated by adopting a parallel algorithm, so that the SAD between each input image block and all the matching blocks is obtained. Then, for each input image block, a matching block corresponding to the SAD at the minimum SAD value is selected from the plurality of SADs as a best matching block for the input image block. Then, theprocessor 52 subtracts the coordinate value of the center point of the best matching block from the coordinate value of the center point of the preset reference image to calculate the best offset corresponding to the input image block, where the best offset is the scatter offset of the pixel point where the center point of the corresponding input image block is located, and positive and negative scatter offsets represent the front-back relationship between the depth of the center point of the input image block and the depth of the preset reference pattern, and the larger the scatter offset is, the farther the plane vertical distance between the center point of the input image block and the preset reference pattern with a known depth distance is. Theprocessor 52 calculates the amount of scatter offset for each pixel in the third image one by one according to the above method. Finally, theprocessor 52 calculates the depth information of each pixel point according to the depth calculation formula by combining the known depth distance of the preset reference pattern and the baseline distance between the structuredlight projector 43 and theinfrared camera 41 according to the calculated offset of each pixel point in the third image. Theprocessor 52 determines whether a plurality of adjacent pixel points with depth information greater than a preset threshold exist in the third image according to the calculated depth information, and determines that depth abnormality information exists in the third image when the number of the adjacent pixel points with depth information greater than the preset threshold exceeds a preset number. In this manner, it can be determined whether the structured light optical element of the structuredlight projector 43 is broken based on the determination of the depth anomaly information in the third image.

Referring to fig. 18, in some embodiments, the third image includes a plurality of frames, and thestep 063 of processing the third image to determine whether the depth anomaly information exists in the third image includes:

0635: processing the multiple frames of third images to judge whether depth abnormal information exists in the multiple frames of third images;

the method for detecting the breakage of the optical element further comprises:

0641: comparing whether the position information of the abnormal depth information in any two frames of third images is consistent; and

0642: and confirming that the structured light optical element is broken when the position information of the depth anomaly information in any two frames of third images is consistent.

Referring to fig. 19, in some embodiments, thedetection apparatus 10 further includes a third comparingmodule 34.Step 0635 may be implemented by thethird processing module 33,step 0641 may be implemented by the third comparingmodule 34, andstep 0642 may be implemented by the third confirmingmodule 35. That is, thethird processing module 33 may be configured to process the plurality of frames of the third image to determine whether depth anomaly information exists in the plurality of frames of the third image. The third comparingmodule 34 can be used to compare whether the position information of the abnormal depth information in any two frames of the third image is consistent. The third confirmingmodule 35 may be configured to confirm that the structured light optical element is broken when the position information of the depth anomaly information in any two frames of the third image is consistent.

Referring back to fig. 3, in some embodiments, the computer readable instructions 511, when executed by theprocessor 52, enable theprocessor 52 to further perform the operations of processing the plurality of frames of the third image to determine whether there is depth anomaly information in the plurality of frames of the third image, comparing whether the position information of the anomaly depth information in any two frames of the third image is consistent, and confirming that the structured light optical element is broken when the position information of the depth anomaly information in any two frames of the third image is consistent.

Specifically, upon confirming that the cameraoptical element 411 of theinfrared camera 41 is not broken, the structuredlight projector 43 and theinfrared camera 41 are turned on at the same time. After theinfrared camera 41 captures the infrared laser light projected by the structuredlight projector 43 to capture a plurality of frames of third images, theprocessor 52 detects each frame of third image to determine whether depth anomaly information exists in each frame of third image. Upon detecting that there is depth anomaly information for each frame of the third image,processor 52 extracts its position information in the corresponding third image according to each depth anomaly information identified. Subsequently, theprocessor 52 compares the position information of the depth anomaly information in the third images of the multiple frames two by two, and when each comparison result shows that the position information of the depth anomaly information in the corresponding two frames of third images is consistent, the structured light optical element is considered to be broken. The consistency of the position information of the depth anomaly information in the two frames of third images means that the distance difference between the position information of the depth anomaly information in the two frames of third images is smaller than a preset distance difference. It is understood that when the structured light optical element of the structuredlight projector 43 is broken, the distance between the structuredlight projector 43 and theinfrared camera 41 is fixed because the crack of the structured light optical element is fixed, and therefore, theprocessor 52 calculates the position information of the depth anomaly information of each frame of the third image from each frame of the third image captured by theinfrared camera 41 to be constant. Therefore, whether the structured light optical element of the structuredlight projector 43 is broken or not can be further determined by the position information of the depth anomaly information, so that the accuracy of the detection of the breakage of the structured light optical element can be improved.

In some embodiments, when it is determined that the cameraoptical element 411 of theinfrared camera 41 is not broken, the infrared fill-inlamp 42 and theinfrared camera 41 may be simultaneously turned on, and whether the fill-in optical element of the infrared fill-inlamp 42 is broken or not may be determined according to whether a light pattern exists in each of a plurality of frames of second images captured by theinfrared camera 41 and whether positions of the light patterns in any two frames of second images are consistent or not, and when the fill-in optical element is not broken, the structuredlight projector 43, the infrared fill-inlamp 42 and theinfrared camera 41 are simultaneously turned on, and whether the structured light optical element of the structuredlight projector 43 is broken or not may be determined according to whether depth anomaly information exists in each of a plurality of frames of third images captured by theinfrared camera 41 and whether position information corresponding to the depth anomaly information of any two frames of third images is consistent or not. Therefore, when the light supplement optical element of the light supplement lamp is not broken, the infraredlight supplement lamp 42 can normally execute the light supplement function, the brightness of the third image acquired by theinfrared camera 41 is high, calculation of depth information is facilitated, and the accuracy of structural light optical element breakage detection can be further improved.

Referring to fig. 20, in some embodiments, the method for detecting the breakage of the optical element further includes:

001: acquiring the movement speed of the camera module;

002: and (5) judging whether the movement speed is greater than a preset speed value or not, and entering the step (01) to acquire a first image when the movement speed is greater than the preset movement speed value.

Referring to fig. 21, in some embodiments, thedetection apparatus 10 further includes a fourth obtainingmodule 16 and a fourth processing module 17. Step 001 may be implemented by the fourth obtainingmodule 16 and step 002 may be implemented by the fourth processing module 17. That is, the fourth obtainingmodule 16 may be configured to obtain a moving speed of the camera module, and the fourth processing module 17 may be configured to determine whether the moving speed is greater than a preset speed value, and enterstep 01 to obtain the first image when the moving speed is greater than the preset moving speed value.

Referring to fig. 3, in some embodiments, when the computer readable instructions 511 are executed by theprocessor 52, theprocessor 52 further executes the operation of acquiring the moving speed of the camera module, determining whether the moving speed is greater than the preset speed value, and enteringstep 01 to acquire the first image when the moving speed is greater than the preset moving speed value.

Wherein, the speed sensor can be adopted to detect the movement speed of the camera module. When the movement speed of the camera module is higher, it indicates that the computer device 100 may fall, at this time, the infrared camera 41 is turned on to detect whether the camera optical element 411 of the infrared camera 41 is broken, and when the camera optical element 411 is not broken, the infrared fill-in lamp 42 and the infrared camera 41 are turned on at the same time to detect whether the fill-in optical element of the infrared fill-in lamp 42 is broken, and then the structured light projector 43 and the infrared camera 41 are turned on at the same time to detect whether the structured light optical element of the structured light projector 43 is broken; or, when the camera optical element 411 is not broken, the structured light projector 43 and the infrared camera 41 are turned on at the same time to detect whether the structured light optical element of the structured light projector 43 is broken, and then the infrared fill-in light lamp 42 and the infrared camera 41 are turned on at the same time to detect whether the fill-in light optical element of the infrared fill-in light lamp 42 is broken; or, when the camera optical element 411 is not broken, the infrared fill-in lamp 42 and the infrared camera 41 are turned on simultaneously to detect whether the fill-in optical element of the infrared fill-in lamp 42 is broken, and when the fill-in optical element is not broken, the structured light projector 43, the infrared fill-in lamp 42 and the infrared camera 41 are turned on simultaneously to detect whether the structured light optical element of the structured light projector 43 is broken. In this way, the detection of the breakage of the optical element is performed when the moving speed of thecomputer apparatus 100 is large, without the detection of the breakage of the homogeneous optical element every time the optical element is used, reducing the power consumption of thecomputer apparatus 100.

The present invention also provides a non-transitory computer-readable storage medium containing one or more computer-executable instructions. The computer-executable instructions, when executed by the one ormore processors 52, cause theprocessors 52 to perform the method for detecting a break in an optical element as described in any of the above embodiments.

For example, the computer-executable instructions, when executed by the one ormore processors 52, cause theprocessors 52 to perform the operations of:

01: acquiring a first image;

As another example, the computer-executable instructions, when executed by the one ormore processors 52, cause theprocessors 52 to perform the operations of:

FIG. 22 is a schematic diagram of internal modules ofcomputer device 100, under an embodiment. As shown in fig. 22, thecomputer apparatus 100 includes aprocessor 52, a memory 51 (e.g., a nonvolatile storage medium), aninternal memory 54, adisplay 55, and aninput device 56, which are connected by asystem bus 53. Thememory 51 of thecomputer device 100 has stored therein an operating system and computer readable instructions 511 (shown in FIG. 3). The computer readable instructions 511 are executable by theprocessor 52 to implement the method for detecting a break in an optical element according to any of the above embodiments. Theprocessor 52 may be used to provide computing and control capabilities to enable the overall operation of thecomputing device 100. Theinternal memory 54 of thecomputer device 100 provides an environment for the computer-readable instructions 511 in thememory 51 to run. Thedisplay 55 of thecomputer device 100 may be a liquid crystal display or an electronic ink display, and theinput device 56 may be a touch layer covered on thedisplay 55, a case, a trackball or a touch pad arranged on the housing of thecomputer device 100, or an external keyboard, a touch pad or a mouse. Thecomputer device 100 may be a mobile phone, a tablet computer, a notebook computer, a personal digital assistant, or a wearable device (e.g., a smart bracelet, a smart watch, a smart helmet, smart glasses), etc. It will be understood by those skilled in the art that the configuration shown in fig. 22 is only a schematic diagram of a part of the configuration related to the solution of the present invention, and does not constitute a limitation to thecomputer device 100 to which the solution of the present invention is applied, and aspecific computer device 100 may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.

Referring to fig. 23, acomputer apparatus 100 according to an embodiment of the invention includes animage processor circuit 80. Theimage processing circuit 80 may be implemented using hardware and/or software. Various Processing units may be included that define an ISP (Image Signal Processing) pipeline. FIG. 23 is a diagram of animage processing circuit 80 in one embodiment. As shown in fig. 23, for convenience of explanation, only aspects of the image processing technique related to the embodiment of the present invention are shown.

As shown in fig. 23, the image processing circuit includes an ISP processor (which may be theprocessor 52 or a part of the processor 52) and control logic. The image data captured by theinfrared camera 41 is first processed by the ISP processor 63, and theISP processor 82 analyzes the image data to capture image statistics that may be used to determine one or more control parameters of theinfrared camera 41. Theinfrared camera 41 may include acamera optics 411 and animage sensor 412, thecamera optics 411 including one or more lenses. Theimage sensor 412 may acquire light intensity and wavelength information captured by each imaging pixel and provide a set of raw image data that may be processed by theISP processor 82. The sensor 81 (e.g., gyroscope) may provide parameters of the acquired image processing (e.g., anti-shake parameters) to theISP processor 82 based on the sensor interface type. The sensor interface may be an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above.

In addition, theimage sensor 412 may also send raw image data to thesensor 81, thesensor 81 may provide the raw image data to theISP processor 82 based on the sensor interface type, or the sensor may store the raw image data in thememory 51.

TheISP processor 82 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and theISP processor 82 may perform one or more image processing operations on the raw image data, gathering statistical information about the image data. Wherein the image processing operations may be performed with the same or different bit depth precision.

TheISP processor 82 may also receive image data from thememory 51. For example, the sensor interface sends raw image data to thememory 51, and the raw image data in thememory 51 is then provided to theISP processor 82 for processing.

Upon receiving raw image data from the image sensor interface or from thesensor 81 interface or from thememory 51, theISP processor 82 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 62 for additional processing before being displayed. TheISP processor 82 receives the processing data from thememory 51 and performs image data processing on the processing data. The image data processed by theISP processor 82 may be output to a display screen for viewing by a user and/or further processed by a Graphics Processing Unit (GPU). In addition, the output of theISP processor 82 may also be sent to theimage memory 82, and thedisplay screen 55 may read the image data from thememory 51. In one embodiment,memory 51 may be configured to implement one or more frame buffers. In addition, the output of theISP processor 82 may be sent to an encoder/decoder 84 for encoding/decoding the image data. The encoded image data may be saved and decompressed before being displayed on thedisplay screen 55. The encoder/decoder 84 may be implemented by a CPU or GPU or coprocessor.

The statistical data determined by theISP processor 82 may be sent to thecontrol logic unit 83. For example, the statistical data may include image sensor statistics such as auto-exposure, auto-focus, flicker detection, black level compensation, lens shading correction, and the like.Control logic 83 may include a processing element and/or microcontroller that executes one or more routines (e.g., firmware) that determine camera control parameters andISP processor 82 control parameters based on the received statistical data. For example, the control parameters of theinfrared camera 41 may includesensor 81 control parameters (e.g., gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, lens control parameters (e.g., focal length for focusing or zooming), or a combination of these parameters.

It will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments of the methods. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), or the like.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.