Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with aspects of embodiments of the application as detailed in the accompanying claims.
It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and/or components.
First, the terms related to the present application will be explained.
1. Color histogram-refers to a color feature that is widely used in many image retrieval systems. It describes the proportion of different colours in the whole image.
2. Edge intensity refers to the magnitude of the gradient of the edge points of the image. An edge point means that the gray values of the pixels on both sides thereof are significantly different. Edge points exist between a pair of adjacent points, one inside the lighter area and the other outside.
With the deep digital transformation of enterprises, information visualization has become an important tool for enterprise real-time monitoring, business analysis and decision support. And a display screen (large screen data) is displayed as a core carrier for information visualization, and is widely applied to the scenes of operation monitoring, production scheduling, data display and the like of enterprises. However, due to factors such as complex data sources, various display forms, and fluctuation of network environments, the stability and accuracy of display of the display often face many challenges.
In order to ensure the stability and accuracy of display screen display, background data sources of the display screen can be analyzed, however, the background data may not reflect whether the display image of the display screen is abnormal or not, and the adaptability is poor.
In one example, the stability and accuracy of display displays often face a number of challenges, mainly expressed as 1-4:
1. and the data loss is caused by abnormal interfaces, failure in updating data sources, system breakdown and the like, so that partial content areas cannot be normally loaded, and the problems of blank charts, data non-display and the like can be caused.
2. Common problems with data presentation errors include data scrambling, over-range values, and format anomalies, which often result from coding errors, data transmission failures, or presentation rule misapplication.
3. And the page component is missing, namely a page key module is not loaded or lost, such as a chart, a monitoring index or a navigation bar is missing, and the integrity of data display is directly affected.
4. The page component is abnormal, namely the page component can have the conditions of blurring, color deviation and the like due to network jitter, abnormal rendering or resolution, and the understanding of the user on the data is affected.
The development efficiency is low, in the interface development process, a developer needs to write a large amount of calling codes manually, so that the time and the labor are wasted, and errors are easy to occur. In addition, the lack of unified development tools and plug-in support also limits the improvement in development efficiency.
In some embodiments, the inspection information can be collected through two-dimension codes, NFC, video monitoring and other modes, and the inspection data is uploaded to a remote processing platform for analysis. However, analyzing inspection information collected in a two-dimensional code, NFC, video monitoring, etc. manner according to a remote processing platform has the following drawbacks 1-6:
1. Too relying on identification of physical tags, the system relies on two-dimensional code tags or NFC tags on the device. If the tag is damaged, blocked or lost, the inspection function cannot be effectively completed.
2. The real-time performance is insufficient, the data acquisition and processing needs to be uploaded to a remote server for analysis, and the problem that the real-time response is impossible due to network delay can be solved. The dependence of remote processing increases the demands on the network environment, and data loss or delay is likely to occur in a scenario where the network is unstable.
3. The support for dynamic content is inadequate. The current platform records static information mainly through a video monitoring module and a routing inspection information acquisition module, and lacks the processing capability of dynamically updating content (such as real-time data refreshing, chart changing and the like). The analysis of the video monitoring data depends on manual work, and the degree of automation is low.
4. Lacks anomaly detection and intelligent analysis capabilities. Existing platforms lack the ability to automatically detect presentation anomalies (e.g., data loss, format errors, component loss, etc.) for large screens, only providing basic support for video or image recordings. And can not realize the deep intelligent analysis of the inspection data.
5. Visualization and interaction of inspection results are not supported. The platform function is mainly focused on data acquisition and uploading, and lacks visual display and interaction functions on inspection results, so that abnormal information cannot be visually presented. The lack of a user-friendly interface design is unfavorable for improving the inspection efficiency.
6. The general line is poor and the adaptability is limited. The current design depends on specific inspection equipment (such as equipment with a built-in two-dimensional code/NFC module), and has poor adaptability to other types of inspection scenes. There is a lack of versatility support for multiple scenarios, multiple data sources (such as large screen data presentation).
In view of the above, an embodiment of the present application provides an image detection method, which includes acquiring an image to be detected, determining an abnormal item of the image to be detected according to an image parameter of the image to be detected, where the abnormal item is an item having a display defect.
The method provided by the embodiment of the application is described in detail below with reference to the attached drawings.
It should be noted that, the network system described in the embodiment of the present application is for more clearly describing the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application, and those skilled in the art can know that, with the evolution of the network system and the appearance of other network systems, the technical solution provided in the embodiment of the present application is applicable to similar technical problems.
Fig. 1 is a schematic diagram of an image detection system 10 according to an embodiment of the present application. As shown in fig. 1, the image detection system 10 may include a terminal device 11 and an image detection apparatus 12.
Wherein the terminal device 11 may be adapted to display an image of the inspection area. Also referred to as a terminal, mobile Station (MS), mobile Terminal (MT), etc., is a device that provides voice and/or data connectivity to a user, and for example, the terminal device 11 may be a handheld device, an in-vehicle device, etc., having a wireless connection function. Specifically, a smart phone (mobile phone), a pocket computer (pocket personal computer, PPC), a palm computer, a Personal Digital Assistant (PDA), a notebook computer, a tablet computer, a wearable device, or a vehicle-mounted device, etc. may be used. The embodiment of the present application does not limit the specific technology, the specific number and the specific device configuration adopted by the terminal device 11.
The image detection device 12 according to the embodiment of the present application may be configured to acquire an image displayed by the terminal device 11, and determine an abnormal item existing in the image according to an image parameter of the image. For example, the processing device may be an electronic device having a processing function such as a computer or a server. For example, the image detection device 12 may be a computer, a server, or the like. The server may be a single server or may be a server cluster formed by a plurality of servers. In some implementations, the server cluster may also be a distributed cluster. The embodiment of the present application is not limited to the specific technique, specific number and specific apparatus configuration of the image detection device 12.
The image detection system can comprise a user authority management module, a patrol task configuration module, a large screen capture and data acquisition module, an anomaly detection and analysis module, an anomaly feedback and work order generation module and a model optimization module.
1. The user authority management module is used for realizing authority hierarchical control of the user, and ensuring that only authorized personnel can configure and view the patrol task and the result. Through verification of the identity and the role of the user, the security of the system operation and the confidentiality of data are ensured. The management process of the user rights management module may include the following S1-S2:
s1, a user authority management module is based on user authentication and authority control of an existing enterprise system.
The user rights management module may interface with a Single Sign On (SSO) system or an OA system of the enterprise to obtain existing account and account information of the enterprise employee, avoid repeated registration, and perform user authentication using OAuth2 or LDAP protocols.
The account information may include, among other things, job number (employee id), name (full name), department name (DEPARTMENT NAME), job title (job title), and contact phone (phone number).
S2, performing authority classification based on role-based access control (RBAC).
The user rights management module may set three main roles and their rights ranges. The three main roles may include administrator, general user, auditor.
The authority range of the administrator can include configuring the patrol task, modifying system parameters (such as patrol frequency and exception handling rule), and viewing global patrol reports (tasks and reports of all users).
The authority range of the common user can include checking the patrol task and result of the appointed task (which need to be bound with the department to which the user belongs), and downloading or deriving the patrol task and result.
Authority ranges of auditors may include viewing system logs and anomaly detection records, but not downloading or operating tasks.
2. The inspection task configuration module is used for allowing a user to flexibly define inspection parameters such as inspection targets, frequencies, screenshots and the like, so that the inspection task has configurability and high efficiency, and the diversified requirements are met.
The user may define inspection parameters through the system interface, and the inspection parameters may be as shown in table 1 below:
TABLE 1 inspection parameter schematic form
It should be noted that table 1 is only an exemplary illustration, and the inspection parameters may also include other types of parameters, which are not limited herein.
The user can also define configuration parameters corresponding to the inspection task through the system interface.
Configuration parameters may include screenshot resolution (screenshot _resolution), screenshot file save path (screenshot _save_path), whether to start dynamic content delay capture (enable_delay_capture), and specified monitoring area.
Where the screenshot resolution supports manual input or selection of standard resolution (e.g., 1920x1080, 1366x 768).
The screenshot file save path may be automatically stored to the enterprise's unified data repository by default, or the path may be specified by the user (e.g./screenshots/2024/task 1 /).
For dynamically refreshing pages (e.g., carousel graphics), the user may enable a time-lapse capture function. The option is True or False.
The designated monitoring area may be a key detection area in a user manually frame a screenshot, such as a chart or table location. The coordinate ranges are represented by the upper left corner (x 1, y 1) and the lower right corner (x 2, y 2).
The routing inspection task configuration module can flexibly define the execution time of the routing inspection task by adopting a Cron expression and support the following scheduling modes, namely a timing mode, a manual mode and a triggering mode.
Wherein the timing mode is used to designate that the patrol be triggered daily, hourly, and every minute. For example, 0, 12,18 means that tasks are performed 8:00, 12:00, and 18:00 per day.
Wherein the manual mode is manually triggered by a user through an interface. The trigger mode is used for automatically triggering after binding certain events (such as abnormal alarms).
The patrol task configuration module can dynamically allocate computing resources based on task priorities, so that priority execution of high-priority tasks is ensured. The parameter name may be task_priority, and the range may be 1-5 (1 being highest).
3. The large screen capturing and data collecting module is used for realizing efficient collection and storage of large screen contents, supporting compatibility of multi-screen resolution and dynamic contents, and meeting data collecting requirements under complex service scenes.
For example, the large screen shot and data acquisition module may use Selenium or Puppeteer to control the browser to automatically load the target page. And the main stream browser (such as Chrome and Edge) is supported, and the compatibility is ensured.
For pages where dynamic refresh exists (such as carousel graphics or animated content), dynamic content may be captured by way of delayed shots and frame-by-frame shots.
And (3) time delay shooting, namely waiting for screenshot after the loading of the dynamic content of the page is completed by adding a time.
And capturing images frame by frame, namely capturing the page change in a short time frame by frame, and storing the page change in a GIF or video form.
And (5) automatically adapting to different screen resolutions to ensure that the screenshot content is complete and undistorted. Resolution parameters (e.g., 1920x1080 or 4K modes) may be customized.
In the image compression process, the large screen capture and data acquisition module can use PNGQuant or OptiPNG to realize lossless compression, so that the storage space is reduced. For the case where fast loading is required, conversion to JPEG format (lossy compression) is supported.
During image storage, the large screen shots and data acquisition module may be stored in a unified manner to an enterprise data repository or distributed file system (e.g., HDFS). Standardized naming of file names and paths (e.g., company_task_ YYYYMMDD _hhmm. Png) facilitates indexing and retrieval.
In some embodiments, the large screen shot and data acquisition module may record metadata for each screen shot, including the time of the screen shot, resolution, task name, etc., for facilitating subsequent analysis.
4. The abnormality detection and analysis module is used for detecting four types of large screen problems including data deletion, data display abnormality, page component deletion and component display abnormality through an artificial intelligence technology.
5. The abnormality feedback and work order generation module is used for realizing automatic processing after abnormality detection, and comprises the steps of generating an abnormality report and a work order, and providing a multi-channel notification and abnormality tracking mechanism to ensure timely response and closed-loop solution of problems.
For example, the anomaly feedback and worksheet generation module may generate a concrete description for each type of problem and extract the anomaly region in the screenshot. And automatically generating a graphic-text combined inspection report based on the report template.
The patrol report comprises a problem abstract, a screenshot mark and recommended measures.
The problem abstract comprises a problem type, occurrence time and detection task. Screenshot labels are used to highlight abnormal regions with borders or masks. The recommended action is used to provide the user with possible repair suggestions.
The output format of the inspection report can be PDF, HTML, etc., which is convenient for cross-platform viewing.
The notification content of the multi-channel notification may include an exception summary, a screenshot, a report link, a priority level, and the like.
The notification means may include:
short message pushing summary of key questions and work order number.
And E, attaching a complete report and a screenshot to the mail, and supporting one-key access to the worksheet page.
And the enterprise OA system is used for sending the abnormal information to a designated work group or task system through API docking.
The pushing mechanism of the inspection report can be pushing through a message queue (such as Kafka or RabbitMQ) so as to ensure timeliness and reliability of notification.
In the process of generating the work order, the work order can be automatically generated according to the abnormal classification and the severity.
The work order content may include information such as anomaly descriptions, screenshot links, responsibility departments, etc. And setting high, medium and low priorities according to the business influence of the problems.
The work order life cycle management can comprise state switching of new construction, processing, solution and the like. After the work order is completed, the system automatically files and updates the detection result database.
The abnormal feedback and work order generation module can be in butt joint with a work order management system (such as Jira and Buddhist channels) in an enterprise, so that automatic synchronization of abnormal information is realized.
6. The model optimization module is used for continuously optimizing the detection model according to abnormal data generated in the inspection process and user feedback, and improving the accuracy and adaptability of detection. Through a feedback mechanism and a dynamic updating flow, the system can learn and adapt to different service scenes.
In one example, for a first display defect, the user feedback may include a marked false positive area, a supplemental false negative area.
The model optimization module can extract image parameters (brightness and edge intensity of different pixels) of the user marked area, add training data, optimize a threshold range of brightness mean and variance according to user feedback, and optimize a detection model by using YOLOv transfer learning technology.
In some embodiments, for the first display defect, the user feedback may include the actual existence of the user mark but misjudged as missing components, the missing component that the user complements (e.g., unidentified custom chart).
The optimization method for identifying the first display defects comprises the steps of template library expansion and target detection model optimization.
The template library expansion refers to automatically updating a standard chart template library according to user feedback, grouping and classifying the commonly used chart types, and improving the matching efficiency.
Target detection model optimization refers to fine tuning Detectron the model using the missed data of the user markers. More types of samples are added in the training process, so that the generalization capability of the model is improved.
In one example, for the second display defect, the user feedback may include identifying the wrong text, and the correct numerical format or business scope for correction.
The optimization method for identifying the second display defect comprises the steps of OCR model enhancement, rule base updating and migration learning.
OCR model enhancement refers to combining actual text content corrected by a user in feedback, performing secondary training on the OCR model, and reducing cold start problems by utilizing a pre-trained multi-language OCR model (such as Google Vision API).
Rule base updating refers to dynamically expanding regular expression rules to adapt to more business scenes. Context verification rules (such as the relationship of a KPI to other indicators) for a particular service are added.
Transfer learning refers to converting feedback data into a small training set and fine-tuning a model by using a transfer learning method.
In one example, for a third display defect, the user feedback may include the actual clear component of the user mark being misinterpreted as blurred and the correct color range of the user-indicated abnormal color region.
The optimization method for identifying the third display defect comprises the steps of updating an image quality evaluation model and optimizing a color detection model.
And updating an image quality evaluation model, namely adjusting a definition detection threshold value and increasing detection adaptation to a dynamic fuzzy (such as a carousel image) scene by using a misjudgment area marked by a pointer.
Color detection model optimization refers to expanding a color histogram template according to user marks, accommodating more color changes, and adding a recognition and tolerance adjustment mechanism for specific color distribution in the model.
The model optimization module also includes a dynamic update and iteration mechanism. Dynamic update and iteration mechanisms include data cleansing, periodic fine tuning, model evaluation and verification.
And data cleaning, namely cleaning and normalizing the feedback data in a designated period to remove repeated or invalid samples.
Periodic fine tuning refers to periodically fine tuning the detection model using the cleaned feedback data.
The model evaluation and verification means that the performance improvement is verified by an A/B test before the updated model is on line, wherein the A test comprises the changes of detection accuracy, missed judgment rate and misjudgment rate. The B test includes a satisfaction score for the user feedback.
In particular, each device in fig. 1 may adopt the constituent structure shown in fig. 2 or include the components shown in fig. 2. Fig. 2 is a schematic structural diagram of an image detection apparatus 200 according to an embodiment of the present application, where the image detection apparatus 200 may be a network device, or the image detection apparatus 200 may be a chip or a system on a chip in the network device. As shown in fig. 2, the image detection apparatus 200 includes a processor 201, a communication interface 202, and a communication line 203.
Further, the image detection device 200 may further include a memory 204. The processor 201, the memory 204, and the communication interface 202 may be connected by a communication line 203.
The processor 201 is a CPU, a general-purpose processor, a network processor (network processor, NP), a digital signal processor (DIGITAL SIGNAL processing, DSP), a microprocessor, a microcontroller, a programmable logic device (programmable logic device, PLD), or any combination thereof. The processor 201 may also be other devices with processing functions, such as, without limitation, circuits, devices, or software modules.
Communication interface 202 is used to communicate with other devices or other communication networks. The communication interface 202 may be a module, a circuit, a communication interface, or any device capable of enabling communication.
A communication line 203 for transmitting information between the respective components included in the image detection apparatus 200.
Memory 204 for storing instructions. Wherein the instructions may be computer programs.
The memory 204 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and/or instructions, a random access memory (random access memory, RAM) or other type of dynamic storage device capable of storing information and/or instructions, an EEPROM, a CD-ROM (compact disc read-only memory) or other optical disk storage, an optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, etc.
It should be noted that the memory 204 may exist separately from the processor 201 or may be integrated with the processor 201. Memory 204 may be used to store instructions or program code or some data, etc. The memory 204 may be located inside the image detection device 200 or outside the image detection device 200, and is not limited. The processor 201 is configured to execute instructions stored in the memory 204 to implement an image detection method according to the following embodiment of the present application.
In one example, processor 201 may include one or more CPUs, such as CPU0 and CPU1 in fig. 2.
As an alternative implementation, the image detection device 200 comprises a plurality of processors, e.g. in addition to the processor 201 in fig. 2, a processor 205 may be included.
It should be noted that the constituent structures shown in fig. 2 do not constitute limitations of the respective apparatuses in fig. 1, and that the respective apparatuses in fig. 1 may include more or less components than those shown in fig. 2, or may combine some components, or may be arranged differently, in addition to those shown in fig. 2.
In the embodiment of the application, the chip system can be composed of chips, and can also comprise chips and other discrete devices.
Further, actions, terms, and the like, which are referred to between embodiments of the present application, are not limited thereto. The message names of interactions between the devices or parameter names in the messages in the embodiments of the present application are just an example, and other names may be used in specific implementations without limitation.
In order to clearly describe the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same item or similar items having substantially the same function and effect. It will be appreciated by those of skill in the art that the words "first," "second," and the like do not limit the amount and order of execution, and that the words "first," "second," and the like do not necessarily differ.
In the present application, the words "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or" describes an association of associated objects, meaning that there may be three relationships, e.g., A and/or B, and that there may be A alone, while A and B are present, and B alone, where A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a, b, or c) of a, b, c, a-b, a-c, b-c, or a-b-c may be represented, wherein a, b, c may be single or plural.
The image detection method provided by the embodiment of the application is described below with reference to the image detection system shown in fig. 1.
Fig. 3 is a schematic flow chart of an image detection method according to an embodiment of the present application, as shown in fig. 3, the method includes the following steps S301 to S303:
S301, acquiring an image to be detected.
The image to be detected may be an image corresponding to any video frame in the video data.
As a possible implementation manner, the image detection method may be based on using a screenshot tool to intercept a required image frame at a display interface to obtain an image to be detected.
The screenshot tool can be set according to requirements. For example, selenium or Puppeteer.
In some embodiments, the image to be detected may be a page where there is a dynamic refresh (e.g., a carousel view or animated content).
The image detection device can wait for the screenshot after the page dynamic content loading is completed by adding a time. Or the page changes in a short time can be photographed frame by frame and saved in GIF or video form.
After the image to be detected is acquired, the image detection device can directly store the image to be detected at a preset address.
The preset address may be set as needed. For example, it may be an enterprise data warehouse or a distributed file system (e.g., HDFS), etc.
In some embodiments, in order to reduce the memory load, the image detection device may compress the image to be detected, and store the compressed image to be detected at a preset address.
It should be noted that, in order to distinguish different images to be detected, the image detecting apparatus may store image metadata of the images to be detected. The image metadata may include screenshot time, resolution, task name, etc.
S302, detecting image parameters of the image to be detected according to a plurality of abnormality detection rules to obtain a detection result corresponding to each abnormality detection rule.
Wherein, different anomaly detection rules correspond to different image parameters. The abnormality detection rule includes at least one of a display content missing detection rule, a display content error detection rule, and a display effect abnormality detection rule.
Different anomaly detection rules are used to detect different display defects. For example, the display defects may include a first display defect, a second display defect, and a third display defect. The first display defect is used for indicating that the display content of the image to be detected is missing. The second display defect is used for indicating display content errors of the image to be detected. The third display defect is used for indicating that the display effect of the image to be detected is abnormal.
As one possible implementation manner, the image detection device may determine, for each of the plurality of image parameters, whether the image parameter matches a preset image parameter, determine, when the image parameter does not match the preset image parameter, a display defect corresponding to the image parameter as an abnormal item in which the image to be detected exists, and determine, when the image parameter matches the preset image parameter, a display defect corresponding to the image parameter as an abnormal item in which the image to be detected does not exist.
In an example, the types of the image parameters may include a first type image parameter, a second type image parameter, and a third type image parameter, and the image detection apparatus may determine whether the first type image parameter matches a preset first type image parameter, and determine a first display defect corresponding to the first type image parameter as an abnormal item existing in the image to be detected if the first type image parameter does not match the preset first type image parameter.
The image detection device may further determine whether the second type image parameter is matched with a preset second type image parameter, and determine a second display defect corresponding to the second type image parameter as an abnormal item existing in the image to be detected when the second type image parameter is not matched with the preset second type image parameter.
The image detection device may further determine whether a third type of image parameter is matched with a preset third type of image parameter, and determine a third display defect corresponding to the third type of image parameter as an abnormal item of the image to be detected when the third type of image parameter is not matched with the preset third type of image parameter.
In some embodiments, the image detection device may determine an image parameter in a target area of the image to be detected, and determine an abnormal item of the image to be detected according to the image parameter in the target area of the image to be detected.
The target area is a text display area of the image to be detected. The text display area may include text, charts, text boxes, and the like.
In one example, the image detection apparatus may acquire a target region of the image to be detected according to a pre-trained target detection model.
The pre-trained target detection model can be set as required. For example YOLOv, etc.
It should be noted that, according to the image parameters in the target area of the image to be detected, the specific description of determining the abnormal item of the image to be detected may refer to the description of the subsequent part, which is not repeated here.
After determining that the abnormal item exists in the image to be detected, the image detection device can also automatically generate a detection report with combined image and text based on the report template, and send the detection report to a manager.
Where the image to be detected is a screenshot in a patrol video, the detection report may include a specific description (question type, occurrence time, detection task) generated for each type of abnormal item, an abnormal region in the image to be detected (e.g., may be highlighted with a frame or mask), a repair suggestion, and the like.
The format of the detection report may be set as desired for ease of viewing. For example, PDF, HTML, etc. may be used.
It should be noted that, the notification manner of sending the detection report to the manager may include, but is not limited to, at least one of a sms, a mail, and an OA system.
In order to ensure timeliness and reliability of sending the detection report, the image detection device may send the detection report to a manager based on a real-time push mechanism.
The real-time pushing mechanism may send a detection report to a manager through a preset message queue. The preset message queue may be Kafka or rabitmq, etc.
S303, determining whether the image to be detected is abnormally displayed or not based on detection results corresponding to a plurality of abnormal detection rules.
As a possible implementation manner, the image detection device may determine that the image to be detected is displayed abnormally if any detection result corresponding to the plurality of abnormality detection rules indicates that the image to be detected is abnormal. And under the condition that detection results corresponding to the plurality of abnormality detection rules show that no abnormality exists in the image to be detected, determining that the image to be detected is normally displayed.
Based on the technical scheme provided by the application, the abnormal item of the image to be detected is determined according to the image parameters of the image to be detected. The image parameters are used for representing the actual display effect of the image to be detected. Therefore, whether the display image of the display screen is abnormal or not can be reflected based on the parameters corresponding to the actual display effect, and compared with the background data based on the display screen for analysis, the method and the device can more accurately determine the abnormality of the display image according to the parameter analysis of the actual display image of the display screen.
In one possible embodiment, as shown in fig. 4, in order to determine the abnormal item of the image to be detected, the present application may further include the following S401-S402.
S401, determining whether the image to be detected has a first display defect according to the mean value and the variance of the brightness of different pixels and the edge intensity.
The first display defect is used for indicating that the display content of the image to be detected is missing. For example, the display content absence may be that the target area is blank.
As a possible implementation manner, the image detection device may determine that the image to be detected has the first display defect in a case where the average value of the luminances of the different pixels is smaller than the luminance threshold, the variance of the luminances of the different pixels is smaller than the variance threshold, and the edge intensity is larger than the edge intensity threshold.
The brightness threshold, variance threshold, and edge intensity threshold may be set as needed. For example, the brightness threshold may be 300nit or the like. The variance threshold may be 0.1, etc., and the edge strength threshold may be 50, etc.
In determining the edge intensity, the image detection device may determine the edge intensity threshold using an edge detection algorithm.
The edge detection algorithm may be set as desired. For example, a Canny edge detection algorithm may be used.
In some embodiments, the image parameter may further include a page component, and the image detection device may determine that a template matching algorithm matches the page component in the image to be detected with the page component template, and determine whether the first display defect exists.
For example, if the matching degree of the page component in the image to be detected and the page component template is smaller than the matching degree threshold, determining that the page component is missing and the first display defect exists. And under the condition that the matching degree of the page component in the image to be detected and the page component template is greater than or equal to a matching degree threshold value, determining that the first display defect does not exist.
The page component templates may include, among other things, ORB feature points of page components (e.g., charts).
S402, determining the first display defect as a detection result corresponding to the display content deletion detection rule under the condition that the image to be detected has the first display defect.
In one example, the image detection may generate an abnormality report of the image to be detected after determining an abnormality item of the image to be detected, and the content of the abnormality report may include that a detection result corresponding to a display content deletion detection rule of the image to be detected is a first display defect.
In one possible embodiment, as shown in fig. 5, the image parameters include text content and corresponding text formats, and in order to determine whether the image to be detected is abnormally displayed, the present application may further include the following S501-S502.
S501, determining whether a second display defect exists in the image to be detected according to the text content of the target area and the corresponding text format.
The second display defect is used for indicating display content errors of the image to be detected. The text content may include words and values.
As a possible implementation manner, the image detection device may use an OCR tool to extract text content of the target area, and determine that the image to be detected has the second display defect if the text content of the target area does not match the preset text content or the text format does not match the preset text format.
And under the condition that the text content of the target area is matched with the preset text content and the text format is matched with the preset text format, determining that the image to be detected has a second display defect.
Wherein the OCR tool may be set as desired. For example TESSERACT, EASYOCR.
The following description is made on a determination process of whether the text content of the target area matches the preset text content or whether the text format matches the preset text format through sections 1-2:
1. whether the text content of the target area is matched with the preset text content or not.
The image detection means may determine whether the text format of the text content of the target area matches a preset text format based on the regular expression.
In one example, the preset text format corresponding to the regular expression may be \d+ (\\d+). The image detection device may determine that the text format of the text content of the target area matches the preset text format in the case where the text format of the text content of the target area is \d+ (\d+), and determine that the text format of the text content of the target area does not match the preset text format in the case where the text format of the text content of the target area is not \d+ (\d+).
In one example, the image detection device may construct composeapplication the object from the application parameters entered by the user. Capability information is generated by means of dynamic proxy. Further, authentication (such as computation of token) and message assembly (such as assembly of request header and request body) are completed, a calling file is run, and an interface to be called is called.
2. And determining whether the text format of the target area is matched with the preset text format.
The image detection device can set a reasonable preset range through service logic, and determine the preset range as preset text content, and determine that the text format of the target area is matched with the preset text format when the text content of the target area is within the preset range, and determine that the text format of the target area is not matched with the preset text format when the text content of the target area is outside the preset range.
For example, the preset range may be 0% -100%, where the text content of the target area is within 0% -100%, the text format of the target area is determined to be matched with the preset text format, and where the text content of the target area is outside 0% -100%, the text format of the target area is determined to be not matched with the preset text format.
S502, determining the second display defect as a detection result corresponding to the display content error detection rule under the condition that the image to be detected has the second display defect.
In one example, the image detection apparatus may generate an abnormality report of the image to be detected after determining the abnormal item of the image to be detected, and the content of the abnormality report may include that the abnormal item of the image to be detected is the second display defect.
In one possible embodiment, as shown in fig. 6, the image parameters include resolution, color histogram, sharpness, and contrast, and in order to determine the abnormal item of the image to be detected, the present application may further include the following S601-S602.
S601, determining whether a third display defect exists in the image to be detected according to the resolution value of the image to be detected, the proportion of different colors in the color histogram, the definition value and the contrast value.
The third display defect is used for indicating that the display effect of the image to be detected is abnormal.
As a possible implementation manner, the image detection device may determine that the image to be detected has the third display defect if the image parameter meets the first preset condition.
The first preset condition satisfies at least one of a resolution value of the target area being less than a resolution threshold, a deviation value between a proportion of different colors in the color histogram and a proportion of preconfigured colors being greater than a deviation threshold, a sharpness value of the target area being less than a sharpness threshold, and a contrast value of the target area being less than a sharpness threshold.
The resolution threshold, the deviation threshold, and the sharpness threshold may be set as needed.
The procedure for determining the resolution value, color histogram, sharpness value, contrast value of the target area is described in the following sections 1-4:
1. Sharpness value of the target area.
The image detection device can calculate the definition value of the image to be detected through Laplace transformation.
And when the definition value is smaller than the set threshold value, judging that the image component to be detected has a blurring problem, namely the display effect corresponding to the third display defect is abnormal. And when the definition value is larger than or equal to the set threshold value, judging that the image component to be detected has no blurring problem.
For example, the image detection device may convert an image to be detected into a gray scale map, transform the gray scale map using a Laplace operator, calculate a Laplace-transformed image variance, and determine the Laplace-transformed image variance as the resolution value of the target area.
2. Color histogram of the target area.
The image detection device may calculate a color histogram of the image by reading the image to be detected using a color histogram acquisition tool.
The color histogram acquisition tool may be set as desired. For example, openCV library, etc.
And under the condition that the deviation value between the ratio of different colors in the color histogram of the target area and the ratio of the preconfigured colors is larger than the deviation threshold value, judging that the image component to be detected has abnormal color display problem, namely abnormal display effect corresponding to the third display defect. And when the deviation value between the ratio of different colors in the color histogram of the target area and the ratio of the preconfigured colors is smaller than or equal to a set threshold value, judging that the image component to be detected has no color display abnormality problem.
In one example, if the color deviation value is outside a set range (e.g., RGB deviation > 20%), a color anomaly is determined.
3. A resolution value of the target area.
In one example, the image detection device may determine a product of the pixel density and the image size and determine the product of the pixel density and the image size as the resolution value of the target region.
And confirming whether the resolution meets the requirement or not through pixel density calculation.
4. Contrast value of the target region.
In one example, the image detection device may use a contrast value of a target region of a contrast detection algorithm.
For example, the contrast detection algorithm may be a Weber contrast algorithm or the like.
S602, determining the third display defect as a detection result corresponding to the abnormal display effect rule when the third display defect exists in the image to be detected.
In one example, the image detection device may generate an abnormality report of the image to be detected after determining the abnormal item of the image to be detected, and the content of the abnormality report may include that the detection result corresponding to the display effect abnormality rule is a third display defect.
The embodiment of the application can divide the functional modules or functional units of the image detection device according to the method example, for example, each functional module or functional unit can be divided corresponding to each function, or two or more functions can be integrated in one processing module. The integrated modules may be implemented in hardware, or in software functional modules or functional units. The division of the modules or units in the embodiment of the present application is schematic, which is merely a logic function division, and other division manners may be implemented in practice.
In the case of dividing the respective functional modules by the respective functions, fig. 7 shows a schematic configuration of an image detection apparatus 700, which may be an image detection apparatus or a chip, a processor, or the like applied to the image detection apparatus. The image detection apparatus 700 may be used to perform the functions of the image detection apparatus as referred to in the above-described embodiments. The image detection device 700 shown in fig. 7 may include an acquisition unit 701, a processing unit 702, the acquisition unit 701 configured to acquire an image to be detected, the processing unit 702 configured to detect image parameters of the image to be detected according to a plurality of anomaly detection rules to obtain a detection result corresponding to each anomaly detection rule, different anomaly detection rules corresponding to different image parameters, and the processing unit 702 further configured to determine whether the image to be detected is displayed abnormally based on the detection results corresponding to the plurality of anomaly detection rules.
Optionally, in the case that the anomaly detection rule is a display content missing detection rule, the processing unit 702 is specifically configured to determine whether a first display defect exists in the image to be detected according to the mean value and the variance of the brightness of different pixels and the edge intensity, where the first display defect is used to indicate that the display content of the image to be detected is missing, and in the case that the first display defect exists in the image to be detected, determine the first display defect as a detection result corresponding to the display content missing detection rule.
Optionally, the processing unit 702 is further configured to determine that the image to be detected has the first display defect if the average value of the luminance of the different pixels is smaller than the luminance threshold, the variance of the luminance of the different pixels is smaller than the variance threshold, and the edge intensity is larger than the edge intensity threshold.
Optionally, the image parameters include text content and a corresponding text format, and the processing unit 702 is specifically further configured to determine, according to the text content and the corresponding text format of the target area, whether the image to be detected has a second display defect, where the second display defect is used to indicate that the image to be detected has a display content error, and determine, when the image to be detected has the second display defect, the second display defect as a detection result corresponding to the display content error detection rule.
Optionally, the processing unit 702 is further configured to determine that the image to be detected has the second display defect if the text content of the target area does not match the preset text content or the text format does not match the preset text format.
Optionally, the image parameters include resolution, color histogram, sharpness, and contrast, and the processing unit 702 is specifically further configured to determine whether a third display defect exists in the image to be detected according to the resolution value of the image to be detected, the ratio of different colors in the color histogram, the sharpness value, and the contrast value, where the abnormality detection rule is a display effect abnormality rule, and determine the third display defect as a detection result corresponding to the display effect abnormality rule when the third display defect exists in the image to be detected.
Optionally, the processing unit 702 is further configured to determine that the image to be detected has a third display defect if the image parameter satisfies a first preset condition, where the first preset condition satisfies at least one of a resolution value of the target area being smaller than a resolution threshold, a deviation value between a ratio of different colors in the color histogram and a ratio of preconfigured colors being larger than a deviation threshold, a sharpness value of the target area being smaller than a sharpness threshold, and a contrast value of the target area being smaller than a contrast threshold.
The embodiment of the application also provides a computer readable storage medium. All or part of the flow in the above method embodiments may be implemented by a computer program to instruct related hardware, where the program may be stored in the above computer readable storage medium, and when the program is executed, the program may include the flow in the above method embodiments. The computer readable storage medium may be an internal storage unit of the image detection apparatus (including the data transmitting end and/or the data receiving end) of any of the foregoing embodiments, for example, a hard disk or a memory of the image detection apparatus. The computer-readable storage medium may be an external storage device of the terminal apparatus, for example, a plug-in hard disk, a smart card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, or a flash memory card (FLASH CARD) provided in the terminal apparatus. Further, the computer-readable storage medium may further include both the internal storage unit and the external storage device of the image detection apparatus. The computer-readable storage medium is used for storing the computer program and other programs and data required by the image detection apparatus. The above-described computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
It should be noted that the terms "first" and "second" and the like in the description, the claims and the drawings of the present application are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
It should be understood that, in the present application, "at least one (item)" means one or more, and "a plurality" means two or more, and "at least two (items)" means two or three or more, and/or "for describing association of associated objects, it means that three kinds of relationships may exist, for example," a and/or B "may mean that only a exists, only B exists, and three cases of a and B exist at the same time, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one of a, b or c may represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
From the foregoing description of the embodiments, it will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of functional modules is illustrated, and in practical application, the above-described functional allocation may be implemented by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts shown as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application. The storage medium includes various media capable of storing program codes such as a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk.
The present application is not limited to the above embodiments, and any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.