Disclosure of Invention
The invention provides a fault monitoring method, a system and a computer medium for a camera collaborative knowledge graph, which are used for solving the technical problem that the traditional knowledge graph in the prior art cannot realize automatic fault early warning.
In order to solve the technical problems, the technical scheme provided by the invention is as follows: the fault monitoring method of the camera collaborative knowledge graph comprises the following steps:
s1 monitoring and shooting a plurality of components of the equipment by using a plurality of cameras and generating component images, wherein each camera monitors one component of the shooting equipment;
s2, comparing the part image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to the comparison result;
the knowledge graph comprises entities and relations, and is a network-shaped image knowledge graph formed by the entities through the relations; the entities comprise a first image entity, a second image entity and a conclusion entity;
the first image entity is a component image of the component in a normal state; the second image entity is a component image when the running state of the component changes; the conclusion entity comprises fault types corresponding to the component images when the running state of the component changes and information of treatment opinions;
preferably, the relationship includes a first relationship between the first image entity and the second image entity and a second relationship between the second image entity and the conclusion entity, the first relationship represents a change reason of the change of the operating state of the component, and the second relationship represents a pointing relationship from the second image entity to the conclusion entity when the change of the operating state of the component may cause a fault.
Preferably, the construction of the knowledge-graph comprises the following steps:
collecting data, wherein collecting data comprises collecting part images and video data of each part of the monitored equipment in different operating states and using description materials of the monitored equipment;
sorting out the sequence and the change reason of the image change of each component in various running states according to the materials, and giving out corresponding fault types and treatment opinions after the corresponding running states of each component;
generating a first image entity, a second image entity and a first relation according to the sequence of component image changes of each component in various running states and the change reasons; giving out corresponding fault types and processing opinions according to corresponding running states of all the parts to generate a second relation and a conclusion entity;
and constructing and storing an initial knowledge graph of the equipment according to the generated first image entity, the second image entity, the conclusion entity, the first relation and the second relation.
Preferably, the constructing of the knowledge-graph further comprises updating the knowledge-graph, and the updating the knowledge-graph comprises the following steps:
each camera monitors a component of the shooting device;
the part image of the part shot by the camera
With the last part image
Comparing, if the part image is
Image feature vector of
And the last part image
Image feature vector of
Is a distance of
Greater than a first threshold
The part image of this time
Comparing the image feature vector with the feature vectors of all second image entities of the component in the knowledge graph, and if not, ending updating the knowledge graph;
if the part image is present
Image feature vector of
The maximum of the distances to the feature vectors of all second image entities of the component in the knowledge-graph is larger than a second threshold
The current part image
Adding a knowledge graph as a second image entity, establishing a relationship and a corresponding conclusion entity, and if not, finishing updating the knowledge graph;
and storing the updated knowledge graph.
Preferably, the method further comprises the steps of judging whether the component image is compared with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to the comparison result, wherein the judging process comprises the following steps:
the part image of this time
With the last part image
Comparing, if the part image is
Feature vector of
With the last part image
Feature vector of
Is a distance of
Greater than a third threshold
Comparing the part image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to a comparison result; otherwise, the comparison is not performed.
Preferably, comparing the component image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to the comparison result includes:
each camera
Part of monitoring shooting equipment
;
The part image of the part shot by the camera
And the parts in the knowledge map
Comparing all second image entities to obtain the current part image
Feature vector of
And the parts in the knowledge map
The distance minimum value of the feature vector relative ratio of all the second image entities is obtained to obtain the second image entity corresponding to the distance minimum value
And a second image entity
Feature vector of
If the part image is present
Feature vector of
With the second image entity
The feature vector of
Is less than a fourth threshold value
Then the second image entity corresponding to the minimum distance value is determined
Marking as
And judging the second image entity corresponding to the minimum distance value
Whether connected to a concluding entity, if so, checking a second map of other components connected to the entity's conclusionWhether or not the image entities are all marked
Otherwise, ending; if the second image entities of other components connected to the entity conclusion are all marked as
And outputting a conclusion entity, otherwise, ending.
The embodiment of the present invention further provides a computer system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the method are implemented.
Embodiments of the present invention also provide a computer medium, on which a computer program is stored, and when the program is executed by a processor, the steps in the above method are implemented.
The invention has the following beneficial effects:
1. the fault monitoring method of the camera collaborative knowledge graph uses a plurality of cameras to monitor and shoot a plurality of components of equipment and generate component images, each camera monitors one component of the shooting equipment, then the component images are compared with second image entities in the knowledge graph, corresponding conclusion entities are obtained according to comparison results, wherein the knowledge graph comprises entities and relations, the knowledge graph is a network-shaped image knowledge graph formed by the entities through the relations, the entities comprise a first image entity, a second image entity and conclusion entities, fault early warning can be given before or when a fault occurs, and meanwhile, the conclusion entities are obtained to provide reference for the assignment of a fault component maintenance plan so as to avoid the loss of the equipment on production operation caused by the fault.
2. According to the invention, the current part image and the last part image shot by a certain part are compared by the camera, whether the current part image is compared with the second image entity in the knowledge map is judged, and if the comparison result of the current part image and the last part image is smaller than the third threshold value, the next comparison is not carried out, so that the calculation steps of the method can be greatly simplified, and the efficiency of fault monitoring and prediction is improved.
3. According to the method, the image knowledge graph is adopted, the second entity image is compared with the component image shot by the camera, the defect that ambiguity may exist in named entities in the text knowledge graph can be avoided, in addition, a large amount of text information is needed for constructing the text knowledge graph, the construction steps are complex, and the image knowledge graph is easy to construct.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the accompanying drawings.
Detailed Description
The embodiments of the invention will be described in detail below with reference to the drawings, but the invention can be implemented in many different ways as defined and covered by the claims.
Referring to fig. 1, the method for monitoring the fault of the camera collaborative knowledge graph includes:
s1 monitoring and shooting a plurality of components of the equipment by using a plurality of cameras and generating component images, wherein each camera monitors one component of the shooting equipment;
s2, comparing the part image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to the comparison result;
the knowledge graph comprises entities and relations, and is a network-shaped image knowledge graph formed by the entities through the relations; the entities include a first image entity, a second image entity and a conclusion entity.
It should be noted that the knowledge graph of the invention is different from the traditional text knowledge graph in that the knowledge graph of the invention is an image knowledge graph, and named entities in the text knowledge graph may have the disadvantage of ambiguity.
The first image entity is a component image of the component in a normal state; the second image entity is a component image when the running state of the component changes; the conclusion entity comprises fault types corresponding to the component images when the running state of the component changes and information of treatment opinions; in practice, the conclusion entity also includes information about what component operating state has changed, i.e., the cause of the fault.
In this embodiment, the relationship includes a first relationship between the first image entity and the second image entity and a second relationship between the second image entity and the conclusion entity, the first relationship represents a change reason of the change of the operating state of the component, and the second relationship represents a pointing relationship from the second image entity to the conclusion entity when the change of the operating state of the component may cause a fault.
Optionally, the constructing of the knowledge graph comprises the following steps:
collecting data, wherein the collecting data comprises collecting data such as part images, videos and the like of each part of the monitored equipment in different running states and using description materials of the monitored equipment;
sorting out the sequence and the change reason of the image change of each component in various running states according to the materials, and giving out corresponding fault types and treatment opinions after the corresponding running states of each component;
generating a first image entity, a second image entity and a first relation according to the sequence of component image changes of each component in various running states and the change reasons; giving out corresponding fault types and processing opinions according to corresponding running states of all the parts to generate a second relation and a conclusion entity;
and constructing and storing an initial knowledge graph of the equipment according to the generated first image entity, the second image entity, the conclusion entity, the first relation and the second relation.
Optionally, the constructing the knowledge graph further includes updating the knowledge graph, and the updating the knowledge graph includes the following steps:
each camera monitors a component of the shooting device;
the part image of the part shot by the camera
With the last part image
Comparing, if the part image is
Image feature vector of
And the last part image
Image feature vector of
Is a distance of
Greater than a first threshold
The part image of this time
Comparing the image feature vector with the feature vectors of all second image entities of the component in the knowledge graph, and if not, ending updating the knowledge graph;
if the part image is present
Image feature vector of
The maximum of the distances to the feature vectors of all second image entities of the component in the knowledge-graph is larger than a second threshold
The current part image
Adding a knowledge graph as a second image entity, establishing a relationship and a corresponding conclusion entity, and if not, finishing updating the knowledge graph;
and storing the updated knowledge graph.
In an optional implementation manner of this embodiment, the method further includes determining whether to compare the component image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to a comparison result, where the determining process includes the following steps:
the part image of this time
With the last part image
Comparing, if the part image is
Feature vector of
With the last part image
Feature vector of
Is a distance of
Greater than a third threshold
Comparing the part image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to a comparison result; otherwise, the comparison is not performed.
Wherein, comparing the component image with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to the comparison result comprises:
each camera
Part of monitoring shooting equipment
;
The part image of the part shot by the camera
And the parts in the knowledge map
Comparing all second image entities to obtain the current part image
Feature vector of
And the parts in the knowledge map
The distance minimum value of the feature vector relative ratio of all the second image entities is obtained to obtain the second image entity corresponding to the distance minimum value
And a second image entity
Feature vector of
If the part image is present
Feature vector of
With the second image entity
The feature vector of
Is less than a fourth threshold value
Then the second image entity corresponding to the minimum distance value is determined
Marking as
And judging the second image entity corresponding to the minimum distance value
If the second image entities of the other components connected with the entity conclusion are all marked as being connected with the conclusion entity or not, if so, checking whether the second image entities of the other components connected with the entity conclusion are all marked as being connected with the conclusion entity
Otherwise, ending; if the second image entities of other components linked to the entity conclusion are all marked as
Then the conclusion entity is output, otherwise, the process is finished.
It should be noted that the equipment failure may cause the change of the operation status of a plurality of components at the same time, so that the above scheme needs to check whether the second image entities of other components connected to the conclusion of the entity are all marked as being all the same
If the second image entity of only one component is marked as
It may be a change in operating state due to an accidental situation, where the device is deemed to have failed, if the second image entities of the other components connected to the entity conclusion are all marked as such
At this point, the device may be deemed to be malfunctioning and the conclusion entity may be output.
Further, it goes to the step "when checking whether the second image entities of the other components connected to the entity conclusion are all marked as being all marked as" when checking "whether the second image entities are all marked as" when checking "the second image entities of the other components connected to the entity conclusion
When there is one or more marks in the second image entity of the other component
Can also be based on the label
The number of second image entities of (a) makes a failure prediction on the degree of device failure.
Further, the distances between the feature vectors of all the second image entities connected to the entity conclusion and the corresponding feature vectors of the first image entities can be calculated, a plurality of groups of distances are obtained at the moment, and the plurality of groups of distances are weighted and summed according to different importance of corresponding components in the equipment so as to predict the fault degree of the equipment.
Example 2:
a computer system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of embodiment 1 when executing the computer program.
Example 3:
a computer medium on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of embodiment 1.
The fault monitoring method of the camera collaborative knowledge graph comprises the steps of monitoring and shooting a plurality of components of equipment by using a plurality of cameras, generating component images, monitoring and shooting one component of the equipment by each camera, comparing the component images with a second image entity in the knowledge graph, and obtaining a corresponding conclusion entity according to a comparison result, wherein the knowledge graph comprises an entity and a relation, the knowledge graph is a network-shaped image knowledge graph formed by the entity and the entity through the relation, the entity comprises a first image entity, a second image entity and a conclusion entity, fault early warning can be given before or when a fault occurs, and the conclusion entity is obtained to provide reference for the specification of a fault component maintenance plan so as to avoid the loss of production operation caused by the fault of the equipment. And the camera is used for comparing the part image shot by a certain part with the last part image, judging whether the part image is compared with a second image entity in the knowledge graph or not, if the comparison result of the part image and the last part image is smaller than a third threshold value, the next comparison is not carried out, the calculation steps of fault monitoring can be greatly simplified, and the efficiency of fault monitoring and prediction is improved.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.