Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that, in the present disclosure, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising" does not exclude the presence of additional identical elements in a process, method, article, or system that comprises the element. In addition, the meaning of "and/or" as it appears throughout includes three parallel schemes, for example "A and/or B", including the A scheme, or the B scheme, or the scheme where A and B are satisfied simultaneously.
In the present invention, if there is a description referring to "first", "second", etc., the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the present invention, suffixes such as "module", "part" or "unit" used for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination.
The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
Analysis of the prior art shows that with the development of internet technology and the popularization of cloud service technology, the contradiction between big data sharing and privacy protection is increasingly stronger. Retrieval is a common function in big data scenarios, typically implemented by a set of retrieval systems. In the current background, the requirements of guaranteeing the privacy security of data and improving the data retrieval capability are important for a retrieval system. Meanwhile, along with the improvement of the technical level and the development of the spirit pursuit of people, the concepts of privacy and data are continuously enriched. Privacy is not limited traditional sensitive information such as identity information and financial information, life, travel and other data are increasingly endowed with privacy concepts, and the data gradually evolve from traditional data concepts such as texts, tables and the like to concepts of coexistence of multiple modes such as video, audio, geography and the like.
Based on the method, the retrieval of the cross-modal data becomes a rigid requirement under the cloud service and big data age, and the semantic representation of the cross-modal data is a key component of a cross-modal data retrieval system.
The cross-modal semantic characterization technology is to perform joint modeling on data of different modalities, and the obtained model can encode the data of different modalities to obtain keywords, so that the keywords of the data of different modalities of the same semantic have higher relevance and can be explicitly calculated. For example, a text mode that "many people are in meeting room" and a meeting room picture of an image mode are respectively input into a model for coding, and the semantic relevance of the obtained text keywords is higher than that of the image keywords and the keywords obtained by data coding of other semantic contents.
The cross-modal privacy semantic representation technology is a technology for adding privacy protection requirements on the basis of the cross-modal semantic representation technology, and the technology requires that a retrieval system can encode cross-modal data to obtain a secret state keyword on the premise that plaintext data is not uploaded to a cloud server, so that the retrieval of privacy semantics is carried out according to the secret state keyword.
The existing cross-modal privacy semantic characterization technology has some problems, such as:
1. keyword extraction of cross-modal data is difficult;
2. Few supported modalities and difficult to amplify;
3. semantic relevance between the secret keywords is poor.
In view of the technical problem of poor semantic relevance among the secret key words in the prior art, the invention provides a cross-mode privacy semantic characterization method, and the general thought is as follows:
Obtaining multi-modal data, obtaining corresponding text data according to the multi-modal data, extracting and encrypting keywords of the text data to obtain a secret key word, dividing the preset knowledge graph according to the secret key word to obtain a secret sub-graph, and embedding the secret sub-graph to obtain a secret representation vector corresponding to the secret key word to obtain a semantic representation result of the multi-modal data.
According to the technical scheme, after corresponding text data is obtained according to the obtained multi-mode data, keyword extraction and encryption are carried out on the text data to obtain a secret state keyword, data privacy can be protected, data safety is guaranteed, a secret state sub-graph is obtained by dividing a preset knowledge graph according to the secret state keyword, semantic information of the secret state keyword is effectively expanded in a sub-knowledge graph mode, semantic concepts of the secret state keyword can be more comprehensively expressed, strong correlation with the secret state keyword is kept, and a secret state representation vector corresponding to the secret state keyword is obtained by carrying out graph embedding on the secret state sub-graph, so that a semantic representation result of the multi-mode data is obtained, more abundant semantic information is encoded, and semantic information correlation among the secret state keywords is guaranteed. The invention realizes semantic representation of cross-modal data on the premise of guaranteeing user data privacy, not only can ensure semantic association among the secret state keywords, but also can provide accurate semantic representation for subsequent retrieval of privacy semantics, and simultaneously, supported modal data can be dynamically increased and decreased according to business requirements, so that the invention has more flexibility.
The method, the device, the equipment and the storage medium for cross-modal privacy semantic characterization provided by the invention are described in detail below by means of specific examples and implementation modes with reference to the accompanying drawings.
Example 1
Referring to the flow diagram of fig. 1, a first embodiment of the cross-modal privacy semantic representation method of the present invention is presented, which is applied to a cross-modal privacy semantic representation device. The cross-mode privacy semantic representation device refers to terminal devices or network devices capable of realizing network connection, and can be terminal devices such as mobile phones, computers and tablet computers, or network devices such as servers and cloud platforms. The method can also be applied to a semantic retrieval system comprising a terminal device and a network device, when the method is applied to the semantic retrieval system, part of steps of the method can be performed on the terminal device, and after the obtained result is sent to the network device, the rest part of steps can be performed on the network device.
Fig. 2 is a schematic diagram of a hardware structure of a cross-modal privacy semantic representation device. The device may include a processor 1001, such as a CPU (Central Processing Unit ), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Those skilled in the art will appreciate that the hardware architecture shown in fig. 2 does not constitute a limitation of the cross-modality privacy semantic characterization device of the present invention, and may include more or fewer components than illustrated, or may combine certain components, or a different arrangement of components.
In particular, communication bus 1002 is configured to enable connective communication between these components;
The user interface 1003 is used for connecting and communicating data with the client, the user interface 1003 may include an output unit such as a display screen, an input unit such as a keyboard, and optionally, the user interface 1003 may include other input/output interfaces such as a standard wired interface, a wireless interface;
The network interface 1004 is used to connect to and communicate data with a background server, and the network interface 1004 may include an input/output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface;
The memory 1005 is used for storing various types of data, which may include, for example, instructions of any application program or method in the cross-modal privacy semantic characterization device, and application program related data, the memory 1005 may be a high-speed RAM memory, or a stable memory, such as a disk memory, and optionally, the memory 1005 may be a storage device independent of the processor 1001;
specifically, with continued reference to fig. 2, the memory 1005 may include an operating system, a network communication module, a user interface module, and a cross-mode privacy semantic representation program, where the network communication module is mainly used to connect with a server and perform data communication with the server;
The processor 1001 is configured to call a cross-modality privacy semantic characterization program stored in the memory 1005, and perform the following operations:
Acquiring multi-mode data;
obtaining corresponding text data according to the multi-mode data;
extracting and encrypting the keywords of the text data to obtain a secret state keyword;
dividing the preset knowledge graph according to the secret state keywords to obtain a secret state subgraph;
And performing graph embedding on the secret sub-graph to obtain a secret representation vector corresponding to the secret key word so as to obtain a semantic representation result of the multi-mode data.
Based on the above-mentioned cross-mode privacy semantic representation device, the cross-mode privacy semantic representation method of the present embodiment is described in detail below with reference to the flowchart shown in fig. 1. The method may comprise the steps of:
step S110, multi-mode data is acquired.
Multimodal data refers to data information having a plurality of different modalities, such as video data, audio data, image data, text data, and the like. The user may manually input through the user interface of the device or may receive data transmitted by other devices through the network interface of the device. The data acquired here may be data including a plurality of modes, or may include data of only one mode, and only data including a plurality of modes is described here as an example.
And step S120, obtaining corresponding text data according to the multi-mode data.
For the multi-modal data acquired in step S110, when the multi-modal data includes non-text mode data such as images, audio or video, mode conversion is required, the data is converted from the non-text mode to the text mode, and when the multi-modal data includes text mode data, mode conversion is not required, and the text mode data is directly used as the text data finally obtained in the step. When the non-text mode data is subjected to mode conversion, the model obtained by training the characteristics of the non-text mode data such as pictures, audio or video can be combined to perform text recognition, the text describing the non-text mode data is obtained, and the accuracy of text description of the non-text mode data can be improved.
In the implementation process, when the multi-modal data contains image data, the text data corresponding to the image data can be obtained by utilizing an image recognition technology, particularly, text description can be conducted by utilizing a pre-trained deep learning model, when the multi-modal data contains audio data, the text data corresponding to the audio data can be obtained by utilizing a voice recognition technology, particularly, text description can be conducted by utilizing a DEEP SPEECH V and other deep learning models, and when the multi-modal data contains Video data, the text data corresponding to the Video data can be obtained by utilizing a text description generation technology, particularly, text description can be conducted by utilizing a pre-trained Video BERT and other text generation models. It should be noted that, no matter how many kinds of data are included in the multi-modal data, after the non-text modal data are respectively converted into text modal data, all the text modal data, that is, the data including the original text modal data and the converted text modal data are summarized, so as to obtain the text data corresponding to the acquired multi-modal data.
And step S130, extracting and encrypting the keywords of the text data to obtain the secret state keywords.
The text data is extracted by keywords because the text data may have more or unnecessary words which easily influence semantic characterization accuracy and the like, and the extracted keywords are encrypted to obtain the secret keywords of the multi-mode data because the privacy of the data is ensured. The keyword extraction may be performed by a Text Rank algorithm, and the extracted keyword may be encrypted by a symmetric encryption algorithm, for example, DES (Data Encryption Standard ) algorithm, RC (RIVEST CIPHER, stream encryption) algorithm, blowFish algorithm (block cipher algorithm), and the like. It should be noted that, when the method of this embodiment is performed on the terminal device or the network device, the processing may be continued on the secret key words to obtain the semantic representation result of the cross-modal data, and when the method of this embodiment is performed on the semantic retrieval system including the terminal device and the network device, the terminal device may send the obtained secret key words to the network device after executing steps S110 to S130, and the network device may process the secret key words after receiving the secret key words to obtain the semantic representation result of the cross-modal data, where the sending and receiving processes are both encrypted, so that the privacy of the data may still be ensured.
And step 140, dividing the preset knowledge graph according to the secret key words to obtain a secret sub-graph.
The preset knowledge graph is a graph structure in which vocabulary entities are in the form of nodes, and relations among the entities are visually described in the form of edges, so that association among the vocabularies can be explicitly described. The preset knowledge graph is an encrypted Wikidata open source knowledge graph, wherein the encryption mode and the secret key are consistent with the encryption mode and the secret key for encrypting the keywords in the step S30, so that the processing of the secret key after the secret key is matched can be ensured, and the privacy of data is ensured. According to the secret key words, the preset knowledge graph is divided, namely the secret key words are matched with nodes in the preset knowledge graph, a sub-knowledge graph centered on the secret key words is obtained according to the set clipping distance, the sub-knowledge graph comprises entities corresponding to the secret key words, other entities in the set clipping distance range around the entities and a set of edges representing the association relationship between the entities, and the sub-knowledge graph obtained by clipping is also in an encryption state because the preset knowledge graph is in the encryption state, so that the secret sub-graph is obtained. The number of the secret state subgraphs is consistent with the number of the secret state keywords, that is to say, the secret state subgraphs are in one-to-one correspondence with the secret state keywords.
The sub-knowledge graph with the secret key words as the center is used as the expression of the secret key words, and the secret key words are characterized in a combined mode by using the entity and the relation with extremely strong correlation with the secret key words, so that the semantic concepts of the secret key words can be expressed more comprehensively, the semantic concepts of the secret key words can be accurately characterized by other related entities and the corresponding relation, and the secret key words which are easy to be confused have more obvious distinguishing degrees.
And step S150, performing graph embedding on the secret sub-graph to obtain a secret representation vector corresponding to the secret key word so as to obtain a semantic representation result of the multi-mode data.
Each secret key word can correspondingly obtain a secret sub-graph, after the secret sub-graph is obtained in step S140, the secret sub-graph is subjected to graph embedding operation based on a random walk algorithm to obtain a secret representation vector of the secret representation vector, namely, a secret representation vector corresponding to the secret key word, and the vector is the semantic representation of the corresponding secret key word. And respectively acquiring corresponding secret state subgraphs for all secret state keywords, respectively performing graph embedding operation to obtain secret state representation vectors, and then obtaining a vector set, namely a semantic representation result of the multi-mode data in the embodiment after summarizing. The semantic characterization result can be directly used for inputting semantic retrieval to obtain a retrieval result, so that the retrieval of cross-modal data is realized, for example, a user can directly input multi-modal data on the device, after the semantic characterization result is obtained by the method of the embodiment, the semantic characterization result is input into a retrieval system to perform retrieval, and the final cross-modal retrieval result can be obtained.
According to the cross-modal privacy semantic representation method, after corresponding text data is obtained according to the obtained multi-modal data, keyword extraction and encryption are carried out on the text data to obtain a multi-modal keyword, data privacy can be protected, data safety is guaranteed, a pre-set knowledge graph is segmented according to the multi-modal keyword to obtain a multi-modal sub-graph, semantic information of the multi-modal keyword is effectively expanded in the form of a sub-knowledge graph, semantic concepts of the multi-modal keyword can be more comprehensively expressed, strong correlation with the multi-modal keyword is kept, and a graph embedding is carried out on the multi-modal sub-graph to obtain a multi-modal representation vector corresponding to the multi-modal keyword, so that a semantic representation result of the multi-modal data is obtained, more abundant semantic information is encoded, and semantic information correlation among the multi-modal keywords is guaranteed. The invention realizes semantic representation of cross-modal data on the premise of guaranteeing user data privacy, not only can ensure semantic association among the secret state keywords, but also can provide accurate semantic representation for subsequent retrieval of privacy semantics, and simultaneously, supported modal data can be dynamically increased and decreased according to business requirements, so that the invention has more flexibility.
Example two
Based on the same inventive concept, referring to fig. 3 to 5, a second embodiment of the cross-modal privacy semantic representation method of the present invention is provided, the method is applied to cross-modal privacy semantic representation equipment, the method can be further applied to a cross-modal privacy semantic retrieval system running on the equipment, and after the system performs privacy semantic representation on multi-modal data by the method, the system performs privacy semantic retrieval according to the semantic representation result to obtain a semantic retrieval result. The semantic search is a process of automatically inquiring and extracting related information from an information source according to requirements under the condition of understanding the relation between the semantic and the vocabulary by correctly analyzing the grammar format, and the privacy semantic search can be based on privacy data for searching, so that the security of the privacy data is further required.
The cross-modal privacy semantic characterization method of the present embodiment is described in detail below with reference to the flowchart shown in fig. 3. The method may comprise the steps of:
step S210, multi-mode data is acquired.
Specifically, the multi-modal data includes data information of at least two different modalities. Here, the modal data types in the multi-modal data are not limited, the modal data quantity is also not limited, and the modal data types and the modal data quantity can be automatically increased or decreased according to the service requirements. The present embodiment is illustrated with multimodal data including three modality data of voice, video, text, and english as an example language.
And step 220, obtaining corresponding text data according to the multi-mode data.
In the implementation process, a text generation model can be utilized to obtain corresponding text data according to the multi-mode data. Specifically, multi-modal data is used as input of a text generation model, and then each modal data is respectively input into a corresponding text generation model to respectively output the text modal data, so that text data describing the multi-modal data is obtained.
Specifically, step S220 may include:
Step S221, when the multi-modal data comprises first modal data of a voice mode, converting the first modal data into first text data by utilizing a voice recognition technology.
Among them, the speech recognition technology is a technology of converting data including human language sounds into text and constructing a mapping relationship between the text and the speech. The mode conversion of non-text mode data is the key of cross-mode data semantic representation and retrieval, in this embodiment, DEEP SPEECH V models can be used as voice recognition models, the models can be trained by using voice data of set language, and English voice data is used for model training, so that voice recognition models are obtained through training. In the implementation process, the first mode data is used as the input of the model, and the corresponding text description is directly output.
Step S222, when the multi-modal data comprises second-modal data of a video modality, converting the second-modal data into second text data by using a trained text generation model.
The trained text generation model is trained based on a data set with text description and video content as training data. In the implementation process, a model is generated by using the trained text, the second mode data is used as the input of the model, and the corresponding text description is directly output.
In this embodiment, the text generation model adopts a Video BERT model, and the Video BERT model is trained by using the BERT model in the language model and a self-supervision learning training method, and when the text generation model is trained, a dataset including text description data and Video content data is used as training data. The training method comprises the steps of firstly extracting feature vectors from video content data, carrying out discretization processing on the feature vectors through a clustering method to construct visual words, then combining text description data to form cross-modal words, then deriving visual marks (token) and linguistic marks from the cross-modal words, inputting the visual marks and the linguistic marks into a text generation model to be trained, and learning bidirectional joint distribution on the mark sequences by the model to construct a mapping relation between the visual marks and the linguistic marks, so as to obtain the trained text generation model.
And S223, when the multi-modal data comprises third modal data of a text modality, directly determining the third modal data as third text data.
The text data is used as text mode data, processing is not needed, the next operation can be directly carried out, the next operation can comprise that after the converted text data and the text mode data are summarized to obtain the text data, keyword extraction and encryption are carried out on the text data, or keyword extraction and encryption are directly carried out on the converted text data or the converted text mode data, the type and the quantity of the multi-mode data are specifically seen, and then the multi-mode data are set according to actual conditions.
In this embodiment, the multi-mode data of three mode data, i.e. voice, video and text, are processed according to the above steps, and three text data are obtained and summarized, so as to obtain the text data. The method has the advantages that the mode data are unified into the text mode, the mode number is not limited, the mode type is not limited, when new mode data are added, only a corresponding text description generation model is needed to be added, and the method has expansibility.
And step S230, extracting and encrypting the keywords of the text data to obtain the secret state keywords.
In one embodiment, step 230 may include:
and 231, extracting and encrypting the keywords of the first text data, the second text data and/or the third text data to obtain the secret state keywords.
Since the types of the modal data of the multi-modal data can be increased or decreased and can be multiple, the corresponding obtained text data can also be multiple, and when multiple text data exist, for example, when three text data in this embodiment exist, part or all of the text data can be summarized and then keyword extraction and encryption processing can be directly performed, so that the secret key words corresponding to the multi-modal data can be obtained.
In another embodiment, step 230 may include:
and 232, extracting keywords from the text data through an unsupervised learning algorithm to obtain keywords.
Keywords are typically single words or phrases composed of multiple words, referring to generalized words or phrases that reflect text topics or meanings. In this embodiment, text keyword extraction is performed using a Text Rank algorithm, which is an unsupervised algorithm, and a single document or Text data may be used as input. The algorithm principle is that text is split into words as network nodes to form a word network diagram, and the related relationship among the words is regarded as a recommendation or voting relationship, so that the importance of each word can be calculated, and then the first N words are obtained by screening and serve as keywords for representing the whole document or the whole text data.
Specifically, step 232 may include:
and 232.1, performing word segmentation processing on the text data to obtain a plurality of words.
In the specific implementation process, operations such as word segmentation, part-of-speech tagging, stop word removal and the like can be performed on the text data, wherein during word segmentation, the bargain word segmentation is adopted, 7 part-of-speech words such as common nouns, proper nouns, common verbs, auxiliary verbs, name verbs, adjectives, adverbs and the like are reserved, and finally a plurality of words can be obtained.
In this embodiment, the text data obtained in step S220 is subjected to word segmentation processing, so as to obtain a plurality of vocabularies.
And 232.2, drawing a vocabulary network diagram according to the plurality of vocabularies, wherein the network nodes of the vocabulary network diagram correspond to the vocabularies, and the edges connecting the two network nodes are provided with attribute values, and the attribute values are determined according to the co-occurrence relation of the plurality of vocabularies.
In the implementation process, one vocabulary is used as a network node, a vocabulary network diagram of a plurality of vocabularies is drawn, and in the diagram, attribute values are arranged between the network nodes, namely edges between the vocabularies, and are determined according to the co-occurrence relation of the vocabularies represented by the two network nodes.
In this embodiment, according to the plurality of vocabularies obtained in step S232.1, a vocabulary network diagram is drawn, where a network node set of the vocabulary network diagram is composed of the plurality of vocabularies, and edges of any two network nodes in the network node set are determined by analyzing co-occurrence relationships between vocabularies represented by the two network nodes, that is, only when vocabularies corresponding to the two network nodes co-occur in a window with a length of K, where K represents a window size, that is, K vocabularies at most co-occur.
And 232.3, sorting and screening the plurality of words according to the word network diagram to obtain keywords representing the text data.
And then sorting the weights of the words, screening to obtain a preset number of words, wherein the preset number of words are keywords for representing the text data. In practical implementation, the sorting mode and the preset number can be set according to practical situations.
And 233, carrying out encryption processing on the keywords through a symmetric encryption algorithm to obtain the encrypted keywords.
The keyword obtained in the step S232.3 is encrypted, and a symmetric encryption algorithm is specifically adopted, so that the algorithm has small calculated amount, high encryption speed and high encryption efficiency, can realize high-speed encryption and decryption processing, can use a long key, has difficult cracking, can ensure the privacy and the safety of multi-mode data, and can also improve the processing speed of the method.
And S240, acquiring a preset knowledge graph.
Specifically, step S240 may include:
s241, determining a basic knowledge graph through an open source knowledge graph;
The knowledge graph is a knowledge base, wherein knowledge is integrated through a data model or topology of a graph structure, the graph structure is used for visually describing the knowledge entities in the form of nodes, and the relationship among the knowledge entities is visually described in the form of edges, so that the association among the knowledge entities is explicitly described.
Fig. 4 is a schematic diagram of a preset knowledge graph in the present embodiment. In this figure, there are multiple entities, one representing each node. In this embodiment, based on the foregoing setting of the example language being english, the open source knowledge map Wikidata is used as the basic knowledge base, and the open source knowledge map is a large database, storing massive information in wikipedia and Freebase, and has the descriptive capability of common things, so that the requirements of this embodiment can be satisfied.
And step S242, carrying out encryption processing on the basic knowledge graph to obtain a preset knowledge graph, wherein an encryption algorithm adopted in the encryption processing is consistent with an encryption algorithm adopted in the encryption of the text data.
And the basic knowledge graph is encrypted, so that the privacy of the data is ensured. It should be noted that the encryption mode of the encryption process is identical to the encryption mode adopted for encrypting the keyword in step S233, and the used secret key is also identical, so that the subsequent secret key and the preset knowledge graph can be successfully matched.
The text keywords and the knowledge graph are encrypted by using the symmetric encryption method, so that the data privacy safety can be ensured, and the matching capability of the secret state keywords can be maintained.
And S250, dividing the preset knowledge graph according to the secret state keywords to obtain a secret state subgraph.
Specifically, step S250 may include:
Step S251, according to the secret state key words, matching entities corresponding to the secret state key words in the preset knowledge graph to obtain knowledge nodes;
And performing entity matching in a preset knowledge graph according to the secret state keywords, namely searching for an entity corresponding to the secret state keywords in the preset knowledge graph, and determining the entity as a knowledge node. When the knowledge nodes have a plurality of secret state keywords, a plurality of knowledge nodes can be obtained, and the knowledge nodes can be irrelevant or relevant in a preset knowledge graph.
In this embodiment, taking a secret key as an example, in the knowledge graph shown in fig. 4, the knowledge node corresponding to the secret key is represented by the entity 1.
In step S252, in the preset knowledge graph, the knowledge nodes are taken as the center, and the preset cutting distances are taken as the center, so that a secret state subgraph is obtained, wherein the length units of the preset cutting distances are edges between two entities, and the secret state subgraph is a set of the entities and the edges in the preset cutting distance range taking the knowledge nodes as the center.
Specifically, the knowledge node is taken as the center, sub-knowledge graph segmentation is carried out according to the preset cutting distance, and a secret state sub-graph corresponding to the secret state key word is obtained, namely, the set of entities and edges in the preset cutting distance range taking the knowledge node as the center. Wherein, each secret key word can obtain a corresponding secret sub-graph. As shown in fig. 4, the length unit of the preset clipping distance is an edge R between two entities, that is, an association relationship between two entities, and each edge is represented by R1, R2, and R. In this embodiment, in the knowledge graph of fig. 4, the entity 1 is taken as the center, and the segmentation is performed according to the preset clipping distance, so as to obtain the dense state subgraph shown in fig. 5. In the method, fig. 5 (a) shows a secret sub-graph obtained when the clipping distance is 1 unit, in the clipping process, an entity and an edge which can be reached by only one edge between the entity 1 and the entity 1 are divided by taking the entity 1 as the center, the obtained set of the entity and the edge is the secret sub-graph of the secret key word represented by the entity 1, fig. 5 (b) shows a secret sub-graph obtained when the clipping distance is 2 units, in the clipping process, an entity and an edge which can be reached by two or less edges between the entity 1 are divided by taking the entity 1 as the center, the obtained set of the entity and the edge is the secret sub-graph of the secret key word represented by the entity 1, and fig. 5 (c) shows the secret sub-graph obtained when the clipping distance is 3 units.
The sub-knowledge graph centering on the secret key words is used as the expression of the secret key words, and the secret key words are characterized by combining the entities and the relations which have extremely strong correlation with the secret key words, so that the semantic concepts of the secret key words can be expressed more comprehensively, the semantic concepts of the secret key words can be accurately characterized by other related entities and relations, the secret key words which are easy to be confused have more obvious discrimination, and the semantic correlation of the multi-mode data codes can be better reserved on the premise of ensuring the privacy of user data.
And step S260, performing graph embedding on the secret sub-graph to obtain a secret representation vector corresponding to the secret key word so as to obtain a semantic representation result of the multi-mode data.
Each secret state keyword can be expressed as a sub-knowledge graph, and based on the sub-knowledge graph, graph embedding operation is carried out on the secret state keyword to obtain a secret state representation vector of the secret state keyword, wherein the vector is semantic representation of the secret state keyword, and the graph embedding operation can be carried out by adopting a random Walk algorithm (Deep Walk algorithm).
In this embodiment, a Deep Walk algorithm is used to perform graph embedding, and the algorithm includes two parts, namely generation and updating, a random Walk generator (Random Walk Generator) is used to generate a random path similar to a sentence, and a random Walk Update program (Update Procedure) is used to input the random path into a Skip-Gram model to obtain a hidden representation of nodes in a secret state subgraph. The method comprises the steps of uniformly distributing and sampling a point on a dense-state subgraph by a random walk generator, taking the point as a starting point of a random path, arbitrarily determining a current point, taking the next point from all neighbors of the current point by uniformly distributing and sampling, determining the point as the current point, repeating until the set maximum length is reached, stopping the process when the current point is not adjacent, connecting all the points after stopping, namely the random path obtained by the random walk generator, wherein the length of each random path is possibly different but the maximum length is not exceeded, and generating a random path for each node in the dense-state subgraph according to the method. After the random path is generated, the random walk updating program regards the random walk updating program as a sentence, inputs the sentence into a Skip-Gram model, calculates an objective function, and updates the hidden representation of the node in the secret state subgraph, wherein the objective function is as follows:
j(Φ)=-logPr(uk|vj;Φ),
Where vj denotes the j-th point on the random path, uk denotes a node in the dense state subgraph that does not include vj, Pr(uk|vj; Φ) denotes the conditional probability that uk occurs given vj, Φ denotes the trainable parameter.
The method of the embodiment realizes the privacy semantic representation of the cross-modal data on the premise of guaranteeing the privacy of the user data and the correlation of the modal data.
For more details in the implementation of steps S210 to S260, reference may be made to the description in the implementation based on steps S110 to S150 in the first embodiment, and for brevity of description, details are not repeated here.
According to the cross-mode privacy semantic representation method, semantic association among the secret state keywords can be guaranteed, supported modes can be dynamically increased or decreased according to service requirements, flexibility and semantic accuracy of subsequent privacy semantic retrieval are guaranteed, and therefore accuracy of retrieval is guaranteed, and the method has important significance in meeting user privacy safety and cross-mode data retrieval requirements.
Example III
Based on the same inventive concept, referring to fig. 6, a first embodiment of the cross-modal privacy semantic representation apparatus of the present invention is presented, which may be a virtual apparatus, applied to a cross-modal privacy semantic representation device.
The cross-modal privacy semantic characterization apparatus provided in this embodiment is described in detail below with reference to a functional block diagram shown in fig. 6, where the apparatus may include:
the data acquisition module is used for acquiring multi-mode data;
The text description module is used for obtaining corresponding text data according to the multi-mode data;
The keyword extraction module is used for extracting and encrypting the keywords of the text data to obtain a secret state keyword;
the map segmentation module is used for segmenting the preset knowledge map according to the secret key words to obtain a secret sub-map;
And the diagram embedding module is used for carrying out diagram embedding on the secret state subgraph to obtain a secret state representation vector corresponding to the secret state keyword so as to obtain a semantic representation result of the multi-mode data.
Further, the multi-modal data includes data information of at least two different modalities, and the text description module may include:
the first data processing unit is used for converting the first modal data into first text data by utilizing a voice recognition technology when the multi-modal data comprises the first modal data of a voice modality;
The second data processing unit is used for converting the second modal data into second text data by utilizing a trained text generation model when the multi-modal data comprises the second modal data of the video modality;
and the third data processing unit is used for directly determining the third modal data as third text data when the multi-modal data comprises the third modal data of the text mode.
Still further, the keyword extraction module may include:
The first keyword extraction unit is connected with the first data processing unit, the first data processing unit and/or the third data processing unit and is used for extracting and encrypting the keywords of the first text data, the second text data and/or the third text data to obtain a secret state keyword.
Further, the keyword extraction module may include:
The keyword extraction sub-module is used for extracting keywords from the text data through an unsupervised learning algorithm to obtain keywords;
and the encryption sub-module is used for carrying out encryption processing on the keywords through a symmetrical encryption algorithm to obtain the encrypted keywords.
Still further, the keyword extraction sub-module may include:
The splitting unit is used for carrying out word segmentation processing on the text data to obtain a plurality of words;
The system comprises a plurality of vocabulary network graphs, a drawing unit and a control unit, wherein the vocabulary network graphs are drawn according to the plurality of vocabularies, network nodes of the vocabulary network graphs correspond to the vocabularies, and edges connecting the two network nodes are provided with attribute values, and the attribute values are determined according to the co-occurrence relation of the plurality of vocabularies;
And the screening unit is used for sequencing and screening the plurality of vocabularies according to the vocabulary network diagram to obtain keywords representing the text data.
Further, the apparatus may further include:
The system comprises a preset knowledge graph acquisition module, a basic knowledge graph acquisition module and a text data encryption module, wherein the preset knowledge graph acquisition module is used for determining a basic knowledge graph through an open source knowledge graph, and carrying out encryption processing on the basic knowledge graph to obtain a preset knowledge graph, wherein an encryption algorithm adopted by the encryption processing is consistent with an encryption algorithm adopted by the text data encryption.
Further, the atlas segmentation module may include:
the matching unit is used for matching the entity corresponding to the secret state keyword in the preset knowledge graph according to the secret state keyword to obtain a knowledge node;
the segmentation unit is used for segmenting the knowledge nodes serving as the centers in the preset knowledge graph according to preset cutting distances to obtain a secret state subgraph, wherein the length unit of the preset cutting distances is an edge between two entities, and the secret state subgraph is a set of the entities and the edges in the preset cutting distance range serving as the centers of the knowledge nodes.
It should be noted that, the functions that can be achieved by each module in the cross-modal privacy semantic representation device and the corresponding achieved technical effects provided in this embodiment may refer to descriptions of specific embodiments in each embodiment of the cross-modal privacy semantic representation method of the present invention, and for brevity of description, no further description is given here.
Example IV
Based on the same inventive concept, referring to fig. 2, a schematic hardware structure of a cross-mode privacy semantic characterization device according to embodiments of the present invention is shown. The embodiment provides cross-mode privacy semantic representation equipment, which can comprise a processor and a memory, wherein the memory stores a cross-mode privacy semantic representation program, and when the cross-mode privacy semantic representation program is executed by the processor, all or part of steps of each embodiment of the cross-mode privacy semantic representation method are realized.
Specifically, the cross-mode privacy semantic representation device refers to terminal devices or network devices capable of realizing network connection, and can be terminal devices such as mobile phones, computers, tablet computers, portable computers and the like, or network devices such as servers and cloud platforms.
It will be appreciated that the device may also include a communication bus, a user interface, and a network interface.
Wherein the communication bus is used to enable connection communication between these components.
The user interface is used for connecting the client and communicating data with the client, and may comprise an output unit, such as a display screen, an input unit, such as a keyboard, and optionally, other input/output interfaces, such as a standard wired interface, a wireless interface.
The network interface is used to connect to and communicate data with the background server, and may include an input/output interface such as a standard wired interface, a wireless interface such as a Wi-Fi interface.
The memory is used to store various types of data, which may include, for example, instructions for any application or method in the cross-modality privacy semantic characterization device, as well as application-related data. The Memory may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as static random access Memory (Static Random Access Memory, SRAM), random access Memory (Random Access Memory, RAM), electrically erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EPROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic or optical disk, optionally, the Memory may also be a storage device independent of the processor.
The Processor is configured to invoke the cross-modal privacy semantic representation program stored in the memory and execute the cross-modal privacy semantic representation method as described above, where the Processor may be an Application-specific integrated Circuit (ASIC), a Digital Signal Processor (DSP), a digital signal processing device (DIGITAL SIGNAL Processing Device DSPD), a programmable logic device (Programmable Logic Device PLD), a field programmable gate array (Field Programmable GATE ARRAY FPGA), a controller, a microcontroller, a microprocessor, or other electronic components configured to execute all or part of the steps of the various embodiments of the cross-modal privacy semantic representation method as described above.
Example five
Based on the same inventive concept, this embodiment provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., having stored thereon a computer program executable by one or more processors, which when executed by the processors may implement all or part of the steps of the various embodiments of the cross-modal privacy semantic characterization method of the present invention.
It should be noted that, the foregoing reference numerals of the embodiments of the present invention are merely for describing the embodiments, and do not represent the advantages and disadvantages of the embodiments.
The foregoing description is only of the optional embodiments of the present invention, and is not intended to limit the scope of the invention, and all equivalent structures or equivalent processes using the descriptions of the present invention and the accompanying drawings, or direct or indirect application in other related technical fields are included in the scope of the invention.