Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides the following technical scheme.
The invention provides a multi-data source NL2SQL system based on semantic rules and a multi-dimensional model, which comprises a service layer for realizing NL2SQL, wherein the service layer comprises:
the configuration module is used for configuring the matching rules;
the data model building module is used for building a data model which can generate SQL sentences by using the table and field information of the database;
and the intention identification module is used for analyzing and matching the input natural sentences according to the matching rules to obtain table and field information of the database, and generating executable SQL sentences by using the data model.
Preferably, the configuration module comprises:
the context configuration module is used for configuring context matching rules;
the user dictionary configuration module is used for configuring a user dictionary;
the table field configuration module is used for configuring tables, fields, table relations and field relations of the database;
and the semantic segment configuration module is used for configuring the semantic segment matching rule.
Preferably, the user dictionary configuration module comprises a user self-defining module and a professional dictionary module, wherein the user self-defining module is used for configuring user-defined synonyms, stop words, keywords and/or entities, and the professional dictionary module is used for configuring a professional dictionary.
Preferably, the user dictionary configuration module further includes a third-party participle and entity recognition model calling module, configured to configure a calling interface of the third-party participle and entity recognition model.
Preferably, the semantic segment matching rules include matching words, semantic rules and element fields.
Preferably, the intention identifying module includes:
the context matching module is used for performing context matching on the natural sentences according to configured context matching rules to obtain context information of the natural sentences;
the tag identification module is used for performing word segmentation and entity identification on the natural sentence according to a configured user dictionary and by combining the context information to obtain an entity type corresponding to the word segmentation;
the semantic fragment matching module is used for performing word segmentation matching according to the configured semantic fragment matching rule to obtain a semantic fragment;
the database field association module is used for obtaining corresponding table and field information and table relations by using the tables and the fields of the database matched and configured by the semantic fragments;
and the SQL statement generating module is used for generating an executable SQL statement by using the data model according to the table and field information obtained by matching and the table relation.
Preferably, the performing word segmentation matching according to the configured semantic segment matching rule to obtain a semantic segment includes:
matching each obtained participle with a word in the semantic segment matching rule to obtain a semantic rule;
and matching according to the semantic rule to obtain the element information of the semantic fragments.
Preferably, the matching each obtained participle with a term in the semantic segment matching rule includes:
converting the word segmentation into a word segmentation vector, and calculating the similarity between the word segmentation vector and the vector of the word in the semantic segment matching rule; and if the similarity reaches a threshold value, replacing the participle with a word in the semantic segment matching rule.
Preferably, the data model building module comprises:
the table selection module is used for loading the data table and selecting the fact table and the dimension table;
the dimension selection module is used for selecting a dimension;
the index selection module is used for selecting indexes;
and the model establishing module is used for selecting a statistical method and establishing a data model according to the selected dimension and the index.
Preferably, the system further comprises a data access layer, a data storage layer and an application layer;
the data access layer is used for receiving data input by a user and configured matching rules;
the data storage layer is used for data storage;
the application layer is used for displaying and inquiring the result of the conversion of the natural statement into the SQL statement.
The invention has the beneficial effects that: the multi-data-source NL2SQL system based on the semantic rules and the multi-dimensional model provided by the invention does not depend on the labeled linguistic data and the model, and can realize the NL2SQL based on the intention through simple configuration of the interface; the data model building module supports multi-choice nesting and can be flexibly configured; the requirement on the data set is reduced through the intention identification module, and the success rate of SQL generation is improved. Therefore, the system provided by the invention can quickly construct a service scene according to different projects and support the expansion of multiple data sources such as NoSQL, relational databases and the like.
Detailed Description
In order to better understand the technical solution, the technical solution will be described in detail with reference to the drawings and the specific embodiments.
The system provided by the present invention may be implemented in the following terminal environment, which may include one or more of the following components: a processor, a memory, and a display screen. Wherein the memory has stored therein at least one instruction that is loaded and executed by the processor to implement the system described in the embodiments below.
A processor may include one or more processing cores. The processor connects various parts within the overall terminal using various interfaces and lines, performs various functions of the terminal and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory, and calling data stored in the memory.
The Memory may include a Random Access Memory (RAM) or a Read-Only Memory (ROM). The memory may be used to store instructions, programs, code sets, or instructions.
The display screen is used for displaying user interfaces of all the application programs.
In addition, those skilled in the art will appreciate that the above-described terminal configurations are not intended to be limiting, and that the terminal may include more or fewer components, or some components may be combined, or a different arrangement of components. For example, the terminal further includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a power supply, and other components, which are not described herein again.
Example one
As shown in fig. 1, an embodiment of the present invention provides a multiple data source NL2SQL system based on semantic rules and a multidimensional model, including a business layer for implementing NL2SQL, where the business layer includes:
the configuration module is used for configuring the matching rules;
the data model building module is used for building a data model which can generate SQL sentences by using the table and field information of the database;
and the intention identification module is used for analyzing and matching the input natural sentences according to the matching rules to obtain table and field information of the database, and generating executable SQL sentences by using the data model.
In the actual use process, the configuration module can be displayed in the form of an interface, after a user logs in the system and opens the configuration module, the configuration interface appears, and the user can configure corresponding contents, such as different projects or service fields, according to requirements in the interface, wherein the configured contents are different. Specifically, the content to be configured may be set in the template, and the user may configure the corresponding content in the template by selecting, dragging, or inputting.
Specifically, in the embodiment of the present invention, the configuration module includes:
the context configuration module is used for configuring context matching rules;
the user dictionary configuration module is used for configuring a user dictionary;
the table field configuration module is used for configuring tables, fields, table relations and field relations of the database;
and the semantic segment configuration module is used for configuring the semantic segment matching rule.
The context matching rules configured in the context configuration module can be used as a basis for determining the context of the input sentences in the using process. And in the process of processing the input natural sentences, the context is determined firstly, so that the foundation can be laid for the further sentence processing process, the subsequent processing of the sentences can be performed in the environment of the matched context, and the processing efficiency and the recall accuracy are improved. As an example, such as: the user needs to query the travel record of zhang san, can analyze elements of the scene of 'travel', then extract corresponding element configuration rules, for example, the record includes a series of characteristic words representing travel such as element people, travel, track, and the like, and complete context matching according to the rules.
The user dictionary configuration module may include a user-defined module for configuring user-defined synonyms, stop words, keywords, and/or entities and a professional dictionary module for configuring a professional dictionary.
For a specific business professional field, more professional languages or newly generated words exist, a conventional dictionary often cannot be used for accurately segmenting natural sentences in the professional field, and in order to solve the problem, a professional dictionary module and a user-defined module are arranged in the embodiment of the invention, and a user can configure a professional dictionary by using the professional dictionary module. In the professional dictionary, explanations of the professional language and/or newly generated words and the like, and descriptions of entity types and the like are included. In addition, the user can enrich the professional language and/or newly generated words and the like by customizing synonyms, stop words, keywords, entities and the like so as to assist the processing capacity of the professional dictionary on the words. If a reference word corresponds to multiple synonyms or multiple stop words, etc., the words are separated by space numbers. Because the matching degree of the professional dictionary and the professional sentence is better, the professional dictionary is adopted for word segmentation and entity recognition, the precision can be improved, and the method is more suitable for business scenes. Meanwhile, the deep cognition of the user to the professional field is utilized, the content of the professional dictionary is supplemented through self-defined synonyms, stop words, keywords and/or entities and the like, and the precision of word segmentation and entity recognition can be further improved.
In a preferred embodiment of the present invention, the user dictionary configuration module may further include a third-party participle and entity recognition model invoking module, configured to configure an invoking interface of the third-party participle and entity recognition model. In the actual use process, a third-party word segmentation and entity recognition model can be directly called through an interface, and word segmentation and entity recognition processing of natural sentences is completed through the model.
In the table field configuration module, configuring the fields to be converted into SQL in the database, wherein the specific information to be configured includes: database type, table name, table field name, type, etc. As an example, the format of its configuration may be as shown in fig. 2.
In the semantic segment configuration module, the content capable of extracting field information from the semantic segment union is configured in an interface configuration manner, for example, the following information is included: matching words, semantic rules, semantic fragments, rule types, element fields, and the like.
In one embodiment of the invention, the data model building module comprises:
the data source selection module is used for selecting a data source;
the table selection module is used for loading the data table and selecting the fact table and the dimension table;
the dimension selection module is used for selecting a dimension;
the index selection module is used for selecting indexes;
and the model establishing module is used for selecting a statistical method and establishing a data model according to the selected dimension and the index.
The constructed data model mainly comprises a database table and a table relation or a table and field relation, and can be a single table or a plurality of tables. The structure of the generated data model may be as shown in fig. 3.
The data model building module has the main task of building a model capable of generating the SQL relationship, and a user can complete the building of the model in a dragging and configuring mode. The process of construction is shown in fig. 4, and comprises: selecting a data source; loading a data table, and selecting a fact table and a dimension table; selecting dimensions and indexes; and selecting a statistical method, and establishing a data model based on the selected dimension and the index by using the statistical method.
The data model constructed by the data model construction module provided by the invention has the following functional characteristics:
the method can support various data source selections, including relational databases such as MySQL, Oracle and PostgreSQL and the like, and also can select non-relational databases such as ES, HBase and hive; selecting a configuration derived from a table field by a data source;
a user selects a dimension field and an index field on a visual interface in a dragging and checking mode, and a data model is flexibly constructed;
the configured index may support various modes such as SUM, COUNT, MAX, MIN, AVG, COUNT _ DISTINCT, and the like.
In a preferred embodiment of the present invention, the intention identifying module includes:
and the context matching module is used for performing context matching on the natural sentences according to configured context matching rules to obtain the context information of the natural sentences. For example, for an input natural sentence "a track of a person going to place a and place B in the last 1 year", the travel context is obtained according to the context matching rule.
And the label identification module is used for performing word segmentation and entity identification on the natural sentence according to the configured user dictionary and by combining the context information to obtain an entity type corresponding to the word segmentation.
And the semantic fragment matching module is used for performing word segmentation matching according to the configured semantic fragment matching rule to obtain the semantic fragments.
And the database field association module is used for obtaining corresponding table and field information and table relations by using the table and the field of the database matched and configured by the semantic fragments. The output of the database field association module is optionally in JSON format. For example:
and the SQL statement generating module is used for generating an executable SQL statement by using the data model according to the table and field information obtained by matching and the table relation. Optionally, the SQL statement generation module selects the data model from the plurality of constructed data models according to the context information. For example, the output SQL is as follows:
Select
*
From
dm_person_locusdm_person_locus,
dm_entity_persondm_entity_person
where
dm_entity_person.id=dm_person_locus.persion_id
and dm _ entity _ precision.xm like '% of person'
and dm_person_locus.cfsj>‘2019-08-23 17:15:07’
and dm_person_locus.cfsj<‘2020-08-23 17:15:07’
and(
dm _ person _ los.ddd like '% A to ground'
or dm _ person _ los.ddd like '% of ground'
)
and(1=1)
In the actual application process, the intention identification module combines various matching rules configured in the configuration module and the data model constructed in the data model construction module to complete the generation of the SQL statement. Therefore, in the business layer, the configuration of the rules and the construction process of the data model are mainly completed, and the process of converting the input natural statements into the SQL statements by using the rules and the model is mainly completed.
The specific workflow of the system can be as shown in fig. 5.
In FIG. 5, A-G represent the flow of a user configuring NL2SQL functions:
a, selecting a data configuration table field from a data source;
b, constructing a data model according to the configuration information of the table fields;
c, obtaining a constructed data model;
d, configuring a user dictionary;
e, configuring information in the intention identification module;
f, storing the configuration information and the data model into a data source;
g uses configuration information and data models for intent recognition.
The first to the ninth steps represent the process of converting the natural sentence query input by the user into the SQL sentence query:
firstly, a user inputs a query statement, and a system matches a natural statement to context configuration according to a context matching rule;
performing word segmentation and entity recognition by using a label model combining a user dictionary and entity recognition;
carrying out semantic rule matching on the words after word segmentation to obtain semantic fragments;
fourthly, according to the semantic fragment correlation database field, completing semantic fragment mapping to obtain table and field information and table relation;
splicing the data model into executable SQL according to the table association rule of the data model;
seventhly, SQL is converted into different database query languages through a data adapter;
and eighthly, inquiring the database data according to the SQL rule and returning an inquiry result.
The structure of the intention identifying module can be as shown in fig. 6.
In an embodiment of the present invention, the word segmentation matching is performed according to the configured semantic segment matching rule to obtain a semantic segment;
matching each obtained participle with a word in the semantic segment matching rule to obtain a semantic rule;
and matching according to the semantic rule to obtain the element information of the semantic fragments.
Wherein the matching of each obtained participle with a term in the semantic segment matching rule comprises:
converting the word segmentation into a word segmentation vector, and calculating the similarity between the word segmentation vector and the vector of the word in the semantic segment matching rule; and if the similarity reaches a threshold value, replacing the participle with a word in the semantic segment matching rule.
By adopting the semantic matching mode, the recall rate and the generalization capability of semantic recognition can be greatly improved.
As shown in fig. 7, the system provided by the present invention further includes a data access layer, a data storage layer, and an application layer;
the data access layer is used for receiving data input by a user and configured matching rules, such as context matching rules, a user dictionary, table fields of a database, semantic fragment matching rules and the like;
the data storage layer is used for data storage, such as storing a user dictionary, a context configuration, a data model, a semantic segment, a field relation mapping, a user NL2SQL log and the like, and providing rules and data configured by a user;
the application layer is used for displaying and inquiring the result of the conversion of the natural statement into the SQL statement.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention. It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.