CN110851357A - Test data automatic construction method based on multiple database types - Google Patents
Test data automatic construction method based on multiple database typesDownload PDFInfo
- Publication number
- CN110851357A CN110851357ACN201911066556.XACN201911066556ACN110851357ACN 110851357 ACN110851357 ACN 110851357ACN 201911066556 ACN201911066556 ACN 201911066556ACN 110851357 ACN110851357 ACN 110851357A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- configuration
- module
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The invention belongs to the field of internet, and particularly relates to a test data automatic construction method based on multiple database types.
Background
Today's internet is in the big data age, and data is the main heart bone of products. To ensure normal execution of data and quality of big data products, the data source is required to be started in the test process: and constructing diversified test data according to the service characteristics, and constructing mass data for testing to check whether the system service is normal or not. Therefore, in the big data test work, constructing test data is necessary prerequisite work.
When test data are constructed, different types of database tables are created one by one, data are created one by one and inserted, and therefore efficiency is low and steps are complex. On the other hand, the test data needs to satisfy the input constraint rules and cover as much as possible different types of data that satisfy the rules. In view of the above, the automatic construction of data can greatly improve the test efficiency.
Along with the increase of data types, the data size is increased, the application requirements are diversified, the types of databases are continuously increased, and the current data construction tool has some defects: random switching of multiple database types cannot be provided, the data types are not rich enough, the data generation rule is not flexible enough, and the execution efficiency of tens of millions of data quantity is not controllable.
Disclosure of Invention
In view of the above, the present invention is directed to a method for automatically constructing test data based on multiple database types, so as to improve the execution efficiency of data construction.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
a test data automatic construction method based on multiple database types comprises the following steps:
s1, inputting configuration information;
s2, a configuration analysis module reads configuration information;
s3, the scheduling module reads the type of the database and calls a corresponding code;
s4, connecting a database to judge whether the database table exists or not;
A1. if so, step S5 is skipped;
A2. if not, executing the steps in sequence;
s5, calling a database operation module to create a database table;
s6, reading the data volume and the thread number by the execution module, and uniformly distributing the data volume of the single thread;
s7, a data generation module reads the configured data rule to generate data and combines the generated data of all the table fields into a piece of data to be put in storage;
s8, performing insertion operation of the database operation module, and inserting the combined data into a target base table;
s9, judging whether the thread number and the data volume cycle are finished or not;
B1. ending, and executing the steps in sequence;
B2. if not, jumping to step S7;
and S10, outputting a result.
Further, the step S1 of inputting the configuration information includes two methods:
C1. directly inputting through a visual interface;
C2. and uploading the yaml configuration file containing the configuration information, wherein the configuration information comprises database type configuration, database connection information configuration, data volume configuration, database table structure configuration, table field type configuration, data generation rule configuration and execution thread number configuration.
Further, the step S2 is to configure the parsing module, and the method for reading the configuration information includes:
the configuration analysis module analyzes the configuration file, performs structural processing and quantization on the related information of the database, and can be directly called later; and generating SQL statements according to the type and the table structure information of the database as required for later use when executing database operation.
Further, the method for the scheduling module to read the database type to call the corresponding code in step S3 includes:
and the scheduling module reads the database type, calls the corresponding database execution code and executes the database connection operation.
Further, the method for executing module reading data volume and thread number in step S6 to evenly distribute data volume of single thread includes:
the execution module can read the data volume and the thread number, and calculate the data storage volume required to be executed by a single thread according to the data volume and the thread number so as to achieve balanced concurrent execution.
Further, the method of step S7 includes:
the scheduling module calls a data generation module corresponding to the data type according to the data type of each table field in the configuration file, the data generation module generates data according to the data generation rule, and the data of each field is combined after being generated to form a piece of test data to be inserted
Further, the step S10 outputs the result including: and the system outputs the total successful data volume of the warehousing and the execution duration.
Compared with the prior art, the invention has the following advantages:
(1) the invention simplifies the usability of the tool: configuration information is visually input, information such as the type of the database and the type of the data field is reasonably checked, after execution is finished, the total number of the data fields and the storage time length of the data fields are returned to a user by the system for checking, and the interactivity of the tool is improved.
(2) The functions of the data construction tool are further enriched: the data construction functions of various database types are integrated; besides the common data types, types supported by non-relational databases such as arrays, Json and the like are added; and enriching flexible data generation rules.
(3) In order to effectively control the execution efficiency of the data construction, a user can set the thread number according to the data volume, and the system can evenly distribute the data volume of a single thread according to the thread number and the data volume so as to reasonably control the execution efficiency of the data construction.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an embodiment of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow chart of a system according to an embodiment of the present invention;
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention. Furthermore, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
According to FIG. 1, the method comprises the following steps
1) Configuration information is input, and the method is realized through two ways: 1. and directly inputting through a visual interface. 2. By uploading a. yaml profile containing the configuration information. The configuration information comprises database type configuration, database connection information configuration, data volume configuration, database table structure configuration, table field type configuration, data generation rule configuration and execution thread number configuration.
2) The configuration analysis module analyzes the configuration file, performs structural processing and quantization on the related information of the database, and can be directly called later; generating SQL sentences according to the type and the table structure information of the database as required for later use when executing database operation;
3) the scheduling module reads the database type, calls a corresponding database execution code and executes the operation of connecting the database;
4) after the database is successfully connected, the system can automatically judge whether the target database table exists. If not, automatically executing the creation base table SQL of the database operation module; if the base table already exists, directly executing the next step;
5) the execution module reads the data volume and the thread number, and calculates the data storage volume required to be executed by a single thread according to the data volume and the thread number so as to achieve balanced concurrent execution;
6) the scheduling module calls a data generation module corresponding to the data type according to the data type of each table field in the configuration file, the data generation module generates data according to the data generation rule, and the data of each field is combined after the data generation is finished to form a piece of test data to be inserted;
7) and the execution module calls a data insertion code through the calling module and inserts the data combined in the step 6 into the target base table.
8) Embedding a data volume loop in each thread, and executing the step 6 and the step 7 in the loop until the thread number and the data volume loop are finished;
9) and after all threads are executed, outputting the total successful data volume and the execution duration of the warehousing by the system.
The module comprises:
a configuration analysis module: the standard of the configuration file is yaml file, and the configuration file is slightly different due to different types of databases and different internal storage structures. The configuration file analysis module comprises a plurality of classes (a configuration file processing class and an SQL statement generation class which take the database types as dimensions), respectively processes configuration files of different database types, and has the following main functions:
1. structuring the configuration information: and extracting database connection information, database table names and database field information (type/length/rule) and storing the information in the dictionary in a key/value mode.
The configuration file supports the following rules:
1. data of numerical value type (integer, decimal) generates corresponding data according to the initial value (start, end) and step length (step); and if no step length exists, generating a random value in the data range.
2. Time type data, generating corresponding data according to the initial values (start, end) and the step length (s/min/hour/day/mouth/year); if no step length exists, a random value in the range of the initial value is generated.
3. And generating a character string with the character string length within the range of 0-length according to the input length by using the data of the character string type.
4. The array type data generates corresponding array data according to the array type (array _ type), the data length (num) and the corresponding rule (refer to 1/2/3).
5. Boolean type data, randomly generating true/false.
Json data, based on the Json level number (num), generating random type data (write in template)
2. Generating a Sql statement: and generating createsql statements (judging whether the database tables exist, do not exist and do not exist) and insertsql statements according to the database table information in the configuration file.
A data generation module: and generating data for the dimension according to the data type, acquiring the type of each field in the configuration file, and generating the data according to the corresponding rule.
Among the rules supported by the configuration file are (basically covering the common data types):
1. data of numerical value type (integer, decimal) generates corresponding data according to the initial value (start, end) and step length (step); and if no step length exists, generating a random value in the data range.
2. Time type data, generating corresponding data according to the initial values (start, end) and the step length (s/min/hour/day/mouth/year); if no step length exists, a random value in the range of the initial value is generated.
3. And generating a character string with the character string length within the range of 0-length according to the input length by using the data of the character string type.
4. The array type data generates corresponding array data according to the array type (array _ type), the data length (num) and the corresponding rule (refer to 1/2/3).
5. Boolean type data, randomly generating true/false.
Json data, based on the Json level number (num), generating random type data (write in template)
A database execution module: the module encapsulates different database types, currently comprises data operation types of different database types such as myslq, mongo, hbase and the like, and mainly comprises connection database operation, creation base table (creation if the creation is not performed or not performed), data insertion operation, data information query operation and data volume query operation.
An execution module:
1. and distributing the data warehousing quantity required to be executed by a single thread according to the total data quantity and the thread number so as to achieve balanced concurrent execution. Example (c): 100w of data, 10 threads of code can automatically create 10 threads for concurrence, wherein the first thread is 1-10w, the second thread is 10w + 1-20 w, and the like, and 100w of non-repeated data import is realized.
2. And in the data import process, the corresponding database execution module is scheduled according to the type of the database input by the user. Respectively executing the following steps: connecting the databases, creating a base table, concurrently inserting 10w data (inserting data operation) into 10 threads, and finishing the current data size of the real-time printing database after each thread executes.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (7)
1. A test data automatic construction method based on multiple database types is characterized in that: the method comprises the following steps:
s1, inputting configuration information;
s2, a configuration analysis module reads configuration information;
s3, the scheduling module reads the type of the database and calls a corresponding code;
s4, connecting a database to judge whether the database table exists or not;
A1. if so, step S5 is skipped;
A2. if not, executing the steps in sequence;
s5, calling a database operation module to create a database table;
s6, reading the data volume and the thread number by the execution module, and uniformly distributing the data volume of the single thread;
s7, a data generation module reads the configured data rule to generate data and combines the generated data of all the table fields into a piece of data to be put in storage;
s8, performing insertion operation of the database operation module, and inserting the combined data into a target base table;
s9, judging whether the thread number and the data volume cycle are finished or not;
B1. ending, and executing the steps in sequence;
B2. if not, jumping to step S7;
and S10, outputting a result.
2. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the step S1 of inputting configuration information includes two methods:
C1. directly inputting through a visual interface;
C2. and uploading the yaml configuration file containing the configuration information, wherein the configuration information comprises database type configuration, database connection information configuration, data volume configuration, database table structure configuration, table field type configuration, data generation rule configuration and execution thread number configuration.
3. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the step S2 is to configure the parsing module, and the method for reading the configuration information includes:
the configuration analysis module analyzes the configuration file, performs structural processing and quantization on the related information of the database, and can be directly called later; and generating SQL statements according to the type and the table structure information of the database as required for later use when executing database operation.
4. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the method for calling the corresponding code by the scheduling module reading the database type in the step S3 includes:
and the scheduling module reads the database type, calls the corresponding database execution code and executes the database connection operation.
5. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the step S6 is that the method for the execution module to read the data size and the thread number and evenly distribute the data size of the single thread includes:
the execution module can read the data volume and the thread number, and calculate the data storage volume required to be executed by a single thread according to the data volume and the thread number so as to achieve balanced concurrent execution.
6. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the method of step S7 includes:
and the scheduling module calls a data generation module corresponding to the data type according to the data type of each table field in the configuration file, the data generation module generates data according to the data generation rule, and the data of each field is combined after being generated to form a test array to be inserted.
7. The method of claim 1 for automatic construction of test data based on multiple database types, wherein: the step S10 outputs the result including: and the system outputs the total successful data volume of the warehousing and the execution duration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911066556.XACN110851357A (en) | 2019-11-04 | 2019-11-04 | Test data automatic construction method based on multiple database types |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911066556.XACN110851357A (en) | 2019-11-04 | 2019-11-04 | Test data automatic construction method based on multiple database types |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110851357Atrue CN110851357A (en) | 2020-02-28 |
Family
ID=69599100
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911066556.XAPendingCN110851357A (en) | 2019-11-04 | 2019-11-04 | Test data automatic construction method based on multiple database types |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110851357A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131105A (en)* | 2020-09-16 | 2020-12-25 | 电信科学技术第十研究所有限公司 | Test data construction method and device |
| CN112783785A (en)* | 2021-02-01 | 2021-05-11 | 紫光云技术有限公司 | General test method for Nosql database performance |
| CN114153911A (en)* | 2021-12-21 | 2022-03-08 | 浪潮软件集团有限公司 | Method and system for self-defined database test data generation based on VBA technology |
| CN114238343A (en)* | 2021-12-23 | 2022-03-25 | 南京华飞数据技术有限公司 | Implementation method of multidimensional variability automatic data model based on big data |
| CN114371988A (en)* | 2021-11-19 | 2022-04-19 | 山东齐鲁数通科技有限公司 | Method, device, terminal and medium for assisting batch generation of test data |
| CN114416544A (en)* | 2022-01-05 | 2022-04-29 | 辽宁振兴银行股份有限公司 | Device and method for multi-library multi-manufacturing test data |
| CN120448284A (en)* | 2025-07-10 | 2025-08-08 | 卓望数码技术（深圳）有限公司 | Method and device for generating massive database table test data based on configuration |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6138112A (en)* | 1998-05-14 | 2000-10-24 | Microsoft Corporation | Test generator for database management systems |
| CN105512042A (en)* | 2015-12-22 | 2016-04-20 | 广东金赋信息科技有限公司 | Method, device and test system for automatically generating test data of a database |
| CN109726096A (en)* | 2017-10-27 | 2019-05-07 | 中移(苏州)软件技术有限公司 | Method and device for generating test data |
| CN109815252A (en)* | 2018-12-14 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Test data generating method, device, computer equipment and readable storage medium storing program for executing |
| US20190188118A1 (en)* | 2017-12-15 | 2019-06-20 | Unisys Corporation | System and method for generating database independent input test data |
| CN110222128A (en)* | 2019-06-12 | 2019-09-10 | 浪潮软件集团有限公司 | A kind of method and device generating data initialization sql |
| CN110221967A (en)* | 2019-05-20 | 2019-09-10 | 深圳壹账通智能科技有限公司 | Test data building method, device, computer equipment and storage medium |
- 2019
- 2019-11-04CNCN201911066556.XApatent/CN110851357A/enactivePending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6138112A (en)* | 1998-05-14 | 2000-10-24 | Microsoft Corporation | Test generator for database management systems |
| CN105512042A (en)* | 2015-12-22 | 2016-04-20 | 广东金赋信息科技有限公司 | Method, device and test system for automatically generating test data of a database |
| CN109726096A (en)* | 2017-10-27 | 2019-05-07 | 中移(苏州)软件技术有限公司 | Method and device for generating test data |
| US20190188118A1 (en)* | 2017-12-15 | 2019-06-20 | Unisys Corporation | System and method for generating database independent input test data |
| CN109815252A (en)* | 2018-12-14 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Test data generating method, device, computer equipment and readable storage medium storing program for executing |
| CN110221967A (en)* | 2019-05-20 | 2019-09-10 | 深圳壹账通智能科技有限公司 | Test data building method, device, computer equipment and storage medium |
| CN110222128A (en)* | 2019-06-12 | 2019-09-10 | 浪潮软件集团有限公司 | A kind of method and device generating data initialization sql |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131105A (en)* | 2020-09-16 | 2020-12-25 | 电信科学技术第十研究所有限公司 | Test data construction method and device |
| CN112131105B (en)* | 2020-09-16 | 2023-05-30 | 电信科学技术第十研究所有限公司 | Method and device for constructing test data |
| CN112783785A (en)* | 2021-02-01 | 2021-05-11 | 紫光云技术有限公司 | General test method for Nosql database performance |
| CN114371988A (en)* | 2021-11-19 | 2022-04-19 | 山东齐鲁数通科技有限公司 | Method, device, terminal and medium for assisting batch generation of test data |
| CN114153911A (en)* | 2021-12-21 | 2022-03-08 | 浪潮软件集团有限公司 | Method and system for self-defined database test data generation based on VBA technology |
| CN114238343A (en)* | 2021-12-23 | 2022-03-25 | 南京华飞数据技术有限公司 | Implementation method of multidimensional variability automatic data model based on big data |
| CN114238343B (en)* | 2021-12-23 | 2022-10-28 | 南京华飞数据技术有限公司 | Implementation method of multidimensional variability automatic data model based on big data |
| CN114416544A (en)* | 2022-01-05 | 2022-04-29 | 辽宁振兴银行股份有限公司 | Device and method for multi-library multi-manufacturing test data |
| CN120448284A (en)* | 2025-07-10 | 2025-08-08 | 卓望数码技术（深圳）有限公司 | Method and device for generating massive database table test data based on configuration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110851357A (en) | Test data automatic construction method based on multiple database types | |
| CN104572122A (en) | Software application data generating device and method | |
| CN109446072A (en) | The generation method and device of test script | |
| CN108920140B (en) | Front-end and back-end unified verification method | |
| CN111596902A (en) | Method, device, equipment and storage medium for building front-end and back-end development framework | |
| CN108241642B (en) | File analysis method and device | |
| CN1592908B (en) | Database system having heterogeneous object types | |
| CN111026670B (en) | Test case generation method, test case generation device and storage medium | |
| CN102708203A (en) | A Dynamic Database Management Method Based on XML Metadata | |
| CN110198327B (en) | Data transmission method and related equipment | |
| CN112084167B (en) | A permission filtering method, device and storage medium | |
| CN112988782A (en) | Hive-supported interactive query method and device and storage medium | |
| CN114238527A (en) | Data processing method and data processing device based on object relation mapping | |
| CN112802607B (en) | A medical data management system | |
| CN105959469B (en) | Short message display method, display device and terminal | |
| CN110969000B (en) | Data merging processing method and device | |
| CN117827901A (en) | Nashort-based method and device for dynamically executing database script | |
| CN116341514A (en) | File analysis method and device based on dynamic configuration | |
| CN109918391B (en) | Streaming transaction processing method and system | |
| CN112540978A (en) | Wide table generation method and device and electronic equipment | |
| CN112269785B (en) | Method and system for dynamically filling field in subject state details in EDC (electronic data communication) system | |
| CN113868112A (en) | Device testing method and apparatus, storage medium, and electronic apparatus | |
| CN113934726B (en) | A random smoothing table partitioning method, terminal and storage medium | |
| CN104166737B (en) | Amending method and device for database-name | |
| CN114168614A (en) | Data management method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20200228 |