wherein k is a load balance index of the server before ETL processing is performed on the service data to be analyzed, A is a CPU utilization rate of the server, B is a memory utilization rate of the server,

expressed as a current service rate of the server,

expressed as a target number of working nodes,

representing the current usage data amount corresponding to the current work task of the ith target work node,

expressed as the maximum allocated data volume for the ith target worker node,

expressed as the expected usage data volume of the ith target worker node when the job stability is highest,

performance index expressed as ith target work node；

Calculating a target time length for ETL processing on the service data to be analyzed by using the current load balancing index according to the load balancing index before ETL processing is carried out on the service data to be analyzed by the server:

wherein,

expressed as a target duration for performing ETL processing on the service data to be analyzed by using the current load balancing index,

expressed as a preset desired length of time,

the target data quantity allocated when the ith target working node performs ETL processing on the service data to be analyzed is represented,

expressed as the current remaining data volume of the ith target worker node,

the working efficiency of the ith target working node of the server under the load balancing index is represented;

whether the target time length is greater than or equal to a preset limited time length is confirmed, if yes, work node task process optimization is carried out on the server to liberate the use data volume of the target work node, and otherwise, follow-up operation is not needed;

and after the server is optimized, carrying out ETL processing on the service data to be analyzed by utilizing the server.

A solution system for implementing data visualization and data integration ETL, the system comprising:

the system comprises a creating module, a data processing module and a data transmission module, wherein the creating module is used for acquiring to-be-analyzed business data uploaded by a target user and creating data connection according to the to-be-analyzed business data;

the processing module is used for determining whether integration processing needs to be carried out on the service data to be analyzed, and if so, ETL processing is carried out on the service data to be analyzed to obtain the processed service data to be analyzed;

the replacing module is used for replacing the service data to be analyzed in the data connection with the processed service data to be analyzed;

the drawing module is used for creating a data analysis model according to the data connection and drawing a data report based on the processed business data to be analyzed by using the created data analysis model;

and the issuing module is used for sharing the data report and issuing the data report to the cockpit.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.

The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.

FIG. 1 is a flowchart illustrating a solution for implementing data visualization and data integration ETL according to the present invention;

FIG. 2 is another flowchart of a solution for implementing data visualization and data integration ETL according to the present invention;

FIG. 3 is a flowchart illustrating a solution for implementing data visualization and data integration ETL according to the present invention;

FIG. 4 is a screenshot of an embodiment of a solution for implementing data visualization and data integration ETL according to the present invention;

fig. 5 is a schematic structural diagram of a solution system for implementing data visualization and data integration ETL according to the present invention.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

The data analysis work is beneficial to deep development and utilization of data materials, and the purpose of data analysis is to bring more business values to enterprises, help the enterprises to avoid or reduce loss caused by risks, improve data quality and solve problems for the enterprises. The prior art data analysis method is to perform correlation modeling by using the existing data and then draw a data report for the staff to refer to, but the method has the following disadvantages: the data to be analyzed which is huge in quantity and complex can only be subjected to simple correlation processing, and deeper processing cannot be performed on the data to be analyzed, so that a finally generated data report is too simple and has no reference, and the experience of workers is seriously influenced. In order to solve the above problem, the present embodiment discloses a solution for implementing data visualization and data integration ETL.

A solution for implementing data visualization and data integration ETL, as shown in fig. 1, includes the following steps:

step S101, acquiring to-be-analyzed service data uploaded by a target user, and creating data connection according to the to-be-analyzed service data;

step S102, confirming whether integration processing needs to be carried out on the service data to be analyzed, if so, carrying out ETL processing on the service data to be analyzed to obtain processed service data to be analyzed;

step S103, replacing the service data to be analyzed in the data connection with the processed service data to be analyzed;

step S104, creating a data analysis model according to the data connection, and drawing a data report based on the processed business data to be analyzed by using the created data analysis model;

and step S105, sharing the data report and releasing the data report to the cockpit.

The working principle of the technical scheme is as follows: the method comprises the steps of obtaining business data to be analyzed uploaded by a target user, creating data connection according to the business data to be analyzed, confirming whether integration processing needs to be carried out on the business data to be analyzed, carrying out ETL processing on the business data to be analyzed if the integration processing needs to be carried out on the business data to be analyzed, obtaining the processed business data to be analyzed, replacing the business data to be analyzed in the data connection with the processed business data to be analyzed, creating a data analysis model according to the data connection, drawing a data report based on the processed business data to be analyzed by using the created data analysis model, and sharing and issuing the data report to a cockpit.

The beneficial effects of the above technical scheme are: the ETL processing is carried out on the service data to be analyzed, so that a data integration ETL process can be created, very complex service data can be processed, a more complex data form is provided for data modeling, the output result of the ETL can still be taken as the input of the ETL process again, the processing can be repeatedly carried out according to the service requirement, the final data report is more diversified and accurate, a worker can quickly and intuitively carry out decision-making work according to the final data report, and the use experience of the ETL is improved. The problem of in the prior art can only do simple correlation processing to huge and complicated data to be analyzed, can not carry out deeper processing to the data to be analyzed and consequently the data report that finally generates is too simple not have the referential, has seriously influenced staff's experience sense is solved.

In one embodiment, the acquiring the service data to be analyzed uploaded by the target user and creating a data connection according to the service data to be analyzed includes:

determining the data type of the preprocessed service data to be analyzed;

The beneficial effects of the above technical scheme are: the database table matching work related to the data type can be quickly carried out by determining the data type of the business data to be analyzed, the working efficiency is improved, further, the data in the matching database table can be called for data analysis by constructing data connection between the data type and the matching database table, the analysis sample and the analysis accuracy are provided, and the accuracy of the final data result is ensured.

In an embodiment, as shown in fig. 2, the determining whether integration processing needs to be performed on the service data to be analyzed, and if so, performing ETL processing on the service data to be analyzed to obtain processed service data to be analyzed includes:

step S201, sending a request for judging whether integration processing is needed to be carried out to a terminal where the target user is located, and receiving a feedback instruction sent by the terminal;

step S202, when the feedback instruction confirms that integration processing is required, displaying a plurality of components processed by ETL to the terminal, and acquiring a target component selected by a target user;

step S203, selecting a proper target data connection based on the target component, inputting the service data to be analyzed into the target data connection in a specific format for ETL processing, and obtaining processed service data to be analyzed;

and step S204, outputting the processed service data to be analyzed in the specific format.

The beneficial effects of the above technical scheme are: the target assembly selected by the target user is utilized to carry out ETL processing, so that the user can select different processing assemblies according to the self requirement, the practicability is improved, further, the subsequent data analysis modeling and data integration ETL are seamlessly converted, and the problem of the very complicated data processing can be easily solved by adopting a data integration ETL process.

In one embodiment, the plurality of components includes: a set component, an aggregation component, an expression component, a connection component, a filtering component, a ranking component, and a desensitization component;

the specific format includes: table format and Excel format.

In an embodiment, as shown in fig. 3, the creating a data analysis model according to the data connection, and drawing a data report based on the processed business data to be analyzed by using the created data analysis model includes:

step S301, importing the processed service data to be analyzed in the data connection, and detecting the processed service data to be analyzed to obtain detection information;

step S302, obtaining an attribute factor in the detection information, and mapping the attribute factor to a preset initial data model to obtain a target data analysis model corresponding to the processed to-be-analyzed service data;

step S303, carrying out data analysis on the processed business data to be analyzed by using the target data analysis model to obtain an analysis result;

and step S304, creating a report center, and drawing a data report based on the processed business data to be analyzed in the report center according to the analysis result.

The beneficial effects of the above technical scheme are: the attribute factors are obtained to be mapped to the initial data model so as to obtain a target data analysis model corresponding to the service data to be analyzed, and then data analysis is carried out, so that a unique accurate analysis result of the service data to be analyzed can be obtained, the accuracy of the analysis result is ensured, the influence of interference data is removed, and the accuracy of a data report is further improved.

In one embodiment, the sharing and publishing the data report into the cockpit includes:

The beneficial effects of the above technical scheme are: both can make all people all can look over the data schoolbag, guaranteed the integrality of leading-in data report again when having improved user's experience sense for staff's reference sample is more accurate and then makes reasonable decision.

In one embodiment, the method further comprises:

The beneficial effects of the above technical scheme are: the template can be selected when a subsequent user creates the report center, so that a delicate report can be drawn very quickly, the working efficiency is improved, and meanwhile, the confidentiality of the template can be ensured by setting the encryption key.

In an embodiment, the importing the processed service data to be analyzed in the data connection, detecting the processed service data to be analyzed, and obtaining detection information includes:

The beneficial effects of the above technical scheme are: the processed business data to be analyzed can be detected in a more appropriate and reasonable detection mode by determining the target detection condition of the processed business data to be analyzed, the accuracy of detection information is ensured, a reasonable data base is provided for subsequent modeling and data report generation, furthermore, the detection condition determined by utilizing the abnormal information sequence in the processed business data information to be analyzed can accurately screen out the detection condition which has no influence on the abnormal information sequence so as to keep the integrity detection of the processed business data to be analyzed, and the practicability of the data is further improved.

In one embodiment, before subjecting the service data to be analyzed to ETL processing, the method further includes:

determining the current work task of each target work node;

expressed as a current service rate of the server,

expressed as a target number of working nodes,

denoted as the ith targetThe current usage data volume corresponding to the current work task of the work node,

expressed as the maximum allocated data volume for the ith target worker node,

expressed as the performance index of the ith target working node;

wherein,

expressed as a preset desired length of time,

expressed as the current remaining data volume of the ith target worker node,

The beneficial effects of the above technical scheme are: the busy condition of the server before ETL processing is carried out on the service data to be analyzed can be effectively judged by calculating the load balancing index of the server, whether the ETL processing can be carried out on the service data to be analyzed by using the server on the current basis can be judged, when the ETL processing can be carried out, whether the required time length of the service data to be analyzed under the load balancing index is qualified or not can be evaluated in advance by calculating the processing target time length, and then the server is optimized when the required time length of the service data to be analyzed is unqualified so as to ensure the ETL processing efficiency of the service data to be analyzed, the overlong waiting of a target user is avoided, the working efficiency is improved, and the experience of the user is also improved.

In one embodiment, the method further comprises: and converting the user expression into a final query statement segment of the target database by using a preset expression parser, and setting a conversion system from a standard function to a data source dialect in the created data connection.

The beneficial effects of the above technical scheme are: the preset expression parser can be used for realizing rapid conversion into the same query statement segment aiming at expressions of different modes input by the same person with different expression intentions, so that more persons are suitable for the method, the practicability and the experience feeling of users are improved, further, the conversion system from the standard function to the data source dialect can be effectively used for expanding the range of the actual content of the service data to be analyzed corresponding to the data connection authority to the dialect field, and a democratic data report can be obtained.

In one embodiment, as shown in fig. 4, includes: different types of data connections (including Excel, MySQL, Oracle, SQLServer, RestFul API, etc.) are created, and the data connection types can be combined to be associated with the creation model at will. The report center can refer to different model drawing reports, and the report chart component supports more than 36 types. The drawn data report can support sharing and releasing to a cockpit, sharing refers to sharing report linkage, people who know report linkage are free from logging in to view the report, and the report is released to a place where a pilot of the cockpit centrally views data analysis results, so that data support is provided for leader decision making. The drawn data report can be stored as a template, and the template can be selected when the report center is created, so that the exquisite report can be drawn very quickly. The method can create a data integration ETL process, process very complex business data, output the processing result to the database table or Excel type data connection with connection, provide more complex data form for data modeling, and output result of ETL can be taken as input of the ETL process again, and can be processed repeatedly according to business needs.

The embodiment also discloses a solution system for implementing data visualization and data integration ETL, as shown in fig. 5, the system includes:

a creatingmodule 501, configured to acquire service data to be analyzed uploaded by a target user, and create a data connection according to the service data to be analyzed;

aprocessing module 502, configured to determine whether integration processing needs to be performed on the to-be-analyzed service data, and if yes, perform ETL processing on the to-be-analyzed service data to obtain processed to-be-analyzed service data;

a replacingmodule 503, configured to replace the to-be-analyzed service data in the data connection with the processed to-be-analyzed service data;

adrawing module 504, configured to create a data analysis model according to the data connection, and draw a data report based on the processed to-be-analyzed service data by using the created data analysis model;

and thepublishing module 505 is used for sharing the data report and publishing the data report to the cockpit.

The working principle and the advantageous effects of the above technical solution have been explained in the method claims, and are not described herein again.

It will be understood by those skilled in the art that the first and second terms of the present invention refer to different stages of application.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A solution for realizing data visualization and data integration ETL is characterized by comprising the following steps:

the data report is shared and published into the cockpit.

2. The solution for implementing data visualization and data integration ETL according to claim 1, wherein the acquiring the to-be-analyzed service data uploaded by the target user and creating a data connection according to the to-be-analyzed service data comprises:

determining the data type of the preprocessed service data to be analyzed;

3. The method according to claim 1, wherein the determining whether the business data to be analyzed needs to be integrated, and if so, performing ETL processing on the business data to be analyzed to obtain processed business data to be analyzed includes:

4. The solution for implementing data visualization and data integration ETL as claimed in claim 3, wherein said plurality of components comprises: a set component, an aggregation component, an expression component, a connection component, a filtering component, a ranking component, and a desensitization component;

the specific format includes: table format and Excel format.

5. The solution for implementing data visualization and data integration ETL according to claim 1, wherein the creating a data analysis model according to the data connection and using the created data analysis model to draw a data report based on the processed business data to be analyzed comprises:

6. The ETL solution for data visualization and data integration according to claim 1, wherein the sharing and publishing the data report to the cockpit comprises:

7. The solution for implementing data visualization and data integration ETL according to claim 1, wherein the method further comprises:

8. The method according to claim 5, wherein the step of importing the processed service data to be analyzed in the data connection, detecting the processed service data to be analyzed to obtain detection information includes:

9. The solution for implementing data visualization and data integration ETL according to claim 1, wherein before ETL processing the business data to be analyzed, the method further comprises:

determining the current work task of each target work node;