Disclosure of Invention
Aiming at the defects of the prior art, the technical problem to be solved by the invention is to provide a city operation index monitoring method based on multi-source isomerism, which realizes the convergence, fusion and treatment of isomerism data resources among different departments and industries of city operation and systemizes the quantitative city treatment state.
The technical scheme is that the urban operation index monitoring method based on multi-source isomerism comprises the following steps:
Step 1, determining an urban operation monitoring target and developing a monitoring index design;
step 2, analyzing the monitoring indexes to determine data sources and developing convergence treatment of multi-source data;
step 3, constructing an urban operation index monitoring model with self-adaptive capacity;
And 4, carrying out monitoring application of urban operation indexes in a classified and classified manner.
The step 1 comprises the following steps:
Step 1-1, surrounding the scenes of urban operation such as economic management, urban management, public safety and the like, carrying out panoramic scanning on urban operation, and constructing index items capable of dynamically monitoring urban operation conditions, such as economic indexes of designing GDP growth rate, average person available income, enterprise registration quantity growth rate, tax income and the like, so as to monitor the overall conditions and trends of urban economic development.
Step 1-2, designing different types of indexes, including quantitative indexes, qualitative indexes, trend indexes and comparison indexes, by adopting multidimensional analysis aiming at different application scenes, so as to form a basic index library for daily monitoring of urban operation.
Step 1-3, determining a unique index for each record of an index library, wherein the index library consists of index codes, index names and index attributes. The index code is a unique identifier of the index, the index name is a Chinese name or English abbreviation of the index, a certain index can be uniquely determined, the data index is stored, and the index attribute is used for defining the meaning and extension of the index in detail, including the contents such as index interpretation, calculation method, logic relationship, precision and the like.
And step 1-4, on the basis of planning and classifying indexes and data, a complete standard management flow of an index system is formulated, and the indexes to be issued can be uniformly managed in a changing mode, an evaluating and issuing mode, an index data access mode, on-line training and the like. Meanwhile, the technical means such as a data interface and an API are adopted to realize automatic acquisition and change of data;
And 1-5, designing index items by adopting a four-level system, covering a first-level field, a second-level thematic, a third-level index and a fourth-level data item, wherein aiming at each index, the overall process and technical path of data acquisition, processing, analysis and display at least comprise index names, index units, updating frequency, responsibility departments, alarm thresholds, target thresholds and the like.
The step 2 comprises the following steps:
and 2-1, adopting various data acquisition modes such as an API interface, direct connection of a database, file transmission and the like, and ensuring the diversity and flexibility of data sources. In the data acquisition process, data integrity, consistency and accuracy verification are implemented, such as field non-empty inspection, data type matching, data range verification and the like. Setting reasonable data acquisition frequency according to index requirements and data source characteristics, such as actual acquisition, daily acquisition, weekly acquisition and the like;
And 2-2, performing correlation analysis and cluster analysis on the indexes by using statistical software, a data mining tool and the like, and identifying the indexes of repeatability and relevance. Based on the analysis result, formulating a definite screening and merging strategy, such as reserving the index with the most representation, calculating the comprehensive index and the like;
step 2-3, developing investigation and opinion collection, carrying out statistical analysis, text mining and the like on the collected investigation data, and optimizing selected indexes according to solicited opinions and suggestions to form a set of index system which accords with actual operability;
step 2-4, a data tracing mechanism is established, full-chain information from the source to the mapping index of the data can be obtained, the full-chain information comprises data sources, processing procedures, analysis results and the like, tools such as data query, report generation, data visualization and the like are provided, and a manager can conveniently check and verify the accuracy and the reliability of the data;
Step 2-5, establishing an update monitoring mechanism, formulating a data source update strategy combining regular update and on-demand update, realizing automatic execution of index data update by utilizing technologies such as a timing task, a trigger and the like, tracking the update condition of the data source in real time, and finding and solving the problems in the update process in time;
2-6, automatically labeling and classifying the multi-source data by adopting technical means such as machine learning, natural language processing and the like, automatically adapting the labels according to data sources, constructing an associated mapping list of index items and data resources, establishing a unified index item visual configuration function, and carrying out centralized management and display on the converged multi-source data;
And 2-7, managing tasks for converging data by adopting technologies such as ETL (Extract-Transform-Load), data warehouse and the like, forming tasks and automatically executing data convergence, so as to realize data quality monitoring, data flow monitoring, system performance monitoring and the like in the processes of extracting, cleaning, transforming and loading heterogeneous multi-source data. Synchronously establishing an exception handling mechanism, and timely responding and handling the exception conditions in the data acquisition, processing and transmission processes;
step 2-8, supporting access control mechanisms based on roles, authorities and the like, adopting encryption algorithms such as AES (Advanced Encryption Standard ), RSA (Rivest-Shamir-Adleman) and the like, performing security rule setting, automatic identification of sensitive data, data encryption and desensitization, data classification and classification, minimum access control, data authorization approval, data security watermarking and the like on the converged data, and establishing a data security and credibility space.
The step 3 comprises the following steps:
And 3-1, cleaning the collected data, including removing repeated data, processing missing values, correcting error data and the like. For unstructured data (e.g., text, images, etc.), it is converted to structured or semi-structured data using parsing tools and techniques including OCR (Optical Character Recognition ), NLP (Natural Language Processing, natural language processing);
And 3-2, aiming at the cleaned and analyzed data, reducing the dimension of the data by using dimension reduction technologies such as PCA (PRINCIPAL COMPONENT ANALYSIS ), t-SNE (t-Distributed Stochastic Neighbor Embedding, t-distributed random neighborhood embedding) and the like, reducing the computational complexity and removing redundant information. Meanwhile, converting data of different dimensions into the same dimension by adopting a Z-score standardization method;
and 3-3, defining a prototype of the monitoring index according to the service requirement and the data characteristics, wherein the prototype comprises the name, description, calculation formula and the like of the index. Determining data objects related to the index, including data sources, data fields, data types and the like;
step 3-4, based on the index library prototype and the data object, selecting algorithm frameworks such as linear regression, decision trees, neural networks and the like, initially constructing an original model of the monitoring index, and setting initial parameters of the model;
Step 3-5, setting reasonable weights, thresholds and evaluation standards for the model according to service requirements and data characteristics, and training the model by using historical data or simulation data to adjust parameters of the model to achieve optimal performance;
step 3-6, constructing a knowledge graph through tools such as machine learning, natural language analysis and the like to represent the association and the relation between data, and providing technical support for model optimization;
Step 3-7, performing parameter tuning on the model by using machine learning algorithms such as gradient descent, random gradient descent, adam optimizer and the like, and gradually optimizing weight and bias items of the model in the iterative training process so as to improve the prediction accuracy of the model;
Step 3-8, automatically extracting relation information related to specific indexes from multi-source heterogeneous data by using relation extraction technologies such as a rule-based method and a statistical-based method, mapping the extracted relation information with an index model, and establishing a relation mapping table or a relation map;
Step 3-9, carrying out automatic arrangement management on the extraction flow of the associated data according to the relation mapping table or the relation map, including flow design, task scheduling, error processing and the like, and automatically extracting the associated data related to the specific index;
And 3-10, on-line publishing the verified and optimized index model, outputting a model calculation result by using a data instrument panel, three-dimensional simulation and other visualization technologies, and forming an urban operation panoramic image.
Step 4 comprises:
According to the operation of the steps 1-4, the operation monitoring index design, data aggregation and model establishment work are carried out in the fields of classification and classification, and the operation steps are similar, wherein the key fields of urban operation such as industrial economy, urban treatment, green ecology, happiness and livability are oriented.
The method is characterized in that the method is used for carrying out operation monitoring index design, data aggregation and model establishment work in the urban operation grading classification field, establishing a complete index system standardized management flow by constructing a basic index library and a source data collection management library for daily monitoring of urban operation, carrying out dimension reduction and standardization processing on multi-source heterogeneous data by adopting technical means such as machine learning, natural language processing and the like, selecting algorithm frames such as linear regression, decision trees, neural networks and the like, constructing an original model of monitoring indexes, establishing a relation mapping or relation map of the monitoring indexes and the multi-source data, and realizing urban situation monitoring, urban event perception and urban situation analysis, and perceiving urban operation risks and development trends.
Compared with the prior art, the method has the advantages that by adopting technical means such as machine learning, natural language processing and the like, the method performs dimension reduction and standardization processing on the data and unstructured data of different sources produced in urban operation, performs automatic tag adaptation on data sources and index items, and establishes a relation mapping or relation map of monitoring indexes and multi-source data by constructing an operation monitoring index model, can automatically update and extract model association data corresponding to different indexes, and solves the problem that urban operation monitoring is inaccurate and incomplete under multi-source heterogeneous data.
Detailed Description
Embodiments of the present invention will be described below with reference to the accompanying drawings.
As shown in fig. 1 and 2, the invention provides a city operation index monitoring method based on multi-source heterogeneous, which comprises the following steps:
1. determining an urban operation monitoring target and developing a monitoring index design;
1-1. Determining monitoring index items
Around the economic management scene of urban operation, an index item capable of dynamically monitoring the economic operation condition is constructed. For example, in economic governance, data such as sustainable development of economy, GDP growth rate, average available income, enterprise registration number growth rate, etc.;
Establishing a monitoring index collection mechanism, and periodically analyzing economic indexes by establishing a data sharing channel with a sending change, an industrial letter, a personal society, tax and the like to evaluate the speed and stability of urban economic growth;
1-2 multidimensional analysis and construction of an index library
Aiming at different application scenes, multidimensional analysis is adopted to design different types of indexes, including quantitative indexes, qualitative indexes, trend indexes and comparison indexes, so as to form a basic index library for daily monitoring of urban operation;
Specific numerical indexes such as GDP, average income, number of enterprises on the rule and the like are determined aiming at the indexes capable of being quantitatively analyzed, and a statistical method and a data analysis tool are used for quantitatively analyzing the quantitative indexes;
Aiming at indexes capable of being qualitatively analyzed, descriptive indexes such as economic development situation, economic development business environment and the like are formulated, and the quantitative analysis is carried out by collecting multi-source data such as enterprise transaction, enterprise income, enterprise number and the like and carrying out weight setting;
aiming at trend analysis indexes, indexes capable of reflecting the change trend, such as the change trend of a price index, the increase and decrease trend of employment people and the like, are selected, and the trend indexes are monitored and analyzed by using methods such as time sequence analysis and the like;
Aiming at the comparison analysis indexes, determining indexes which are compared with other cities or regions, such as urban competitive ranking, historical contemporaneous economic development level and the like, collecting relevant comparison data, and carrying out transverse comparison and analysis;
1-3 index encoding and attribute definition
Determining a unique index for each record of an index library, wherein the index library consists of an index code, an index name and an index attribute;
adopting the combination of numbers, letters or other characters to formulate a set of unified coding rules, ensuring that each index has unique codes so as to facilitate identification and management;
Defining the names of the indexes, and determining clear and accurate Chinese names or English abbreviations of the indexes, so that the indexes can be uniquely expressed, and fuzzy or ambiguous words are avoided;
Defining index attributes, and defining the connotation and extension of each index in detail, wherein the contents comprise index interpretation, calculation methods, logic relations, precision and the like, so that the indexes are convenient to maintain and manage;
1-4, formulating an index system management flow
On the basis of planning and classifying indexes and data, a complete standard management flow of an index system is formulated, and the indexes to be issued can be changed, evaluated and issued, an index data access mode, online training and the like are uniformly managed;
The process of making index change comprises the steps of making change application, checking change content, approving change and the like.
Establishing an evaluation release mechanism to ensure the accuracy and reliability of the index;
the data source and the acquisition method of each index are defined, for example, data are collected in the modes of statistics report, sensor monitoring, questionnaire investigation, API interface and the like, a timing task can be configured, and the data can be automatically acquired or pushed;
On-line training is carried out, the capability of data processing and analysis is improved, and deep analysis and mining are carried out on data by utilizing technologies such as machine learning, data mining and the like;
1-5 designing a four-level index system
And a four-level system is adopted to design index items, so that the depth and the fineness of monitoring are ensured. Designing a four-level system, namely, a first-level field (such as economic management), a second-level thematic (such as an industrial structure), a third-level index (such as GDP growth rate) and a fourth-level data item (such as GDP of a first industry);
setting key parameters such as update frequency, responsibility departments, alarm thresholds, target thresholds and the like of corresponding data sources of each index aiming at each index so as to ensure timeliness and effectiveness of monitoring;
2. analyzing and monitoring indexes to determine data sources and develop convergence treatment of multi-source data
2-1 Data acquisition and verification
And determining a proper data acquisition mode according to the data source corresponding to the monitoring index. For a data source with an API interface, acquiring data by calling the API, establishing database connection for data extraction for the data source which can be directly connected with a database, and setting a file transmission mechanism for the data source which provides data in a file form to receive the data;
In the data acquisition process, writing a data verification rule and program, and performing real-time verification on acquired data, for example, checking whether a field is empty, whether a data type is matched with an expected one, whether a data value is in a reasonable range, and the like. Recording and marking the data which does not accord with the verification rule, and adopting corresponding processing measures, such as re-acquisition, data correction or notifying a data source provider to carry out correction;
and reasonably setting the frequency of data acquisition according to the requirements of the monitoring indexes and the update frequency of the data sources. For some data sources which are updated regularly, such as statistical reports, corresponding acquisition frequencies are set according to the release period of the data sources, such as daily acquisition or weekly acquisition;
2-2. Associating index items with data sources
And carrying out correlation analysis and cluster analysis on the monitored indexes by using statistical software (such as SPSS, SAS) and a data mining tool (such as Weka, rapidMiner) to identify indexes of repeatability and relevance. By clustering analysis, grouping the indexes and the data according to the similarity, and finding out an index set with stronger relevance;
2-3 analysis optimization of index items
And carrying out investigation and opinion collection, and carrying out statistical analysis and text mining on the collected investigation data. And carrying out descriptive statistics, correlation analysis and the like on the quantitative data by using a statistical analysis method, and carrying out keyword extraction, semantic analysis and the like on the qualitative data by using a text mining technology. Optimizing the selected indexes to form a set of index system which accords with the actual operability;
2-4 traceability of index data
Recording the whole process information of data from acquisition to processing and analysis, including the source, acquisition time, acquisition mode, processing step, analysis method, result and the like, ensuring the integration with a data acquisition, processing and analysis system and realizing the automatic recording and tracking of the data;
by means of tools such as data query, report generation, data visualization and the like, various reports such as a data summary report, a trend analysis report and the like can be automatically generated according to user requirements, and a manager can conveniently query, analyze and verify the data;
2-5 updating of index data
Defining the updating time interval, updating content and range, and formulating a data source updating strategy combining regular updating and on-demand updating according to the characteristics of the data source and the requirements of monitoring indexes;
For data sources with fixed update period, such as GDP, average income and the like, a periodical update mode is adopted;
Automatically triggering data updating operation when the data source changes by setting a timing task or a trigger;
2-6. Automatic labeling and Classification
The multi-source data is automatically marked and classified by using a machine learning algorithm such as a classification algorithm, a clustering algorithm and the like and natural language processing technologies such as lexical analysis, syntactic analysis, semantic understanding and the like;
Establishing an association mapping list of index items and data resources, and defining data sources and data fields corresponding to each index item;
through visual configuration management of the index items, the user can set the label attribute of the index items, such as index names, labeling methods, display forms and the like;
2-7 data Convergence and administration
The ETL tool and the data warehouse technology are adopted to manage the task of converging data, the ETL tool is used for realizing the operations of extracting, cleaning, converting and loading heterogeneous multi-source data, and the data is integrated into the urban operation management database from each data source;
In the data aggregation process, data quality monitoring, data flow monitoring and system performance monitoring are implemented. By setting monitoring indexes and thresholds, the data quality problem, the data flow abnormality and the system performance bottleneck are found in time;
An exception handling mechanism is established, and timely response and handling are carried out on the exception conditions occurring in the data acquisition, processing and transmission processes. For example, when the data quality is not satisfactory, data cleaning and correction are carried out, and when the data flow is overlarge or the system performance is reduced, optimization measures are adopted or resource allocation is carried out;
2-8 data trusted space
Establishing an access control mechanism based on roles and authorities, managing users and data in a classified manner, and distributing corresponding authorities for data access to users with different roles;
encryption algorithms such as AES, RSA and the like are adopted to encrypt the converged data, so that confidentiality and integrity of the data are protected;
implementing an automatic sensitive data identification technology, automatically identifying and marking sensitive information in the data, and adopting corresponding encryption desensitization measures such as data replacement, data blurring and the like;
Marking and tracking data by adopting a data security watermarking technology, so as to prevent illegal copying and spreading of the data;
3. Building urban operation index monitoring model with self-adaption capability
3-1 Data cleaning and parsing
Performing de-duplication processing on the collected data, removing repeated data records, and ensuring the uniqueness of the data;
the error data is identified and corrected by setting a reasonable data range and logic rules, and error values in the data, such as abnormal values, data which do not accord with logic and the like, are checked and cleaned;
For unstructured data, such as text data, natural Language Processing (NLP) technology is used for analysis, including lexical analysis, syntactic analysis, semantic understanding and the like, and the text data is converted into a structured or semi-structured form;
For unstructured data such as images, an image recognition technology (OCR) is used for converting the unstructured data into text information, or feature information of the images is extracted and converted into structured data which can be used for analysis;
3-2 data dimension reduction and standardization
Selecting a proper dimension reduction technology, mapping high-dimension data to a low-dimension space, reducing the dimension of the data, reducing the computational complexity, processing the cleaned and parsed data by adopting Principal Component Analysis (PCA) or t-distributed random neighborhood embedding (t-SNE), and removing redundant information;
The data is subjected to standardization processing by adopting a Z-Score standardization method, the mean value and standard deviation of each data point are calculated, the data value is converted into standard fraction, so that the data with different dimensions have comparability and are converted into the same dimension;
3-3 definition of monitoring index prototype
Formulating a calculation formula of the index, and definitely calculating data fields and calculation methods related to the index value;
Determining data objects associated with each index, including data sources (e.g., database, file system, etc.), data fields (e.g., specific table fields), data types (e.g., numeric, character, date, etc.);
3-4. Preliminary construction of monitoring model
And selecting a proper algorithm framework according to the characteristics of the index and the distribution condition of the data. If the data has a linear relationship, a linear regression algorithm can be selected, if the data has a complex nonlinear relationship, a neural network algorithm can be selected, if classification and decision are required, a decision tree algorithm can be selected, etc.;
Constructing an original model of the monitoring index according to the index library prototype and the data object by using the selected algorithm frame and tool;
Setting initial parameters of the model, wherein the initial parameters can be set according to an empirical value or a random initialization method, such as coefficient initial values in linear regression, splitting criteria and depth limitation in a decision tree, the number of layers and the number of neurons in a neural network and the like;
3-5 model training and parameter optimization
According to the service requirements and the data characteristics, reasonable parameter weights and index thresholds are set for the model, the weights are used for adjusting the influence degree of different characteristics on the output of the model, and the thresholds are used for determining the boundaries of classification or decision;
dividing the data into a training set, a verification set and a test set, training the model by using historical data or simulation data, and optimizing the performance of the model by adjusting the parameters of the model;
3-6, constructing a knowledge graph
Analyzing and understanding the data by utilizing machine learning and natural language analysis technology, and extracting entity, relationship and attribute information in the data;
using a graph database or a knowledge graph construction tool to represent complex association and relationship among data, and constructing the extracted entity, relationship and attribute information into a knowledge graph;
Combining the constructed knowledge graph with the monitoring index model, and optimizing the structure and parameters of the model through the relation information in the knowledge graph;
3-7 model parameter tuning
And selecting an Adam optimizer to perform parameter tuning on the model. In the iterative training process, gradually adjusting the weight and bias items of the model according to the algorithm and the objective function of the optimizer;
monitoring parameter changes and model performance in the tuning process, and timely adjusting parameters and training strategies of an optimizer according to training results so as to ensure that the model can quickly converge to an optimal solution;
3-8 relation extraction and mapping
Extracting entities and relationships related to the metrics from the text data according to predefined rules and patterns using a rule-based approach; automatically identifying relationship information related to the index through statistical analysis and pattern discovery of the data by using a statistical-based method;
Mapping the extracted relationship information with the index model, establishing a relationship mapping table or relationship map, and defining the corresponding relationship between the index and the relationship information, so that the query and the use are convenient;
3-9. Automated extraction of associated data
Designing an extraction flow of associated data according to a relation mapping table or a relation map, and determining the source of the data, the extraction method and step, the conversion and processing flow of the data and the like;
The workflow engine or the workflow management tool is used for automatically arranging and managing the extraction workflow, so that definition, configuration and execution of the workflow are realized, and the workflow comprises scheduling of tasks, allocation of resources, monitoring and control of the workflow and the like;
3-10 model publishing and visualization
Deploying the verified and optimized index model into a production environment to realize online operation and service provision of the model;
establishing a monitoring and maintenance mechanism of the model, and periodically evaluating and updating the model to ensure the performance and accuracy of the model;
Displaying and outputting a model calculation result by using visualization technologies such as a data instrument panel, three-dimensional simulation and the like;
4. According to the flow, the operation monitoring index design work is carried out in a classified manner in the key fields of urban operation, such as industrial economy, urban management, green ecology, happiness and livability, and the like, and the operation steps are similar.
In a specific implementation, the application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the summary of the urban operation index monitoring method based on multi-source heterogeneous and part or all of the steps in each embodiment when being executed by the data processing unit. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
It will be apparent to those skilled in the art that the technical solutions in the embodiments of the present invention may be implemented by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied essentially or in the form of a computer program, i.e. a software product, which may be stored in a storage medium, and include several instructions to cause a device (which may be a personal computer, a server, a single-chip microcomputer, MUU or a network device, etc.) including a data processing unit to perform the methods described in the embodiments or some parts of the embodiments of the present invention.
The invention provides a city operation index monitoring method based on multisource isomerism, and the method and the way for realizing the technical scheme are numerous, the above is only a preferred embodiment of the invention, and it should be pointed out that a plurality of improvements and modifications can be made to a person skilled in the art without departing from the principle of the invention, and the improvements and the modifications are also regarded as the protection scope of the invention. The components not explicitly described in this embodiment can be implemented by using the prior art.