Disclosure of Invention
In order to overcome the deficiencies of the prior art, at least one technical problem presented in the background art is solved.
The technical scheme adopted by the invention for solving the technical problems is that the data storage method based on the artificial intelligent accelerator comprises the following steps:
Analyzing the characteristic indexes to obtain characteristic index secondary values, further analyzing the characteristic index secondary values to obtain characteristic representation values, and further analyzing the data to be stored based on comparison of the characteristic representation values and characteristic representation threshold values;
The method comprises the steps of obtaining a critical data block quantity value and an information entropy overrun value through analysis processing of an information entropy value, and carrying out weighted ratio processing on the critical data block quantity value and the information entropy overrun value to obtain a critical representation value;
Screening the data to be stored in the data set according to the access frequency to obtain a high-frequency data block quantity ratio and an access frequency deviation ratio, and carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value;
Combining and analyzing the key representation value and the data access value to generate a data sequence table to be stored, and distributing the data to be stored based on the data sequence table to be stored;
The invention further adopts the technical scheme that the characteristic representation value is obtained by the following steps:
counting the unqualified times of feature extraction, and carrying out ratio processing on the unqualified times of feature extraction and the total times of feature extraction to obtain the unqualified ratio of feature extraction;
Summing all the secondary difference ratios of the characteristic indexes, and taking an average value to obtain secondary deviation values of the characteristic indexes;
Marking the characteristic extraction failure ratio as BHG, and marking the characteristic index secondary deviation value as YPC;
Performing data processing on the feature extraction failure ratio BHG and the feature index secondary deviation value YPC, and using a formulaCalculating to obtain a characteristic representation value BX, wherein s1 and s2 are preset proportion coefficients;
the invention further adopts the technical scheme that the characteristic extraction unqualified ratio BHG is obtained by the following steps:
comparing the secondary value of the characteristic index with the secondary threshold value of the characteristic index, wherein the specific comparison process is as follows:
if the feature index secondary value is larger than the feature index secondary threshold value, the feature extraction is unqualified;
if the feature index secondary value is smaller than or equal to the feature index secondary threshold value, the feature extraction is qualified;
In a data storage period, counting the times of unqualified feature extraction, and carrying out ratio processing on the times of unqualified feature extraction and the total times of feature extraction to obtain the unqualified feature extraction ratio;
The invention further adopts the technical scheme that the characteristic index secondary deviation value YPC is obtained by the following steps:
Based on unqualified characteristic indexes, carrying out difference processing on the characteristic index secondary values and the characteristic index secondary threshold values, taking absolute values to obtain characteristic index secondary differences, carrying out ratio processing on the characteristic index secondary differences and the characteristic index secondary threshold values to obtain characteristic index secondary difference ratios, summing all the characteristic index secondary difference ratios, and taking an average value to obtain characteristic index secondary deviation values;
the key representation value GJ is obtained by the following steps:
Carrying out weighted ratio processing on the key data block quantity value and the information entropy overrun value to obtain a key representation value, and marking the key representation value as GJ;
the method for acquiring the critical data block quantity value and the information entropy overrun value comprises the following steps:
Summing all the information entropy values, taking an average value to obtain an information entropy average value, carrying out difference value calculation on the information entropy values and the information entropy average value, and taking an absolute value to obtain an information entropy difference;
Comparing the information entropy difference with a preset information entropy difference threshold value, wherein the specific comparison process is as follows:
If the information entropy difference is smaller than the information entropy difference threshold, marking the data block as a key data block;
if the information entropy deviation is greater than or equal to the information entropy deviation threshold, marking the data block as a non-key data block;
counting the number of key data blocks, and calculating the ratio of the number of the key data blocks to the number of the data blocks to obtain the number value of the key data blocks;
based on any one key information block;
Performing difference calculation on the information entropy difference and an information entropy difference threshold value to obtain information entropy deviation, performing ratio calculation on the information entropy deviation and an information entropy mean value to obtain an information entropy deviation ratio, summing all the information entropy deviation ratios, and taking the mean value to obtain an information entropy super-limit value;
the further technical scheme of the invention is that the data access value FW is obtained by the following steps:
the high-frequency data block quantity ratio and the access frequency deviation ratio are weighted and summed to obtain a data access value, and the data access value is marked as FW;
the invention further adopts the technical scheme that the acquisition mode of the high-frequency data block quantity ratio and the access frequency deviation ratio is as follows:
Counting the number of data blocks exceeding a datum line in the access frequency curve, and performing ratio processing on the number of the data blocks to obtain the number ratio of the high-frequency data blocks;
selecting data points exceeding a datum line in the access frequency curve, and connecting the data points to obtain a high-frequency data curve;
Based on any one data point of the high frequency data curve;
performing difference processing on the access frequency and the access frequency standard value to obtain an access frequency difference, and performing ratio processing on the access frequency difference and the access frequency standard value to obtain an access frequency difference ratio;
Summing all the access frequency difference ratios, and taking an average value to obtain an access frequency deviation ratio;
the further technical scheme of the invention is that the acquisition mode of the stored characterization value CH is as follows:
the obtained key representation value GJ and the data access value FW are subjected to data processing, by the formulaCalculating to obtain a stored characterization value CH, wherein a1 and a2 are preset proportion coefficients;
The feature analysis module is used for preprocessing the data to be stored, extracting features based on the preprocessed data to be stored to obtain feature indexes, analyzing the feature indexes to obtain feature index secondary values, further analyzing the feature index secondary values to obtain feature expression values, comparing the feature expression values with feature expression thresholds, and further analyzing the data to be stored if the feature expression values are smaller than the feature expression thresholds based on comparison results;
The analysis signal processing module is used for obtaining a critical data block quantity value and an information entropy overrun value based on analysis processing of the information entropy value, carrying out weighted ratio processing on the critical data block quantity value and the information entropy overrun value to obtain a critical representation value, screening data to be stored in a data set according to access frequency to obtain a high-frequency data block quantity ratio and an access frequency deviation ratio, and carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value;
And the storage allocation module is used for carrying out combination analysis on the key representation value and the data access value, generating a data sequence table to be stored, and allocating the data to be stored based on the data sequence table to be stored.
The beneficial effects of the invention are as follows:
1. Preprocessing data to be stored, carrying out feature extraction based on the preprocessed data to be stored to obtain feature indexes, analyzing the feature indexes to obtain feature index secondary values, further analyzing the feature index secondary values to obtain feature expression values, comparing the feature expression values with feature expression thresholds, and further analyzing the data to be stored if the feature expression values are smaller than the feature expression thresholds based on comparison results;
2. Generating an information entropy value of a data set according to an information entropy method, obtaining a key data block quantity value and an information entropy overrun value based on analysis processing of the information entropy value, carrying out weighted ratio processing on the key data block quantity value and the information entropy overrun value to obtain a key representation value, screening data to be stored in the data set according to access frequencies, carrying out analysis processing on access frequencies of a data subset according to screening results to obtain a high-frequency data block quantity ratio and an access frequency deviation ratio, carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value, carrying out combined analysis on the key representation value and the data access value to generate a data to be stored sequence table, and carrying out distribution of the data to be stored based on the data to be stored.
Detailed Description
The invention is further described in connection with the following detailed description in order to make the technical means, the creation characteristics, the achievement of the purpose and the effect of the invention easy to understand.
Example 1
The data storage method based on the artificial intelligence accelerator according to the embodiment of the invention shown in fig. 1 comprises the following steps:
preprocessing data to be stored, carrying out feature extraction based on the preprocessed data to be stored to obtain feature indexes, analyzing the feature indexes to obtain feature index secondary values, and further analyzing the feature index secondary values to obtain feature representation values;
the feature indexes comprise access frequency and importance;
In some embodiments, merging data to be stored to obtain a data set, cleaning the data set, removing repeated data in the data set, and performing feature extraction on the data set by using a Recurrent Neural Network (RNN) and variants thereof (such as LSTM) to obtain feature indexes, wherein the feature indexes comprise access frequency and importance of the data to be stored;
Specifically, based on any one of the characteristic indexes;
comparing the characteristic index with a characteristic index standard value, wherein the specific comparison process is as follows:
if the characteristic index is greater than or equal to the characteristic index standard value, the processing is not performed;
if the characteristic index is smaller than the characteristic index standard value, marking as an unimportant characteristic index;
calculating the difference value between the characteristic index and the characteristic index standard value, and taking the absolute value to obtain a characteristic index difference;
It should be noted that, the characteristic index standard value is set by a person skilled in the art through a large amount of historical data, and the purpose is to judge whether the characteristic index in the data storage process meets the requirement;
counting the number of secondary indexes in the characteristic indexes, and carrying out ratio processing on the number of the secondary indexes and the number of the characteristic indexes to obtain the number ratio of the secondary indexes;
based on any secondary indicator;
Calculating the ratio of the characteristic index difference to the characteristic index standard value, processing to obtain a characteristic index difference value, summing all the characteristic index difference values, and taking the average value to obtain a characteristic index difference ratio;
summing the secondary index quantity ratio and the characteristic index difference ratio to obtain a characteristic index secondary value;
comparing the secondary value of the characteristic index with the secondary threshold value of the characteristic index, wherein the specific comparison process is as follows:
if the feature index secondary value is larger than the feature index secondary threshold value, the feature extraction is unqualified;
if the feature index secondary value is smaller than or equal to the feature index secondary threshold value, the feature extraction is qualified;
In a data storage period, counting the times of unqualified feature extraction, and carrying out ratio processing on the times of unqualified feature extraction and the total times of feature extraction to obtain the unqualified feature extraction ratio;
Based on unqualified characteristic indexes, carrying out difference processing on the characteristic index secondary values and the characteristic index secondary threshold values, taking absolute values to obtain characteristic index secondary differences, carrying out ratio processing on the characteristic index secondary differences and the characteristic index secondary threshold values to obtain characteristic index secondary difference ratios, summing all the characteristic index secondary difference ratios, and taking an average value to obtain characteristic index secondary deviation values;
Marking the characteristic extraction failure ratio as BHG, and marking the characteristic index secondary deviation value as YPC;
Performing data processing on the feature extraction failure ratio BHG and the feature index secondary deviation value YPC, and using a formulaCalculating to obtain a characteristic representation value BX, wherein s1 and s2 are preset proportion coefficients;
The feature expression value BX reflects a quantization standard of the secondary degree of the feature index, and the larger the feature expression value BX is, the smaller the feature extraction failure ratio BHG and the feature index secondary deviation value YPC is, whereas the smaller the feature expression value BX is, the larger the feature extraction failure ratio BHG and the feature index secondary deviation value YPC is, and the larger the feature index secondary degree is;
Comparing the characteristic representation value with a characteristic representation threshold value, and further analyzing the data to be stored if the characteristic representation value is smaller than the characteristic representation threshold value based on the comparison result;
Specifically, comparing the feature expression value with a feature expression threshold;
If the feature representation value is larger than or equal to the feature representation threshold value, further analyzing the data to be stored to generate an analysis signal;
If the feature representation value is less than the feature representation threshold, marking the portion of data as secondary data;
The technical scheme of the embodiment of the invention comprises the steps of preprocessing data to be stored, extracting features based on the preprocessed data to be stored to obtain a feature index, analyzing the feature index to obtain a feature index secondary value, further analyzing the feature index secondary value to obtain a feature representation value, comparing the feature representation value with a feature representation threshold value, and further analyzing the data to be stored if the feature representation value is larger than or equal to the feature representation threshold value based on a comparison result.
Example 2
As shown in fig. 2, based on embodiment 1, the data storage method based on the artificial intelligence accelerator according to the embodiment of the invention includes:
generating information entropy values of the data set according to an information entropy method based on analysis signals, obtaining a critical data block quantity value and an information entropy overrun value based on analysis processing of the information entropy values, and carrying out weighted ratio processing on the critical data block quantity value and the information entropy overrun value to obtain a critical representation value;
It should be noted that, the information entropy value reflects the uncertainty or confusion degree of the data, and a larger information entropy value indicates a more uncertain or chaotic data, whereas a smaller information entropy value indicates a more deterministic or ordered data;
in some embodiments, obtaining an information entropy value for each data block in the data set, classifying the data set based on the information entropy value;
Wherein the data block represents data in the dataset;
Summing all the information entropy values, taking an average value to obtain an information entropy average value, carrying out difference value calculation on the information entropy values and the information entropy average value, and taking an absolute value to obtain an information entropy difference;
Comparing the information entropy difference with a preset information entropy difference threshold value, wherein the specific comparison process is as follows:
If the information entropy difference is smaller than the information entropy difference threshold, marking the data block as a key data block;
if the information entropy deviation is greater than or equal to the information entropy deviation threshold, marking the data block as a non-key data block;
counting the number of key data blocks, and calculating the ratio of the number of the key data blocks to the number of the data blocks to obtain the number value of the key data blocks;
based on any one key information block;
Performing difference calculation on the information entropy difference and an information entropy difference threshold value to obtain information entropy deviation, performing ratio calculation on the information entropy deviation and an information entropy mean value to obtain an information entropy deviation ratio, summing all the information entropy deviation ratios, and taking the mean value to obtain an information entropy super-limit value;
Carrying out weighted ratio processing on the key data block quantity value and the information entropy overrun value to obtain a key representation value, and marking the key representation value as GJ;
the key representation value is represented by a quantization standard of importance degree of the data set, wherein the larger the key representation value is, the higher the importance degree of the data set is, and on the contrary, the smaller the key representation value is, the lower the importance degree of the data set is;
screening the data to be stored in the data set according to the access frequency, analyzing and processing the access frequency of the data subset according to the screening result to obtain the high-frequency data block quantity ratio and the access frequency deviation ratio, and carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value;
Dividing the data set into a plurality of data subsets;
based on any one subset of data;
In some embodiments, an X-Y coordinate system is established, wherein the X axis is the data block order, the Y axis is the access frequency, the access frequency values are marked in the coordinate system, and the data points are connected to obtain an access frequency curve;
marking an access frequency standard value as a reference point in an access frequency curve, and making a reference line;
Counting the number of data blocks exceeding a datum line in the access frequency curve, and performing ratio processing on the number of the data blocks to obtain the number ratio of the high-frequency data blocks;
selecting data points exceeding a datum line in the access frequency curve, and connecting the data points to obtain a high-frequency data curve;
Based on any one data point of the high frequency data curve;
performing difference processing on the access frequency and the access frequency standard value to obtain an access frequency difference, and performing ratio processing on the access frequency difference and the access frequency standard value to obtain an access frequency difference ratio;
Summing all the access frequency difference ratios, and taking an average value to obtain an access frequency deviation ratio;
the high-frequency data block quantity ratio and the access frequency deviation ratio are weighted and summed to obtain a data access value, and the data access value is marked as FW;
It should be noted that the data access value reflects a quantization standard of the data set access frequency, and the higher the data access value is, the higher the data set access frequency is, whereas the lower the data access value is, the lower the data set access frequency is;
Comparing the data access value with a data access threshold, wherein the specific comparison process is as follows:
If the data access value is greater than or equal to the data access threshold, the data subset is accessed at high frequency and marked as hot data;
if the data access value is smaller than the data access threshold value, the data subset is accessed at a low frequency and marked as cold data;
additional partitioning criteria for hot and cold data are:
From the time dimension, marking data within a specified time range as hot data, and conversely as cold data, wherein the specified time range comprises three months, five months and twelve months;
Fifthly, carrying out combined analysis on the key representation value and the data access value to generate a data sequence table to be stored, and carrying out distribution of the data to be stored based on the data sequence table to be stored;
Specifically, the obtained key representation value GJ and the data access value FW are subjected to data processing, and the key representation value GJ and the data access value FW are processed according to the formulaCalculating to obtain a stored characterization value CH, wherein a1 and a2 are preset proportion coefficients;
in some embodiments, sorting the data to be stored according to the storage characterization value CH in order from large to small to generate a storage order table;
Dividing the data storage unit into a secondary storage unit and a primary storage unit;
Storing the secondary data into a secondary storage unit according to the comparison result of the characteristic representation values, and sequentially storing the data into a primary storage unit according to a storage sequence table;
Acquiring an access frequency value of the data to be stored in real time, re-acquiring a characteristic representation value according to the real-time access frequency value, and detecting the data to be stored according to a comparison result of the characteristic representation value and a characteristic representation threshold;
in some embodiments, the feature performance threshold is adjusted higher if the data to be stored is stored back and forth in the primary storage unit and the secondary storage unit.
The method comprises the steps of generating information entropy values of a data set according to an information entropy method, analyzing and processing based on the information entropy values to obtain a key data block quantity value and an information entropy overrun value, carrying out weighted ratio processing on the key data block quantity value and the information entropy overrun value to obtain a key representation value, screening data to be stored in the data set according to access frequencies, analyzing and processing data subset access frequencies according to screening results to obtain a high-frequency data block quantity ratio and an access frequency deviation ratio, carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value, carrying out combined analysis on the key representation value and the data access value to generate a data to be stored sequence table, and carrying out distribution of the data to be stored based on a data to be stored sequence table.
Example 3
The data storage method based on the artificial intelligence accelerator comprises the following steps:
classifying the data to be stored through an artificial intelligent accelerator, classifying importance aspects of the data to be stored, and dividing the data storage units based on importance degrees;
The specific process of the artificial intelligence accelerator to distinguish the stored data is as follows:
S1, preprocessing data to be stored by using a data cleaning tool (such as Pandas and the like), extracting information such as data access frequency, modification time, data size, data type and the like of the data, and cleaning, denoising and formatting the data to be stored as one of the characteristics of data importance so as to ensure the quality and consistency of the data;
S2, screening out the most valuable features by using a feature selection algorithm, reducing the complexity and the calculated amount of the model, screening out the features which are most valuable for evaluating the importance of the data, carrying out normalization, standardization and other treatments on the features, and ensuring that the weights of different features in the model are the same so as to improve the stability and the accuracy of the model;
S3, selecting a proper machine learning or deep learning model (such as a decision tree, a random forest, a neural network and the like), training by utilizing historical data and characteristics, and configuring an artificial intelligent accelerator (such as a GPU) to accelerate the training process so as to obtain a trained model;
S4, inputting the data to be evaluated into the trained model to obtain importance scores of the data, sorting the data according to the score results, and determining the sequence of storing the data;
S5, continuously optimizing and adjusting the model according to the actual application scene and feedback, and processing the importance evaluation task of the stored data in real time by utilizing the computing capability of the accelerator to ensure the timeliness and accuracy of the data;
the data storage unit is divided into a core subunit and an edge subunit, and the data storage units are sequentially stored according to the importance of the data to be stored.
Example 4
An artificial intelligence accelerator based data storage system according to an embodiment of the invention as shown in fig. 3 includes:
The feature analysis module is used for preprocessing the data to be stored, extracting features based on the preprocessed data to be stored to obtain feature indexes, analyzing the feature indexes to obtain feature index secondary values, further analyzing the feature index secondary values to obtain feature expression values, comparing the feature expression values with feature expression thresholds, and further analyzing the data to be stored if the feature expression values are smaller than the feature expression thresholds based on comparison results;
The analysis signal processing module is used for obtaining a critical data block quantity value and an information entropy overrun value based on analysis processing of the information entropy value, carrying out weighted ratio processing on the critical data block quantity value and the information entropy overrun value to obtain a critical representation value, screening data to be stored in a data set according to access frequency to obtain a high-frequency data block quantity ratio and an access frequency deviation ratio, and carrying out weighted summation on the high-frequency data block quantity ratio and the access frequency deviation ratio to obtain a data access value;
And the storage allocation module is used for carrying out combination analysis on the key representation value and the data access value, generating a data sequence table to be stored, and allocating the data to be stored based on the data sequence table to be stored.
The foregoing has shown and described the basic principles, principal features and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and that the above embodiments and descriptions are merely illustrative of the principles of the present invention, and various changes and modifications may be made without departing from the spirit and scope of the invention, which is defined in the appended claims. The scope of the invention is defined by the appended claims and equivalents thereof.