Disclosure of Invention
The invention aims to provide a big data synchronization method based on a distributed network, which can synchronize the process of playing back big data such as flight data, audio data and video well, can be applied to synchronous playback of various data in a local area network, and solves the synchronization problem and the comprehensive playback problem among various flight data.
The invention discloses a big data synchronizing method based on a distributed network, which is characterized by comprising the following steps:
Step 001, performing visual analysis on the big data to be synchronized by using the spatial similarity, performing spatial distribution visual analysis on the data to be synchronized, decomposing the big data of the graph structure, and decomposing and describing the big data of the graph structure into a relation between the big data attribute to be synchronized and the big data attribute to be synchronized;
Step 002, performing time sequence visualization processing on the big data to be synchronized of the spatial distribution visual analysis by using sequence relation mapping, and storing the big data to be synchronized of the big data attributes and the stored data to be synchronized of the big data attributes by applying a distributed database HBase table;
Step 003, initializing parameters, namely carrying out dynamic synchronization analysis on big data to be synchronized by adopting a dynamic visualization analysis method and a shadow tracking feature visualization analysis method of the big data to be synchronized of a learning model, wherein the initial big data to be synchronized comprises attribute identification of the big data to be synchronized, the data to be synchronized, column members in a synchronous column group, synchronization depth and a queue;
Step 004, searching the big data attribute to be synchronized, inserting the big data attribute to be synchronized into the queue, ending the synchronization when the current synchronization depth is larger than the initialized synchronization depth and the queue is empty, and synchronizing column members in a synchronization column group in the HBase table corresponding to the big data attribute to be synchronized by the first synchronization queue.
Further, in the step 001, visual analysis is performed on the big data to be synchronized by using spatial similarity, spatial distribution visual analysis is performed on the big data to be synchronized, the big data of the graph structure is decomposed, and the big data of the graph structure is decomposed and described as a relationship between the big data attribute to be synchronized and the big data attribute to be synchronized, and the method further includes:
Combining the plurality of visual views with the time information to determine a population of aircraft in an active state for a period of time;
Determining the relation between data in spaces with different attributes, and reflecting the use condition of the flight program space by using the occupation conditions of the two areas;
analyzing the efficiency of the flight procedure according to the interval between the airplanes, classifying the flight procedure and displaying the difference in the same kind;
generating a trajectory based on the raw data, normalizing the data to organize the data in time series;
Adopting the attribute of big data to be synchronized to reflect the relation among the attributes of the big data to be synchronized, and distinguishing the attributes of the big data to be synchronized of different big data by utilizing the attribute identification of the big data to be synchronized;
Analyzing the space characteristics of the aircraft, generating a spiral structure view and a safety view to display the space utilization rate of a flight program and the state of the aircraft interval, and generating a track;
Defining an identifier capable of distinguishing each big data to be synchronized big data attribute in the global as a big data attribute identifier to be synchronized, wherein the big data attribute identifier to be synchronized corresponds to the big data attribute to be synchronized of the big data one by one;
Constructing a similarity matrix between tracks by using a fast dynamic time warping algorithm, and classifying the tracks by spectral clustering; features of the flight path data are extracted, and then the actually used flight procedure is reconstructed in the terminal area according to the irregular flight path data.
Further, step 002 performs time sequence visualization processing on the big data to be synchronized in the spatial distribution visual analysis by using sequence relation mapping, stores big data attributes to be synchronized and data stored in association between the big data attributes to be synchronized by applying a distributed database HBase table, and further includes:
representing delay data in the space according to the time sequence relation, and displaying the relation among areas to determine and compare the propagation of the delay;
The attribute identification of the big data to be synchronized is converted into a recorded line keyword; the contact attribute, the synchronous attribute and other attributes are respectively converted into a contact column group, a synchronous column group and other column groups of the HBase table; the data or the data field to be synchronized is stored in the corresponding column members of the synchronous column family;
the visual form of the delay impact factor is defined to be an uncertain weight of the visual impact factor over time in different areas.
Further, the initializing parameters in step 003 include a to-be-synchronized start big data attribute identifier, to-be-synchronized data, and column members, synchronization depth and queues in a synchronization column group, and the to-be-synchronized big data dynamic synchronization analysis is performed by adopting a to-be-synchronized big data dynamic visualization analysis method of a learning model and a to-be-synchronized big data feature visualization analysis method of light shadow tracking, and further includes:
marking the peak value of the scalar field corresponding to the density map as a target in a training set, and identifying a low-density target in the gray map by the model according to a training result;
calculating the optical flow of all pixels in the image based on a dense optical flow method, selecting an optical flow algorithm to obtain an optical flow field of a pair of images, and calculating a motion field between the pair of images;
Setting parameters to generate dynamic Berlin noise motions, wherein initial parameters generate density map animation comprising a small number of density clusters, the significance of the targets is high, and the grabbing interval is short;
resetting parameters, increasing the size of a single density bolus and the number of the density boluses contained in a single area, and expanding the grabbing range and the interval of data collection;
Expanding the collected data set, wherein the expanded data set is also used for model training, and repeating the steps until enough data is collected after a certain amount of data is obtained;
decomposing a multi-target tracking task into target detection and motion detection by adopting a multi-target tracking model, extracting image features, and selecting ResNet to replace VGG in an original model;
The possible states of the moving object are predicted, and the predicted object states are compared with the object measurement states at the next time. The algorithm corrects the matching relation according to the difference between the predicted state and the real state of the target, finally confirms the process of the position movement of the target along with time, and then further describes the movement trend of the target.
Further, step 004 searches for the big data attribute to be synchronized according to the big data attribute identifier to be synchronized, inserts the big data attribute to be synchronized into the queue, and finishes the synchronization when the current synchronization depth is greater than the initialized synchronization depth and the queue is empty, and includes:
When the depth of the non-accessed adjacent big data to be synchronized of the big data attribute to be synchronized of the head of the queue is larger than the synchronization depth, adding the non-accessed adjacent big data attribute to be synchronized of the head of the queue into the queue, recording the current synchronization depth of the accessed big data attribute to be synchronized of the head of the queue and the big data attribute to be synchronized of the head of the queue, and deleting the big data attribute element to be synchronized of the head of the queue;
judging whether the current synchronization depth is larger than the initialized synchronization depth and whether the queue is empty or not is established, ending the synchronization if the current synchronization depth is established, otherwise, turning to step 002.
The invention realizes a big data synchronization method based on a distributed network, and has the advantages compared with the prior art that: the visual analysis of highly dense track data in the terminal area can be analyzed, and the movements of multiple targets in a longer time sequence can be tracked through the improvement of multiple modules by combining a deep learning multiple-target detection model and a target tracking frame of a filtering method; the information tracked by the final frame contains more details of targets through optical flow information training and detail enhancement series methods based on image features; finally, the frame is used for tracking the dynamic big data information to be synchronized in a plurality of scenes, a plurality of evaluation methods are designed, the accuracy and the efficiency of the tracking method are effectively proved, the local data synchronization of the big data of the directed graph and the undirected graph is supported, and the links among the big data attributes to be synchronized are easily added and deleted, namely, the links among the big data attributes to be synchronized are easily maintained.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the device or element being referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, features defining "first", "second" may include one or more such features, either explicitly or implicitly. In the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
The invention discloses a big data synchronizing method based on a distributed network, which is characterized by comprising the following steps:
Step 001, performing visual analysis on the big data to be synchronized by using the spatial similarity, performing spatial distribution visual analysis on the data to be synchronized, decomposing the big data of the graph structure, and decomposing and describing the big data of the graph structure into a relation between the big data attribute to be synchronized and the big data attribute to be synchronized;
Step 002, performing time sequence visualization processing on the big data to be synchronized of the spatial distribution visual analysis by using sequence relation mapping, and storing the big data to be synchronized of the big data attributes and the stored data to be synchronized of the big data attributes by applying a distributed database HBase table;
Step 003, initializing parameters, namely carrying out dynamic synchronization analysis on big data to be synchronized by adopting a dynamic visualization analysis method and a shadow tracking feature visualization analysis method of the big data to be synchronized of a learning model, wherein the initial big data to be synchronized comprises attribute identification of the big data to be synchronized, the data to be synchronized, column members in a synchronous column group, synchronization depth and a queue;
Step 004, searching the big data attribute to be synchronized, inserting the big data attribute to be synchronized into the queue, ending the synchronization when the current synchronization depth is larger than the initialized synchronization depth and the queue is empty, and synchronizing column members in a synchronization column group in the HBase table corresponding to the big data attribute to be synchronized by the first synchronization queue.
In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
Further, in the step 001, visual analysis is performed on the big data to be synchronized by using spatial similarity, spatial distribution visual analysis is performed on the big data to be synchronized, the big data of the graph structure is decomposed, and the big data of the graph structure is decomposed and described as a relationship between the big data attribute to be synchronized and the big data attribute to be synchronized, and the method further includes:
Combining the plurality of visual views with the time information to determine a population of aircraft in an active state for a period of time;
Determining the relation between data in spaces with different attributes, and reflecting the use condition of the flight program space by using the occupation conditions of the two areas;
analyzing the efficiency of the flight procedure according to the interval between the airplanes, classifying the flight procedure and displaying the difference in the same kind;
generating a trajectory based on the raw data, normalizing the data to organize the data in time series;
Adopting the attribute of big data to be synchronized to reflect the relation among the attributes of the big data to be synchronized, and distinguishing the attributes of the big data to be synchronized of different big data by utilizing the attribute identification of the big data to be synchronized;
Analyzing the space characteristics of the aircraft, generating a spiral structure view and a safety view to display the space utilization rate of a flight program and the state of the aircraft interval, and generating a track;
Defining an identifier capable of distinguishing each big data to be synchronized big data attribute in the global as a big data attribute identifier to be synchronized, wherein the big data attribute identifier to be synchronized corresponds to the big data attribute to be synchronized of the big data one by one;
Constructing a similarity matrix between tracks by using a fast dynamic time warping algorithm, and classifying the tracks by spectral clustering; features of the flight path data are extracted, and then the actually used flight procedure is reconstructed in the terminal area according to the irregular flight path data.
Data processing and visualization are carried out based on the spatial similarity, the data construction comprises time, flight number, longitude, latitude, altitude, ground speed and course, the signals are collected according to time sequence, and the coverage area is larger than the actual terminal area; it is necessary to determine which part of the collected data belongs to the terminal area under investigation and the aircraft data on the high-altitude course may interfere with the modeling. Second, different aircraft transmit signals at different frequencies. Some aircraft data are too few to model for some time, and should be properly excluded. Finally, aircraft data collected over a period of time may be part of the complete flight process, and how to correctly classify trajectories based on the partial data is also a challenge. In addition, as described above, the spatial analysis of the terminal area is microscopic compared to the course analysis, and thus the screening of data cannot be too extensive, but in order to avoid unnecessary calculation overhead, it is necessary to simplify the data; unifying the data, rasterizing the track into a grid by using a Bresenham algorithm, and resampling the track; the similarity matrix is constructed by calculating the similarity between any two tracks, track data are ordered according to flight numbers, and the following algorithm is adopted:
aX,Y＝Dist(Xi,Yj)+min[D(Xi-1,Yj),D(Xi,Yj-1),D(Xi-1,Yj-1)]
Wherein a is an element in a distance matrix, X and Y represent two time series data corresponding to tracks acquired by different aircraft, i and j respectively mean lengths corresponding to the X and Y data, the calculated distances are based on Euclidean distances, and a dynamic programming technology is used in the calculation process. The greater the difference between the two sequences X and Y, the higher will be the value of aX,Y. And further constructing similarity matrixes, abbreviated as sim matrixes, of all the tracks in the terminal area. Each element in sim is the difference between the maximum in the distance matrix calculated by the above formula and the corresponding element. Sim is set as a kernel function (e.g., RBF kernel function), and eigenvalues are then calculated using a diagonal matrix and a laplace matrix. In the next step, the data is reclustered from a new matrix of feature vectors.
Based on the airspace information overview of the statistical chart, the visualization of the parallel coordinate axes is improved to represent the use of airspace heights of different periods; based on the safety monitoring of the relative position vector view, the center circle is set as the minimum safety interval (5 km) in the civil aviation system, and the area between the two circles will be changed in proportion according to the difference of the set values. The security interval value may be increased if the user wants to view the data in a larger spatial scale. Conversely, if the value of the horizontal safety interval is reduced, the same inter-element distance in the view will be increased. In this view, the center of the view corresponds to the location of the airport runway, and all aircraft locations are switched to polar coordinates. Discrete color coding rules are selected based on spatial visualization of the spiral structure model. Through the color change of the spiral structure, the user can know the average speed of the aircraft in different areas. Safety interval analysis, by reducing the difference in altitude between flights, finds some aircraft that are too close within the same altitude. The safety interval standard is then reduced to 10 km, a value that is typically used as a universal aviation safety standard.
Further, step 002 performs time sequence visualization processing on the big data to be synchronized in the spatial distribution visual analysis by using sequence relation mapping, stores big data attributes to be synchronized and data stored in association between the big data attributes to be synchronized by applying a distributed database HBase table, and further includes:
representing delay data in the space according to the time sequence relation, and displaying the relation among areas to determine and compare the propagation of the delay;
The attribute identification of the big data to be synchronized is converted into a recorded line keyword; the contact attribute, the synchronous attribute and other attributes are respectively converted into a contact column group, a synchronous column group and other column groups of the HBase table; the data or the data field to be synchronized is stored in the corresponding column members of the synchronous column family;
the visual form of the delay impact factor is defined to be an uncertain weight of the visual impact factor over time in different areas.
The main information in the data comprises: flight number, departure airport, destination, estimated departure time, actual departure time, estimated arrival time, actual arrival time, and delay time. According to the dynamic delay expression of the Poisson distribution design, a Poisson distribution centered on an airport of interest is generated by a rapid Poisson disk sampling mode, and each data point contained in the distribution indicates a delay event. To ensure that each region has a consistent distribution shape, poisson discs are pre-generated in the system as templates to ensure that the data points in this job all follow the same poisson distribution.
The center of the initial poisson distribution is located at the airport location of interest. When the delay level is stepped, the center of the distribution will shift to the direction of the delayed source data, and the newly generated points will be distributed around the updated center of the distribution. Executing the previous step on all relevant airports; each iteration creates a new point on the basis of the existing poisson distribution shape or around the new distribution due to delayed level changes. If a deferred step occurs, the cores move the same distance as the poisson distribution interval and it is checked whether the new core positions overlap with the existing cores.
Generating a delay variation field, and generating a dynamic delay poisson distribution based on the delay events of the continuous time sequence; the contour map is converted according to the 4 delay levels to display the delay state at time t. In experiments, when the color thermodynamic diagram is converted to a contour map, the density of the data generation will be used as a scalar value to calculate the contour. The changing fields between two successive delayed thermal forces are calculated using STREAMMAP and then displayed by the distribution of arrows. Arrows are used to indicate the trend of the delay variation and the vector field is projected onto the contour plane. The density of the arrows indicating the field change is changed according to the length of the boundary, thereby ensuring that the direction of change can be clearly displayed regardless of the length of the field lines.
Further, the initializing parameters in step 003 include a to-be-synchronized start big data attribute identifier, to-be-synchronized data, and column members, synchronization depth and queues in a synchronization column group, and the to-be-synchronized big data dynamic synchronization analysis is performed by adopting a to-be-synchronized big data dynamic visualization analysis method of a learning model and a to-be-synchronized big data feature visualization analysis method of light shadow tracking, and further includes:
marking the peak value of the scalar field corresponding to the density map as a target in a training set, and identifying a low-density target in the gray map by the model according to a training result;
calculating the optical flow of all pixels in the image based on a dense optical flow method, selecting an optical flow algorithm to obtain an optical flow field of a pair of images, and calculating a motion field between the pair of images;
Setting parameters to generate dynamic Berlin noise motions, wherein initial parameters generate density map animation comprising a small number of density clusters, the significance of the targets is high, and the grabbing interval is short;
resetting parameters, increasing the size of a single density bolus and the number of the density boluses contained in a single area, and expanding the grabbing range and the interval of data collection;
Expanding the collected data set, wherein the expanded data set is also used for model training, and repeating the steps until enough data is collected after a certain amount of data is obtained;
decomposing a multi-target tracking task into target detection and motion detection by adopting a multi-target tracking model, extracting image features, and selecting ResNet to replace VGG in an original model;
The possible states of the moving object are predicted, and the predicted object states are compared with the object measurement states at the next time. The algorithm corrects the matching relation according to the difference between the predicted state and the real state of the target, finally confirms the process of the position movement of the target along with time, and then further describes the movement trend of the target.
Creating a density map animation and acquiring a data set in the form of a density map from the density map animation, and cutting and resizing the image to be used as data to be learned. The multi-object in the two finally divided pictures is taken as two non-intersection sets in the two-part pictures, each object can be regarded as vertex processing in the pictures, and the object matching problem is converted into the vertex matching problem. The algorithm finally recursively converts all vertices in the set to saturation points to ensure that all vertices have a matching relationship.
Further, step 004 searches for the big data attribute to be synchronized according to the big data attribute identifier to be synchronized, inserts the big data attribute to be synchronized into the queue, and finishes the synchronization when the current synchronization depth is greater than the initialized synchronization depth and the queue is empty, and includes:
When the depth of the non-accessed adjacent big data to be synchronized of the big data attribute to be synchronized of the head of the queue is larger than the synchronization depth, adding the non-accessed adjacent big data attribute to be synchronized of the head of the queue into the queue, recording the current synchronization depth of the accessed big data attribute to be synchronized of the head of the queue and the big data attribute to be synchronized of the head of the queue, and deleting the big data attribute element to be synchronized of the head of the queue;
judging whether the current synchronization depth is larger than the initialized synchronization depth and whether the queue is empty or not is established, ending the synchronization if the current synchronization depth is established, otherwise, turning to step 002.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.