CN104881481A - Method and device for accessing mass time sequence data - Google Patents

Abstract

The invention relates to the technical field of data cloud storage, in particular to a method and a device for accessing mass time sequence data. The method includes: receiving time sequence data which include a data source identity, data time and data information; enabling the data source identity to generate a plurality of bytes through a Hash algorithm, and creating a plurality of catalogs corresponding to the bytes in a one-to-one manner; storing the time sequence data into the catalogs in a form of a data file which includes a data section and an index section, wherein a file name of the data file is generated by the data time; receiving an inquiry request which includes the data source identity and the data time, and reading corresponding time sequence data from the catalogs according to the inquiry request and the index section. The time sequence data are accessed by combining a means of the data file and index information, so that access efficiency is improved.

Description

A kind of method and device accessing magnanimity time series data

Technical field

The present invention relates to data cloud technical field of memory, be specifically related to a kind of method and the device that access magnanimity time series data.

Background technology

Along with Internet of Things, the broad development of the methods such as cloud storage and application, the data how storing and retrieve magnanimity and time correlation become the difficult problem that system is applied, time series data refers to the data be closely related with the time, the access of data except the key word with data itself mutually outside the Pass also closely related with the time, the structure stored is usually according to time-sequencing, usually retrieve according to data key words and time during retrieval, at specific area, magnanimity time data has himself feature: data and time height correlation, data are usually according to time order and function sequential delivery and storage, data itself are little, but data volume is huge, and storage time is long, and storing frequencies is high, usually mass data is had to need to store all the time, but retrieve data only has when service needed just triggers, this just causes the operation of retrieve data much smaller than the operation storing data, time series data generally comprises: Data Identification, time and numerical value, in actual applications, time series data adopts the mode of database to access usually, has certain limitation.

Summary of the invention

The object of the invention is to propose a kind of method and the device that access magnanimity time series data, in conjunction with index information, time series data can be accessed by the mode of data file, improve access efficiency.

For reaching this object, the present invention by the following technical solutions:

Access a method for magnanimity time series data, comprising:

S10, time of reception sequence data, described time series data comprises data source identification, data time and data message;

S20, described data source identification is generated several bytes by hash algorithm, set up and several bytes described some layers of catalogue one to one;

S30, be stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

S40, reception inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

Preferably, described data segment is used for recording described data message, and described index segment is used for essential information and the index information of recording data files.

Preferably, described essential information comprises data file start time, end of data file time and data source identification, described index information comprises the first index information and the second index information, described first index information records start time corresponding to each data segment and data file deviation post in order, and described second index information comprises wall scroll index data size, the total size of index segment information, end-of-file position and file overall length.

Preferably, described step S20 is specially:

Described data source identification is generated the shape data of four bytes by hash algorithm, by last byte of described four bytes, successively forward, set up and described four bytes root directory, one deck catalogue, two layers of catalogue and three layers of catalogue one to one.

Preferably, described step S30 comprises:

S31, search in catalogue all with the data file of .data ending, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time .data;

If S32 finds with the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount;

S33, be written in described data file by described time series data, the write reference position of described time series data is said write side-play amount;

S34, write index information, be written to described first index information in data segment index corresponding to end-of-file, be written in the secondary index of end-of-file by described second index information.

Preferably, also comprise after described step S34:

If the described data file of step S35 reaches data segment maximum length, close described data file, set up new data file and write.

Preferably, described step S40 comprises:

S41, reception inquiry request, described inquiry request comprises data source identification and data time;

S42, the data source identification of described inquiry request is generated several bytes by hash algorithm, each byte is converted into the character string of 16 systems, and searches corresponding catalogue;

S43, the filename under described catalogue to be sorted according to inverted order, find out first filename less than the data time of described inquiry request;

S44, open data file corresponding to described filename, and the index information at file reading end;

S55, read the data message of the time series data of inquiry request according to described index information.

As another aspect of the present invention, a kind of device accessing magnanimity time series data provided, comprising:

Receiver module, for time of reception sequence data, described time series data comprises data source identification, data time and data message;

Modular converter, for described data source identification is generated several bytes by hash algorithm, sets up and several bytes described some layers of catalogue one to one;

Memory module, for being stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

Read module, for receiving inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

Preferably, described memory module comprises:

Data file sets up unit, all with the data file of .data ending for searching in catalogue, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time .data;

Side-play amount writing unit, for finding with after the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount;

Data message writing unit, for being written in described data file by described time series data, the write reference position of described time series data is said write side-play amount;

Index information writing unit, for writing index information, being written to described first index information in data segment index corresponding to end-of-file, being written in the secondary index of end-of-file by described second index information.

Preferably, described read module comprises:

Inquiry request receiving element, for receiving inquiry request, described inquiry request comprises data source identification and data time;

Inquiry request converting unit, for the data source identification of described inquiry request is generated several bytes by hash algorithm, is converted into the character string of 16 systems, and searches corresponding catalogue by each byte;

Sequencing unit, for sorting according to inverted order to the filename under described catalogue, finds out first filename less than the data time of described inquiry request;

Index information reading unit, for opening data file corresponding to described filename, and the index information at file reading end;

Data message reading unit, for reading the data message of the time series data of inquiry request according to described index information.

Beneficial effect of the present invention is: a kind of method and device accessing magnanimity time series data, and the method comprises: time of reception sequence data, and described time series data comprises data source identification, data time and data message; Described data source identification is generated several bytes by hash algorithm, sets up and several bytes described some layers of catalogue one to one; Be stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time; Receive inquiry request, described inquiry request comprises data source identification and data time, from described catalogue, corresponding time series data is read according to described inquiry request and index segment, the present invention is by the mode of data file and access time series data in conjunction with index information, improves access efficiency.

Accompanying drawing explanation

Fig. 1 is a kind of process flow diagram accessing the method for magnanimity time series data that the embodiment of the present invention one provides

Fig. 2 is a kind of process flow diagram accessing the method for magnanimity time series data that the embodiment of the present invention two provides

Fig. 3 is a kind of process flow diagram accessing the method for magnanimity time series data that the embodiment of the present invention three provides

Fig. 4 is a kind of structural drawing accessing the device of magnanimity time series data that the embodiment of the present invention four provides.

Fig. 5 is the structural drawing of the device of the memory module that the embodiment of the present invention five provides.

Fig. 6 is the structural drawing of the device of the read module that the embodiment of the present invention six provides.

Embodiment

Technical scheme of the present invention is further illustrated by embodiment below in conjunction with Fig. 1-Fig. 6.

Embodiment one

Fig. 1 is a kind of process flow diagram accessing the method for magnanimity time series data that the present embodiment provides.

Access a method for magnanimity time series data, comprising:

S10, time of reception sequence data, described time series data comprises data source identification DKey, data time Time and data message Value;

S20, described data source identification is generated several bytes by hash algorithm, set up and several bytes described some layers of catalogue one to one;

S30, be stored in described catalogue by described time series data with the form of data file DFile, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

S40, reception inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

In the present embodiment, in conjunction with index information, time series data is accessed by the mode of data file, be no longer dependent on database of the prior art, improve access efficiency.

In the present embodiment, described data segment is used for recording described data message, and described index segment is used for essential information and the index information of recording data files.

In the present embodiment, shown data segment information comprises:

Data segment maximum length maxsize, according to actual conditions configuration, such as 1GB, when file storage is greater than this value, closes this file, creates new file and stores.

Data block size: BlockSize, by configuration setting, acquiescence 1MB.

Data block maximum number: MaxBlockNum is calculated by maxsize/BlockSize, acquiescence position 1K.

The Offset that data block sequence number: BlockSeqno writes by current data calculates (BlockSeqno=Offset*MaxBlockNum/maxsize).

In the present embodiment, described essential information comprises data file start time, end of data file time and data source identification, described index information comprises the first index information and the second index information, data bin data is divided into MaxBlockNum block by described first index information, record start time corresponding to each data segment and data file deviation post in order, described second index information chooses 4 records in MaxBlockNum record of Part I, comprises wall scroll index data size, the total size of index segment information, end-of-file position and file overall length.

In the present embodiment, wall scroll index data size: BIndexSize=ST (8byte)+Offset (32byte)=40byte;

The total size of index segment information: MaxBIndexSize=(MaxBlockNum+4) * BIndexSize;

End-of-file position: tailOffset=maxsize+MaxBIndexSize;

File overall length: TotalFileSize=tailOffset+ST (8byte)+ET (8byte)+offset (32byte).

In the present embodiment, described step S20 is specially:

Described data source identification is generated the shape data of four bytes by hash algorithm, by last byte of described four bytes, successively forward, set up and described four bytes root directory, one deck catalogue, two layers of catalogue and three layers of catalogue one to one.

In the present embodiment, the start time .data of filename=this file record of shown data file.

Embodiment two

As shown in Figure 2, in the present embodiment, described step S30 comprises:

S31, search in catalogue all with .data ending data file, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time time.data; In end-of-file tailOffset position log file essential information, { wherein, the start time is data time for startTime, 0,0}, and the end time is that 0 expression does not terminate, current data section writing position 0, and is recorded in the data structure of the file handle in internal memory.

If S32 finds with the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount offset; File reading tail recorded information { startTime, 0, offset}, wherein, the start time is data time, and the end time is that 0 expression does not terminate, current data section writing position offset, and file is write side-play amount be set to offset, and be recorded in the data structure of the file handle in internal memory.

S33, be written in described data file by described time series data, the write reference position of described time series data is said write side-play amount offset;

S34, write index information, be written to described first index information in data segment index corresponding to end-of-file, be written in the secondary index of end-of-file by described second index information.

In the present embodiment, also comprise after described step S34:

If the described data file of step S35 reaches data segment maximum length, close described data file, set up new data file and write.

Embodiment three

As shown in Figure 3, in the present embodiment, described step S40 comprises:

S41, reception inquiry request, described inquiry request comprises data source identification and data time;

S42, the data source identification of described inquiry request is generated several bytes by hash algorithm, each byte is converted into the character string of 16 systems, and searches corresponding catalogue;

S43, the filename under described catalogue to be sorted according to inverted order, find out first filename less than the data time of described inquiry request;

S44, open data file corresponding to described filename, and the index information at file reading end;

S55, read the data message of the time series data of inquiry request according to described index information.

Embodiment four

As shown in Figure 4, a kind of device accessing magnanimity time series data, comprising:

Receiver module 10, for time of reception sequence data, described time series data comprises data source identification, data time and data message;

Modular converter 20, for described data source identification is generated several bytes by hash algorithm, sets up and several bytes described some layers of catalogue one to one;

Memory module 30, for being stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

Read module 40, for receiving inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

Embodiment five

As shown in Figure 5, in the present embodiment, described memory module 30 comprises:

Data file sets up unit 31, all with the data file of .data ending for searching in catalogue, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time .data;

Side-play amount writing unit 32, for finding with after the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount offset;

Data message writing unit 33, for being written in described data file by described time series data, the write reference position of described time series data is said write side-play amount offset;

Index information writing unit 34, for writing index information, being written to described first index information in data segment index corresponding to end-of-file, being written in the secondary index of end-of-file by described second index information.

Embodiment six

As shown in Figure 6, in the present embodiment, described read module 40 comprises:

Inquiry request receiving element 41, for receiving inquiry request, described inquiry request comprises data source identification and data time;

Inquiry request converting unit 42, for the data source identification of described inquiry request is generated several bytes by hash algorithm, is converted into the character string of 16 systems, and searches corresponding catalogue by each byte;

Sequencing unit 43, for sorting according to inverted order to the filename under described catalogue, finds out first filename less than the data time of described inquiry request;

Index information reading unit 44, for opening data file corresponding to described filename, and the index information at file reading end;

Data message reading unit 45, for reading the data message of the time series data of inquiry request according to described index information.

The foregoing is only the specific embodiment of the present invention, these describe just in order to explain principle of the present invention, and can not be interpreted as limiting the scope of the invention by any way.Based on explanation herein, those skilled in the art does not need to pay performing creative labour can associate other specific implementation method of the present invention, and these modes all will fall within protection scope of the present invention.

Claims (10)

1. access a method for magnanimity time series data, it is characterized in that, comprising:

S10, time of reception sequence data, described time series data comprises data source identification, data time and data message;

S20, described data source identification is generated several bytes by hash algorithm, set up and several bytes described some layers of catalogue one to one;

S30, be stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

S40, reception inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

2. a kind of method accessing magnanimity time series data according to claim 1, is characterized in that, described data segment is used for recording described data message, and described index segment is used for essential information and the index information of recording data files.

3. a kind of method accessing magnanimity time series data according to claim 2, it is characterized in that, described essential information comprises data file start time, end of data file time and data source identification, described index information comprises the first index information and the second index information, described first index information records start time corresponding to each data segment and data file deviation post in order, and described second index information comprises wall scroll index data size, the total size of index segment information, end-of-file position and file overall length.

4. a kind of method accessing magnanimity time series data according to claim 3, it is characterized in that, described step S20 is specially:

Described data source identification is generated the shape data of four bytes by hash algorithm, by last byte of described four bytes, successively forward, set up and described four bytes root directory, one deck catalogue, two layers of catalogue and three layers of catalogue one to one.

5. a kind of method accessing magnanimity time series data according to claim 4, it is characterized in that, described step S30 comprises:

S31, search in catalogue all with the data file of .data ending, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time .data;

If S32 finds with the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount;

S33, be written in described data file by described time series data, the write reference position of described time series data is said write side-play amount;

S34, write index information, be written to described first index information in data segment index corresponding to end-of-file, be written in the secondary index of end-of-file by described second index information.

6. a kind of method accessing magnanimity time series data according to claim 5, is characterized in that, also comprise after described step S34:

If the described data file of step S35 reaches data segment maximum length, close described data file, set up new data file and write.

7. a kind of method accessing magnanimity time series data according to claim 1, it is characterized in that, described step S40 comprises:

S41, reception inquiry request, described inquiry request comprises data source identification and data time;

S42, the data source identification of described inquiry request is generated several bytes by hash algorithm, each byte is converted into the character string of 16 systems, and searches corresponding catalogue;

S43, the filename under described catalogue to be sorted according to inverted order, find out first filename less than the data time of described inquiry request;

S44, open data file corresponding to described filename, and the index information at file reading end;

S55, read the data message of the time series data of inquiry request according to described index information.

8. access a device for magnanimity time series data, it is characterized in that, comprising:

Receiver module, for time of reception sequence data, described time series data comprises data source identification, data time and data message;

Modular converter, for described data source identification is generated several bytes by hash algorithm, sets up and several bytes described some layers of catalogue one to one;

Memory module, for being stored in described catalogue by described time series data with the form of data file, described data file comprises data segment and index segment, and the filename of described data file is generated by described data time;

Read module, for receiving inquiry request, described inquiry request comprises data source identification and data time, reads corresponding time series data according to described inquiry request and index segment from described catalogue.

9. a kind of device accessing magnanimity time series data according to claim 8, it is characterized in that, described memory module comprises:

Data file sets up unit, all with the data file of .data ending for searching in catalogue, and the data file that locating file name is maximum, if do not exist with the data file of .data ending, then newdata file, the file of described newdata file is called: data time .data;

Side-play amount writing unit, for finding with after the data file of .data ending, file reading tail information, described end-of-file packets of information is containing write side-play amount;

Data message writing unit, for being written in described data file by described time series data, the write reference position of described time series data is said write side-play amount;

Index information writing unit, for writing index information, being written to described first index information in data segment index corresponding to end-of-file, being written in the secondary index of end-of-file by described second index information.

10. a kind of device accessing magnanimity time series data according to claim 8, it is characterized in that, described read module comprises:

Inquiry request receiving element, for receiving inquiry request, described inquiry request comprises data source identification and data time;

Inquiry request converting unit, for the data source identification of described inquiry request is generated several bytes by hash algorithm, is converted into the character string of 16 systems, and searches corresponding catalogue by each byte;

Sequencing unit, for sorting according to inverted order to the filename under described catalogue, finds out first filename less than the data time of described inquiry request;

Index information reading unit, for opening data file corresponding to described filename, and the index information at file reading end;

Data message reading unit, for reading the data message of the time series data of inquiry request according to described index information.