技术领域technical field
本发明涉及互联网技术领域，具体涉及一种基于LSM-tree结构的数据处理方法及装置。The invention relates to the technical field of the Internet, in particular to a data processing method and device based on an LSM-tree structure.
背景技术Background technique
LSM-tree(日志结构的合并树，Log-Structured Merge-Tree)通过批量存储技术规避了磁盘随机写入问题，大幅度地提高了写性能。具体地，可利用LSM-tree来存储数据条目和元信息条目。在LSM-tree结构中，通过数据文件来记录数据条目和元信息条目，并支持数据删除和数据读取等操作。当根据数据操作请求需要对待操作数据条目进行操作时，首先需要在数据文件中查找该待操作数据条目对应的元信息条目，然后根据该待操作数据条目对应的元信息条目进行相应操作。因此，现有技术中的这种数据处理方式存在着处理效率低下的问题。LSM-tree (Log-Structured Merge-Tree) avoids the problem of disk random writing through batch storage technology, and greatly improves the writing performance. Specifically, an LSM-tree can be used to store data entries and meta information entries. In the LSM-tree structure, data entries and metadata entries are recorded through data files, and operations such as data deletion and data reading are supported. When the data item to be operated needs to be operated according to the data operation request, the metadata item corresponding to the data item to be operated needs to be searched first in the data file, and then the corresponding operation is performed according to the metadata item corresponding to the data item to be operated. Therefore, the data processing method in the prior art has the problem of low processing efficiency.
发明内容Contents of the invention
鉴于上述问题，提出了本发明以便提供一种克服上述问题或者至少部分地解决上述问题的基于LSM-tree结构的数据处理方法及装置。In view of the above problems, the present invention is proposed to provide a data processing method and device based on an LSM-tree structure that overcomes the above problems or at least partially solves the above problems.
根据本发明的一个方面，提供了一种基于LSM-tree结构的数据处理方法，该方法包括：According to one aspect of the present invention, a kind of data processing method based on LSM-tree structure is provided, and the method comprises:
接收数据操作请求，确定与数据操作请求所对应的待操作数据条目；receiving a data operation request, and determining a data item to be operated corresponding to the data operation request;
在缓存中查找待操作数据条目对应的元信息条目；Find the metadata entry corresponding to the data entry to be operated in the cache;
若在缓存中未查找到待操作数据条目对应的元信息条目，则在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存；If the metadata entry corresponding to the data entry to be operated is not found in the cache, the metadata entry corresponding to the data entry to be operated is searched in the data file, and the metadata entry corresponding to the data entry to be operated is found in the data file The entry is written to the cache;
根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。Respond to the data operation request according to the metadata item corresponding to the data item to be operated.
根据本发明的另一方面，提供了一种基于LSM-tree结构的数据处理装置，该装置包括：According to another aspect of the present invention, a kind of data processing device based on LSM-tree structure is provided, and this device comprises:
确定模块，适于接收数据操作请求，确定与数据操作请求所对应的待操作数据条目；A determining module, adapted to receive a data operation request, and determine the data item to be operated corresponding to the data operation request;
第一查找模块，适于在缓存中查找待操作数据条目对应的元信息条目；The first search module is adapted to search the cache for the metadata entry corresponding to the data entry to be operated;
第二查找模块，适于若第一查找模块在缓存中未查找到待操作数据条目对应的元信息条目，则在数据文件中查找待操作数据条目对应的元信息条目；The second search module is adapted to search the data file for the metadata entry corresponding to the data entry to be operated if the first search module does not find the metadata entry corresponding to the data entry to be operated in the cache;
写入模块，适于将第二查找模块在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存；The write module is adapted to write the metadata entry corresponding to the data entry to be operated found by the second search module in the data file into the cache;
响应模块，适于根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。The response module is adapted to respond to the data operation request according to the meta information item corresponding to the data item to be operated.
根据本发明提供的技术方案，接收数据操作请求，确定与数据操作请求所对应的待操作数据条目，在缓存中查找待操作数据条目对应的元信息条目，如果在缓存中未查找到待操作数据条目对应的元信息条目，那么在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，然后根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。本发明提供的技术方案优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目，从而加快了查找元信息条目的速度，有效地提高了数据处理效率，优化了数据处理方式。According to the technical solution provided by the present invention, the data operation request is received, the data item to be operated corresponding to the data operation request is determined, and the meta information item corresponding to the data item to be operated is searched in the cache, if no data to be operated is found in the cache entry corresponding to the entry, then find the metadata entry corresponding to the data entry to be operated in the data file, and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, and then according to the data to be operated The meta information entry corresponding to the entry responds to the data operation request. In the technical solution provided by the present invention, the meta-information entry corresponding to the data entry to be operated is first searched from the cache, and if the meta-information entry corresponding to the data entry to be operated is not found in the cache, it is searched in the data file again, and Write the meta information entries corresponding to the data items to be operated found in the data file into the cache, so that the subsequent data items to be operated can find the corresponding meta information items in the cache, thus speeding up the search for meta information items and effectively improving The data processing efficiency is improved, and the data processing method is optimized.
上述说明仅是本发明技术方案的概述，为了能够更清楚了解本发明的技术手段，而可依照说明书的内容予以实施，并且为了让本发明的上述和其它目的、特征和优点能够更明显易懂，以下特举本发明的具体实施方式。The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.
附图说明Description of drawings
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本发明的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:
图1示出了根据本发明一个实施例的基于LSM-tree结构的数据处理方法的流程示意图；Fig. 1 shows a schematic flow chart of a data processing method based on an LSM-tree structure according to an embodiment of the present invention;
图2示出了根据本发明另一个实施例的基于LSM-tree结构的数据处理方法的流程示意图；FIG. 2 shows a schematic flow diagram of a data processing method based on an LSM-tree structure according to another embodiment of the present invention;
图3示出了根据本发明又一个实施例的基于LSM-tree结构的数据处理方法的流程示意图；FIG. 3 shows a schematic flow diagram of a data processing method based on an LSM-tree structure according to yet another embodiment of the present invention;
图4示出了根据本发明再一个实施例的基于LSM-tree结构的数据处理方法的流程示意图；FIG. 4 shows a schematic flow diagram of a data processing method based on an LSM-tree structure according to yet another embodiment of the present invention;
图5示出了根据本发明一个实施例的基于LSM-tree结构的数据处理装置的结构框图。Fig. 5 shows a structural block diagram of a data processing device based on an LSM-tree structure according to an embodiment of the present invention.
具体实施方式detailed description
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
LSM-tree结构可用于存储数据条目和元信息条目。在LSM-tree结构中，通过数据文件来记录数据条目和元信息条目。现有技术中，当根据数据操作请求需要对待操作数据条目进行操作时，首先需要在数据文件中查找该待操作数据条目对应的元信息条目，然后根据该待操作数据条目对应的元信息条目进行相应操作。因此，现有技术中的这种数据处理方式存在着处理效率低下的问题。在本发明提供的技术方案中，优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的该待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目，从而加快了查找元信息条目的速度，有效地提高了数据处理效率，优化了数据处理方式。The LSM-tree structure can be used to store data entries and meta information entries. In the LSM-tree structure, data entries and meta information entries are recorded through data files. In the prior art, when the data item to be operated needs to be operated according to the data operation request, it is first necessary to search for the metadata item corresponding to the data item to be operated in the data file, and then proceed according to the metadata item corresponding to the data item to be operated Operate accordingly. Therefore, the data processing method in the prior art has the problem of low processing efficiency. In the technical solution provided by the present invention, the meta information entry corresponding to the data entry to be operated is firstly searched from the cache, and if the meta information entry corresponding to the data entry to be operated is not found in the cache, the data file is searched again , and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, so that the subsequent data entry to be operated can search for the corresponding metadata entry in the cache, thereby speeding up the search for the metadata entry. Effectively improve the data processing efficiency and optimize the data processing method.
图1示出了根据本发明一个实施例的基于LSM-tree结构的数据处理方法的流程示意图，如图1所示，该方法包括如下步骤：Fig. 1 shows a schematic flow chart of a data processing method based on an LSM-tree structure according to an embodiment of the present invention. As shown in Fig. 1, the method includes the following steps:
步骤S100，接收数据操作请求，确定与数据操作请求所对应的待操作数据条目。In step S100, a data operation request is received, and a data item to be operated corresponding to the data operation request is determined.
其中，元信息条目和数据条目均是以数据键值对形式进行存储的。具体地，可将hash类型、zset类型、set类型或者list类型等类型的数据根据预设的封装规则封装成数据键值对形式的元信息条目和数据条目。本领域技术人员可根据实际需要设置具体的封装规则，此处不做限定。Wherein, both the meta-information entry and the data entry are stored in the form of data key-value pairs. Specifically, data of hash type, zset type, set type, or list type can be encapsulated into meta information items and data items in the form of data key-value pairs according to preset encapsulation rules. Those skilled in the art may set specific encapsulation rules according to actual needs, which are not limited here.
以hash类型的数据为例，假设该数据具有对应的hashkey，该数据还包括3个成员，这3个成员分别为m1、m2和m3，那么根据该数据可封装得到数据键值对形式的1条元信息条目和3条数据条目，为了便于管理，数据条目的数据键和与该数据条目对应的元信息条目的数据键的至少一部分相同。具体地，该元信息条目的数据键为hashkey，该数据键对应的数据值包括成员数量信息、版本信息或者过期时间信息等信息；第1条数据条目的数据键为hashkeym1，该数据键对应的数据值包括m1对应的数据信息、版本信息等信息；第2条数据条目的数据键为hashkeym2，该数据键对应的数据值包括m2对应的数据信息、版本信息等信息；第3条数据条目的数据键为hashkeym3，该数据键对应的数据值包括m3对应的数据信息、版本信息等信息。Take the hash type of data as an example, assuming that the data has a corresponding hashkey, and the data also includes 3 members, these 3 members are m1, m2 and m3 respectively, then according to the data can be encapsulated to get 1 in the form of data key-value pairs For the convenience of management, the data key of the data item is at least partly the same as the data key of the meta information item corresponding to the data item. Specifically, the data key of the metadata entry is hashkey, and the data value corresponding to the data key includes information such as member number information, version information, or expiration time information; the data key of the first data entry is hashkeym1, and the data value corresponding to the data key The data value includes the data information, version information and other information corresponding to m1; the data key of the second data entry is hashkeym2, and the data value corresponding to the data key includes information such as the data information and version information corresponding to m2; the third data entry The data key is hashkeym3, and the data value corresponding to the data key includes information such as data information and version information corresponding to m3.
在步骤S100中，接收数据操作请求，并根据数据操作请求，确定与数据操作请求所对应的待操作数据条目。其中，数据操作请求可以为数据删除操作请求、数据读取操作请求或者数据写入操作请求等。In step S100, a data operation request is received, and a data item to be operated corresponding to the data operation request is determined according to the data operation request. Wherein, the data operation request may be a data deletion operation request, a data read operation request, or a data write operation request.
步骤S101，在缓存中查找待操作数据条目对应的元信息条目。Step S101, searching the cache for the metadata entry corresponding to the data entry to be operated.
具体地，待操作数据条目的数据键和与待操作数据条目对应的元信息条目的数据键的至少一部分相同，那么在步骤S101中，可根据待操作数据条目的数据键，在缓存中查找该待操作数据条目对应的元信息条目。Specifically, the data key of the data item to be operated is at least partly the same as the data key of the metadata item corresponding to the data item to be operated, then in step S101, the data key of the data item to be operated can be searched for in the cache The metadata item corresponding to the data item to be operated.
步骤S102，判断在缓存中是否查找到待操作数据条目对应的元信息条目；若是，则执行步骤S104；若否，则执行步骤S103。Step S102, judging whether the meta information item corresponding to the data item to be operated is found in the cache; if yes, execute step S104; if not, execute step S103.
如果判断得到在缓存中查找到待操作数据条目对应的元信息条目，则执行步骤S104；如果判断得到在缓存中未查找到待操作数据条目对应的元信息条目，则执行步骤S103。If it is determined that the meta information item corresponding to the data item to be operated is found in the cache, execute step S104; if it is determined that no meta information item corresponding to the data item to be operated is found in the cache, execute step S103.
步骤S103，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存。Step S103, searching the data file for a meta information item corresponding to the data item to be operated, and writing the meta information item found in the data file corresponding to the data item to be operated into the cache.
其中，数据文件中记录有若干个以数据键值对形式存储在LSM-tree结构中的数据条目和元信息条目。在缓存中未查找到待操作数据条目对应的元信息条目的情况下，在步骤S103中，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目查找对应的元信息条目时直接从缓存中进行查找，从而加快了查找元信息条目的速度，进而有助于提高数据处理效率。Wherein, the data file records several data entries and meta-information entries stored in the LSM-tree structure in the form of data key-value pairs. If the meta information entry corresponding to the data item to be operated is not found in the cache, in step S103, the meta information item corresponding to the data item to be operated is searched in the data file, and the data file to be operated is found in the data file The metadata entry corresponding to the data entry is written into the cache, so that the subsequent data entry to be operated can be directly searched from the cache when searching for the corresponding metadata entry, thereby speeding up the search for the metadata entry, which in turn helps to improve data processing efficiency.
步骤S104，根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。Step S104, responding to the data operation request according to the metadata item corresponding to the data item to be operated.
在从缓存中或者数据文件中查找到待操作数据条目对应的元信息条目之后，在步骤S104中，根据待操作数据条目对应的元信息条目，对数据操作请求作出响应，执行数据读取、数据写入或者数据删除等操作。After the meta information entry corresponding to the data entry to be operated is found from the cache or the data file, in step S104, according to the meta information entry corresponding to the data entry to be operated, respond to the data operation request, execute data reading, data Operations such as writing or data deletion.
根据本发明实施例提供的基于LSM-tree结构的数据处理方法，接收数据操作请求，确定与数据操作请求所对应的待操作数据条目，在缓存中查找待操作数据条目对应的元信息条目，如果在缓存中未查找到待操作数据条目对应的元信息条目，那么在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，然后根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。本发明提供的技术方案优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目，从而加快了查找元信息条目的速度，有效地提高了数据处理效率，优化了数据处理方式。According to the data processing method based on the LSM-tree structure provided by the embodiment of the present invention, the data operation request is received, the data item to be operated corresponding to the data operation request is determined, and the meta information item corresponding to the data item to be operated is searched in the cache, if If the metadata entry corresponding to the data entry to be operated is not found in the cache, then the metadata entry corresponding to the data entry to be operated is searched in the data file, and the metadata entry corresponding to the data entry to be operated is found in the data file Write to the cache, and then respond to the data operation request according to the metadata entry corresponding to the data entry to be operated. In the technical solution provided by the present invention, the meta-information entry corresponding to the data entry to be operated is first searched from the cache, and if the meta-information entry corresponding to the data entry to be operated is not found in the cache, it is searched in the data file again, and Write the meta information entries corresponding to the data items to be operated found in the data file into the cache, so that the subsequent data items to be operated can find the corresponding meta information items in the cache, thus speeding up the search for meta information items and effectively improving The data processing efficiency is improved, and the data processing method is optimized.
图2示出了根据本发明另一个实施例的基于LSM-tree结构的数据处理方法的流程示意图，如图2所示，该方法包括如下步骤：Fig. 2 shows a schematic flow chart of a data processing method based on an LSM-tree structure according to another embodiment of the present invention. As shown in Fig. 2, the method includes the following steps:
步骤S200，接收数据删除操作请求，确定与数据删除操作请求所对应的待操作数据条目。In step S200, a data deletion operation request is received, and a data item to be operated corresponding to the data deletion operation request is determined.
其中，元信息条目和数据条目均是以数据键值对形式进行存储的。具体地，数据删除操作请求中可包括待操作数据条目的数据键的至少一部分，那么在步骤S200中，根据接收到的数据删除操作请求中所包括的待操作数据条目的数据键的至少一部分，确定与数据删除操作请求所对应的待操作数据条目。Wherein, both the meta-information entry and the data entry are stored in the form of data key-value pairs. Specifically, the data deletion operation request may include at least a part of the data key of the data item to be operated, then in step S200, according to at least a part of the data key of the data item to be operated included in the received data deletion operation request, Determine the data item to be operated corresponding to the data deletion operation request.
步骤S201，在缓存中查找待操作数据条目对应的元信息条目。Step S201, searching the cache for the metadata entry corresponding to the data entry to be operated.
其中，待操作数据条目的数据键和与待操作数据条目对应的元信息条目的数据键的至少一部分相同。Wherein, the data key of the data item to be operated is at least partly the same as the data key of the metadata item corresponding to the data item to be operated.
步骤S202，判断在缓存中是否查找到待操作数据条目对应的元信息条目；若是，则执行步骤S204；若否，则执行步骤S203。Step S202, judging whether the meta information entry corresponding to the data entry to be operated is found in the cache; if yes, execute step S204; if not, execute step S203.
如果判断得到在缓存中查找到待操作数据条目对应的元信息条目，则执行步骤S204；如果判断得到在缓存中未查找到待操作数据条目对应的元信息条目，则执行步骤S203。If it is determined that the meta information item corresponding to the data item to be operated is found in the cache, execute step S204; if it is determined that no meta information item corresponding to the data item to be operated is found in the cache, execute step S203.
步骤S203，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存。Step S203, searching the data file for a meta information item corresponding to the data item to be operated, and writing the meta information item found in the data file corresponding to the data item to be operated into the cache.
在缓存中未查找到待操作数据条目对应的元信息条目的情况下，在步骤S203中，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目查找对应的元信息条目时直接从缓存中进行查找，从而加快了查找元信息条目的速度，进而有助于提高数据处理效率。In the case that the meta information entry corresponding to the data item to be operated is not found in the cache, in step S203, the meta information item corresponding to the data item to be operated is searched in the data file, and the data file to be operated is found in the data file The metadata entry corresponding to the data entry is written into the cache, so that the subsequent data entry to be operated can be directly searched from the cache when searching for the corresponding metadata entry, thereby speeding up the search for the metadata entry, which in turn helps to improve data processing efficiency.
步骤S204，修改待操作数据条目对应的元信息条目中的版本信息。Step S204, modify the version information in the metadata item corresponding to the data item to be operated.
其中，待操作数据条目中和元信息条目中都包括版本信息。假设待操作数据条目中的版本信息和待操作数据条目对应的元信息条目中的版本信息均为版本1，根据数据删除操作请求可知，需要删除该待操作数据条目，那么在步骤S204中，可直接将该待操作数据条目对应的元信息条目中的版本信息修改为版本2，通过修改元信息条目中的版本信息的方式来标识该元信息条目对应的待操作数据条目为已被删除的数据条目。也就是说，如果某条数据条目中的版本信息不与该操作数据条目对应的元信息条目中的版本信息相符，则说明该数据条目为已被删除的数据条目。Wherein, both the data-to-be-operated entry and the meta-information entry include version information. Assuming that the version information in the data entry to be operated and the version information in the metadata entry corresponding to the data entry to be operated are both version 1, it can be known from the data deletion operation request that the data entry to be operated needs to be deleted, then in step S204, you can Directly modify the version information in the metadata entry corresponding to the data entry to be operated to version 2, and identify the data entry to be operated corresponding to the metadata entry as deleted data by modifying the version information in the metadata entry entry. That is to say, if the version information in a data entry does not match the version information in the metadata entry corresponding to the operation data entry, it means that the data entry is a deleted data entry.
根据本发明实施例提供的基于LSM-tree结构的数据处理方法，优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目；另外，根据该技术方案，可方便地通过修改待操作数据条目对应的元信息条目中的版本信息的方式来标识待操作数据条目为已被删除的数据条目，从而进一步提高了数据处理效率，优化了数据处理方式。According to the data processing method based on the LSM-tree structure provided by the embodiment of the present invention, the meta information entry corresponding to the data entry to be operated is first searched from the cache, and the meta information entry corresponding to the data entry to be operated is not found in the cache Next, search in the data file again, and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, so that the subsequent data entry to be operated can search for the corresponding metadata entry in the cache; in addition, According to this technical solution, it is convenient to identify the data entry to be operated as a deleted data entry by modifying the version information in the metadata entry corresponding to the data entry to be operated, thereby further improving the data processing efficiency and optimizing the data processing efficiency. processing method.
图3示出了根据本发明又一个实施例的基于LSM-tree结构的数据处理方法的流程示意图，如图3所示，该方法包括如下步骤：Fig. 3 shows a schematic flow chart of a data processing method based on an LSM-tree structure according to yet another embodiment of the present invention. As shown in Fig. 3, the method includes the following steps:
步骤S300，接收数据读取操作请求，确定与数据读取操作请求所对应的待操作数据条目。In step S300, a data read operation request is received, and a data item to be operated corresponding to the data read operation request is determined.
具体地，数据读取操作请求中可包括待操作数据条目的数据键的至少一部分，那么在步骤S300中，根据接收到的数据读取操作请求中所包括的待操作数据条目的数据键的至少一部分，确定与数据读取操作请求所对应的待操作数据条目。Specifically, the data read operation request may include at least a part of the data key of the data item to be operated, then in step S300, according to at least part of the data key of the data item to be operated included in the received data read operation request In one part, the data item to be operated corresponding to the data read operation request is determined.
步骤S301，在缓存中查找待操作数据条目对应的元信息条目。Step S301, searching the cache for the metadata entry corresponding to the data entry to be operated.
具体地，待操作数据条目的数据键和与待操作数据条目对应的元信息条目的数据键的至少一部分相同，那么在步骤S301中，可根据待操作数据条目的数据键，在缓存中查找该待操作数据条目对应的元信息条目。Specifically, the data key of the data item to be operated is at least partly the same as the data key of the metadata item corresponding to the data item to be operated, then in step S301, the data key of the data item to be operated can be found in the cache The metadata item corresponding to the data item to be operated.
步骤S302，判断在缓存中是否查找到待操作数据条目对应的元信息条目；若是，则执行步骤S304；若否，则执行步骤S303。Step S302, judging whether the metadata item corresponding to the data item to be operated is found in the cache; if yes, execute step S304; if not, execute step S303.
如果判断得到在缓存中查找到待操作数据条目对应的元信息条目，则执行步骤S304；如果判断得到在缓存中未查找到待操作数据条目对应的元信息条目，则执行步骤S303。If it is determined that the meta information item corresponding to the data item to be operated is found in the cache, execute step S304; if it is determined that no meta information item corresponding to the data item to be operated is found in the cache, execute step S303.
步骤S303，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存。Step S303, searching the data file for a meta information item corresponding to the data item to be operated, and writing the meta information item found in the data file corresponding to the data item to be operated into the cache.
在缓存中未查找到待操作数据条目对应的元信息条目的情况下，在步骤S303中，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目查找对应的元信息条目时直接从缓存中进行查找，从而加快了查找元信息条目的速度，进而有助于提高数据处理效率。In the case that the metadata entry corresponding to the data entry to be operated is not found in the cache, in step S303, the metadata entry corresponding to the data entry to be operated is searched in the data file, and the data file to be operated is searched for The metadata entry corresponding to the data entry is written into the cache, so that the subsequent data entry to be operated can be directly searched from the cache when searching for the corresponding metadata entry, thereby speeding up the search for the metadata entry, which in turn helps to improve data processing efficiency.
步骤S304，判断待操作数据条目中的版本信息是否与待操作数据条目对应的元信息条目中的版本信息相符；若是，则执行步骤S305；若否，则执行步骤S306。Step S304, judging whether the version information in the to-be-operated data entry matches the version information in the meta-information entry corresponding to the to-be-operated data entry; if yes, execute step S305; if not, execute step S306.
其中，待操作数据条目中和元信息条目中都包括版本信息，根据版本信息可判断待操作数据条目是否已被删除。如果判断得到待操作数据条目中的版本信息与待操作数据条目对应的元信息条目中的版本信息相符，说明该待操作数据条目不是已被删除的数据条目，即该待操作数据条目为有效数据条目，则执行步骤S305；如果判断得到待操作数据条目中的版本信息不与待操作数据条目对应的元信息条目中的版本信息相符，说明该待操作数据条目是已被删除的数据条目，即该待操作数据条目为无效数据条目，则执行步骤S306。Wherein, both the data entry to be operated and the metadata entry include version information, and it can be determined whether the data entry to be operated has been deleted according to the version information. If it is judged that the version information in the data item to be operated is consistent with the version information in the metadata item corresponding to the data item to be operated, it means that the data item to be operated is not a deleted data item, that is, the data item to be operated is valid data entry, then execute step S305; if it is judged that the version information in the data entry to be operated does not match the version information in the metadata entry corresponding to the data entry to be operated, it means that the data entry to be operated is a data entry that has been deleted, that is If the data entry to be operated is an invalid data entry, step S306 is performed.
步骤S305，读取得到待操作数据条目。Step S305, read to obtain the data item to be operated.
在待操作数据条目中的版本信息与待操作数据条目对应的元信息条目中的版本信息相符的情况下，在步骤S305中，读取得到待操作数据条目。If the version information in the data item to be operated matches the version information in the metadata item corresponding to the data item to be operated, in step S305, the data item to be operated is read.
步骤S306，展现数据读取失败的提示信息。Step S306, displaying the prompt information of data reading failure.
在待操作数据条目中的版本信息不与待操作数据条目对应的元信息条目中的版本信息相符的情况下，在步骤S306中，展现数据读取失败的提示信息。If the version information in the to-be-operated data item does not match the version information in the meta-information item corresponding to the to-be-operated data item, in step S306, a prompt message of data reading failure is displayed.
可选地，在本实施例的一个可能的实现方式中，如果元信息条目中包括过期时间信息，那么可根据待操作数据条目对应的元信息条目中的过期时间信息，判断待操作数据条目是否为过期数据条目。如果判断得到待操作数据条目为过期数据条目，说明该待操作数据条目为无效数据条目，则展现数据读取失败的提示信息；如果判断得到数据条目不为过期数据条目，说明该待操作数据条目为有效数据条目，则读取得到待操作数据条目。Optionally, in a possible implementation of this embodiment, if the metadata entry includes expiration time information, then it may be determined whether the data entry to be operated is for expired data entries. If it is judged that the data entry to be operated is an expired data entry, it means that the data entry to be operated is an invalid data entry, and the prompt message of data reading failure will be displayed; if it is judged that the data entry is not an expired data entry, it means that the data entry to be operated is invalid is a valid data entry, then the data entry to be operated is obtained by reading.
根据本发明实施例提供的基于LSM-tree结构的数据处理方法，优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目；另外，根据版本信息或过期时间信息可方便、快速地判断待操作数据条目是否为已被删除的或过期的数据条目，从而进一步提高了数据处理效率，优化了数据处理方式。According to the data processing method based on the LSM-tree structure provided by the embodiment of the present invention, the meta information entry corresponding to the data entry to be operated is first searched from the cache, and the meta information entry corresponding to the data entry to be operated is not found in the cache Next, search in the data file again, and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, so that the subsequent data entry to be operated can search for the corresponding metadata entry in the cache; in addition, According to the version information or the expiration time information, it is convenient and fast to judge whether the data entry to be operated is a deleted or expired data entry, thereby further improving the data processing efficiency and optimizing the data processing method.
图4示出了根据本发明再一个实施例的基于LSM-tree结构的数据处理方法的流程示意图，如图4所示，该方法包括如下步骤：Fig. 4 shows a schematic flow chart of a data processing method based on an LSM-tree structure according to yet another embodiment of the present invention. As shown in Fig. 4, the method includes the following steps:
步骤S400，接收数据写入操作请求，确定与数据写入操作请求所对应的待操作数据条目。Step S400, receiving a data write operation request, and determining a data item to be operated corresponding to the data write operation request.
具体地，数据写入操作请求中可包括待操作数据条目的数据键，那么在步骤S400中，根据接收到的数据写入操作请求中所包括的待操作数据条目的数据键，确定与数据写入操作请求所对应的待操作数据条目。Specifically, the data writing operation request may include the data key of the data entry to be operated, then in step S400, according to the data key of the data entry to be operated included in the received data writing operation request, determine the Enter the data entry to be operated corresponding to the operation request.
步骤S401，在缓存中查找待操作数据条目对应的元信息条目。Step S401, searching the cache for the metadata entry corresponding to the data entry to be operated.
其中，待操作数据条目的数据键和与待操作数据条目对应的元信息条目的数据键的至少一部分相同。Wherein, the data key of the data item to be operated is at least partly the same as the data key of the metadata item corresponding to the data item to be operated.
步骤S402，判断在缓存中是否查找到待操作数据条目对应的元信息条目；若是，则执行步骤S404；若否，则执行步骤S403。Step S402, judging whether the metadata entry corresponding to the data entry to be operated is found in the cache; if yes, execute step S404; if not, execute step S403.
如果判断得到在缓存中查找到待操作数据条目对应的元信息条目，则执行步骤S404；如果判断得到在缓存中未查找到待操作数据条目对应的元信息条目，则执行步骤S403。If it is judged that the meta information item corresponding to the data item to be operated is found in the cache, execute step S404; if it is judged that no meta information item corresponding to the data item to be operated is found in the cache, execute step S403.
步骤S403，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存。Step S403, searching the data file for a meta information item corresponding to the data item to be operated, and writing the meta information item found in the data file corresponding to the data item to be operated into the cache.
在缓存中未查找到待操作数据条目对应的元信息条目的情况下，在步骤S403中，在数据文件中查找待操作数据条目对应的元信息条目，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目查找对应的元信息条目时直接从缓存中进行查找，从而加快了查找元信息条目的速度，进而有助于提高数据处理效率。If the meta information entry corresponding to the data item to be operated is not found in the cache, in step S403, the meta information item corresponding to the data item to be operated is searched in the data file, and the data file to be operated is found in the data file The metadata entry corresponding to the data entry is written into the cache, so that the subsequent data entry to be operated can be directly searched from the cache when searching for the corresponding metadata entry, thereby speeding up the search for the metadata entry, which in turn helps to improve data processing efficiency.
在从缓存中或者数据文件中查找到待操作数据条目对应的元信息条目之后，根据待操作数据条目对应的元信息条目，对数据写入操作请求进行响应。具体地，可通过步骤S404和步骤S405进行实现。After the meta information item corresponding to the data item to be operated is found from the cache or the data file, the data write operation request is responded to according to the meta information item corresponding to the data item to be operated. Specifically, it can be implemented through step S404 and step S405.
步骤S404，根据数据写入操作请求中的待写入数据信息，修改待操作数据条目的数据键对应的数据值。Step S404, according to the data information to be written in the data writing operation request, modify the data value corresponding to the data key of the data entry to be operated.
步骤S405，根据待操作数据条目对应的元信息条目中的版本信息，修改待操作数据条目中的版本信息。Step S405, modify the version information in the data item to be operated according to the version information in the metadata item corresponding to the data item to be operated.
假设待操作数据条目中的版本信息为版本1，待操作数据条目对应的元信息条目中的版本信息为版本2，说明该待操作数据条目为已被删除的数据条目，另外，数据写入操作请求中包括待写入数据信息，那么在步骤S404中根据待写入数据信息修改待操作数据条目的数据键对应的数据值，在步骤S405中根据待操作数据条目对应的元信息条目中的版本信息，修改待操作数据条目中的版本信息，即将待操作数据条目中的版本信息修改为版本2，使得待操作数据条目中的版本信息与待操作数据条目对应的元信息条目中的版本信息相符，从而通过修改待操作数据条目中的版本信息的方式来标识该待操作数据条目为最新写入的数据条目。Assuming that the version information in the data item to be operated is version 1, and the version information in the metadata item corresponding to the data item to be operated is version 2, it means that the data item to be operated is a data item that has been deleted. In addition, the data write operation If the request includes data information to be written, then in step S404 modify the data value corresponding to the data key of the data entry to be operated according to the data information to be written, and in step S405 according to the version in the metadata entry corresponding to the data entry to be operated Information, modify the version information in the data entry to be operated, that is, modify the version information in the data entry to be operated to version 2, so that the version information in the data entry to be operated is consistent with the version information in the metadata entry corresponding to the data entry to be operated , so as to identify the data entry to be operated as the latest written data entry by modifying the version information in the data entry to be operated.
根据本发明实施例提供的基于LSM-tree结构的数据处理方法，优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目；另外，根据该技术方案，可方便地通过修改待操作数据条目中的版本信息的方式来标识该待操作数据条目为最新写入的数据条目，从而进一步提高了数据处理效率，优化了数据处理方式。According to the data processing method based on the LSM-tree structure provided by the embodiment of the present invention, the meta information entry corresponding to the data entry to be operated is first searched from the cache, and the meta information entry corresponding to the data entry to be operated is not found in the cache Next, search in the data file again, and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, so that the subsequent data entry to be operated can search for the corresponding metadata entry in the cache; in addition, According to the technical solution, the data entry to be operated can be conveniently identified as the latest written data entry by modifying the version information in the data entry to be operated, thereby further improving the data processing efficiency and optimizing the data processing method.
图5示出了根据本发明一个实施例的基于LSM-tree结构的数据处理装置的结构框图，如图5所示，该装置包括：确定模块510、第一查找模块520、第二查找模块530、写入模块540和响应模块550。Fig. 5 shows a structural block diagram of a data processing device based on an LSM-tree structure according to an embodiment of the present invention. As shown in Fig. 5, the device includes: a determination module 510, a first search module 520, and a second search module 530 , a write module 540 and a response module 550 .
确定模块510适于：接收数据操作请求，确定与数据操作请求所对应的待操作数据条目。The determination module 510 is adapted to: receive a data operation request, and determine a data item to be operated corresponding to the data operation request.
其中，数据操作请求可以为数据删除操作请求、数据读取操作请求或者数据写入操作请求等。具体地，数据操作请求中可包括待操作数据条目的数据键的至少一部分，那么确定模块510根据接收到的数据操作请求中所包括的待操作数据条目的数据键的至少一部分，确定与数据操作请求所对应的待操作数据条目。Wherein, the data operation request may be a data deletion operation request, a data read operation request, or a data write operation request. Specifically, the data operation request may include at least a part of the data key of the data entry to be operated, then the determining module 510 determines the data operation related to the data operation request based on at least a part of the data key of the data item to be operated included in the received data operation request. The data item to be operated corresponding to the request.
第一查找模块520适于：在缓存中查找待操作数据条目对应的元信息条目。The first search module 520 is adapted to: search the cache for the metadata entry corresponding to the data entry to be operated.
其中，待操作数据条目的数据键和与待操作数据条目对应的元信息条目的数据键的至少一部分相同。具体地，第一查找模块520根据待操作数据条目的数据键，在缓存中查找该待操作数据条目对应的元信息条目。Wherein, the data key of the data item to be operated is at least partly the same as the data key of the metadata item corresponding to the data item to be operated. Specifically, the first search module 520 searches the cache for a meta information entry corresponding to the data item to be operated according to the data key of the data item to be operated.
第二查找模块530适于：若第一查找模块520在缓存中未查找到待操作数据条目对应的元信息条目，则在数据文件中查找待操作数据条目对应的元信息条目。The second search module 530 is adapted to: if the first search module 520 does not find the metadata entry corresponding to the data entry to be operated in the cache, then search the data file for the metadata entry corresponding to the data entry to be operated.
如果第一查找模块520在缓存中未查找到待操作数据条目对应的元信息条目，那么第二查找模块530在数据文件中查找待操作数据条目对应的元信息条目。If the first search module 520 does not find the metadata entry corresponding to the data item to be operated in the cache, then the second search module 530 searches the data file for the metadata entry corresponding to the data item to be operated.
写入模块540适于：将第二查找模块530在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存。The writing module 540 is adapted to: write the metadata entry corresponding to the data entry to be operated found in the data file by the second searching module 530 into the cache.
写入模块540将第二查找模块530在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目查找对应的元信息条目时直接从缓存中进行查找，从而加快了查找元信息条目的速度，进而有助于提高数据处理效率。The writing module 540 writes the meta information entry corresponding to the data item to be operated found in the data file by the second search module 530 into the cache, so that when the subsequent data item to be operated searches for the corresponding meta information item, it is directly searched from the cache, Thereby, the speed of finding the meta-information entry is accelerated, which in turn helps to improve the efficiency of data processing.
响应模块550适于：根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。The response module 550 is adapted to: respond to the data operation request according to the meta information item corresponding to the data item to be operated.
如果第一查找模块520在缓存中查找到待操作数据条目对应的元信息条目或者第二查找模块530在数据文件中查找到待操作数据条目对应的元信息条目，则响应模块550根据待操作数据条目对应的元信息条目，对数据操作请求作出响应。If the first search module 520 finds the metadata entry corresponding to the data entry to be operated in the cache or the second search module 530 finds the metadata entry corresponding to the data entry to be operated in the data file, then the response module 550 The meta information entry corresponding to the entry responds to the data operation request.
其中，当数据操作请求为数据删除操作请求时，响应模块550进一步适于：修改待操作数据条目对应的元信息条目中的版本信息。Wherein, when the data operation request is a data deletion operation request, the response module 550 is further adapted to: modify the version information in the metadata item corresponding to the data item to be operated.
当数据操作请求为数据读取操作请求时，响应模块550进一步适于：判断待操作数据条目中的版本信息是否与待操作数据条目对应的元信息条目中的版本信息相符；若是，则读取得到待操作数据条目；若否，则展现数据读取失败的提示信息。When the data operation request is a data read operation request, the response module 550 is further adapted to: determine whether the version information in the data entry to be operated is consistent with the version information in the meta information entry corresponding to the data entry to be operated; if so, read Obtain the data entry to be operated; if not, display the prompt message of data read failure.
可选地，在本实施例的一个可能的实现方式中，如果元信息条目中包括过期时间信息，那么当数据操作请求为数据读取操作请求时，响应模块550进一步适于：根据待操作数据条目对应的元信息条目中的过期时间信息，判断待操作数据条目是否为过期数据条目；若是，则展现数据读取失败的提示信息；若否，则读取得到待操作数据条目。Optionally, in a possible implementation of this embodiment, if the metadata entry includes expiration time information, then when the data operation request is a data read operation request, the response module 550 is further adapted to: The expiration time information in the meta information entry corresponding to the entry determines whether the data entry to be operated is an expired data entry; if so, displays a prompt message of data read failure; if not, reads the data entry to be operated.
当数据操作请求为数据写入操作请求时，响应模块550进一步适于：根据待操作数据条目对应的元信息条目中的版本信息，修改待操作数据条目中的版本信息。具体地，当数据写入操作请求中包括待写入数据信息时，响应模块550进一步适于：根据数据写入操作请求中的待写入数据信息，修改待操作数据条目的数据键对应的数据值。When the data operation request is a data write operation request, the response module 550 is further adapted to: modify the version information in the data entry to be operated according to the version information in the meta information entry corresponding to the data entry to be operated. Specifically, when the data writing operation request includes data information to be written, the response module 550 is further adapted to: modify the data corresponding to the data key of the data entry to be operated according to the data information to be written in the data writing operation request value.
根据本发明实施例提供的基于LSM-tree结构的数据处理装置，优先从缓存中查找待操作数据条目对应的元信息条目，在缓存中未查找到待操作该数据条目对应的元信息条目的情况下，再在数据文件中进行查找，并将在数据文件中查找到的待操作数据条目对应的元信息条目写入缓存，以便后续待操作数据条目在缓存中查找对应的元信息条目，从而加快了查找元信息条目的速度，有效地提高了数据处理效率，优化了数据处理方式。According to the data processing device based on the LSM-tree structure provided by the embodiment of the present invention, the meta-information entry corresponding to the data entry to be operated is first searched from the cache, and the meta-information entry corresponding to the data entry to be operated is not found in the cache Next, search in the data file again, and write the metadata entry corresponding to the data entry to be operated found in the data file into the cache, so that the subsequent data entry to be operated can search for the corresponding metadata entry in the cache, thereby speeding up This improves the speed of searching for metadata entries, effectively improves the data processing efficiency, and optimizes the data processing method.
在此提供的算法和显示不与任何特定计算机、虚拟系统或者其它设备固有相关。各种通用系统也可以与基于在此的示教一起使用。根据上面的描述，构造这类系统所要求的结构是显而易见的。此外，本发明也不针对任何特定编程语言。应当明白，可以利用各种编程语言实现在此描述的本发明的内容，并且上面对特定语言所做的描述是为了披露本发明的最佳实施方式。The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other device. Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.
在此处所提供的说明书中，说明了大量具体细节。然而，能够理解，本发明的实施例可以在没有这些具体细节的情况下实践。在一些实例中，并未详细示出公知的方法、结构和技术，以便不模糊对本说明书的理解。In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.
类似地，应当理解，为了精简本公开并帮助理解各个发明方面中的一个或多个，在上面对本发明的示例性实施例的描述中，本发明的各个特征有时被一起分组到单个实施例、图、或者对其的描述中。然而，并不应将该公开的方法解释成反映如下意图：即所要求保护的本发明要求比在每个权利要求中所明确记载的特征更多的特征。更确切地说，如下面的权利要求书所反映的那样，发明方面在于少于前面公开的单个实施例的所有特征。因此，遵循具体实施方式的权利要求书由此明确地并入该具体实施方式，其中每个权利要求本身都作为本发明的单独实施例。Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.
本领域那些技术人员可以理解，可以对实施例中的设备中的模块进行自适应性地改变并且把它们设置在与该实施例不同的一个或多个设备中。可以把实施例中的模块或单元或组件组合成一个模块或单元或组件，以及此外可以把它们分成多个子模块或子单元或子组件。除了这样的特征和/或过程或者单元中的至少一些是相互排斥之外，可以采用任何组合对本说明书(包括伴随的权利要求、摘要和附图)中公开的所有特征以及如此公开的任何方法或者设备的所有过程或单元进行组合。除非另外明确陈述，本说明书(包括伴随的权利要求、摘要和附图)中公开的每个特征可以由提供相同、等同或相似目的的替代特征来代替。Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
此外，本领域的技术人员能够理解，尽管在此所述的一些实施例包括其它实施例中所包括的某些特征而不是其它特征，但是不同实施例的特征的组合意味着处于本发明的范围之内并且形成不同的实施例。例如，在下面的权利要求书中，所要求保护的实施例的任意之一都可以以任意的组合方式来使用。Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
本发明的各个部件实施例可以以硬件实现，或者以在一个或者多个处理器上运行的软件模块实现，或者以它们的组合实现。本领域的技术人员应当理解，可以在实践中使用微处理器或者数字信号处理器(DSP)来实现根据本发明实施例中的一些或者全部部件的一些或者全部功能。本发明还可以实现为用于执行这里所描述的方法的一部分或者全部的设备或者装置程序(例如，计算机程序和计算机程序产品)。这样的实现本发明的程序可以存储在计算机可读介质上，或者可以具有一个或者多个信号的形式。这样的信号可以从因特网网站上下载得到，或者在载体信号上提供，或者以任何其他形式提供。The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the embodiments of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.
应该注意的是上述实施例对本发明进行说明而不是对本发明进行限制，并且本领域技术人员在不脱离所附权利要求的范围的情况下可设计出替换实施例。在权利要求中，不应将位于括号之间的任何参考符号构造成对权利要求的限制。单词“包含”不排除存在未列在权利要求中的元件或步骤。位于元件之前的单词“一”或“一个”不排除存在多个这样的元件。本发明可以借助于包括有若干不同元件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的单元权利要求中，这些装置中的若干个可以是通过同一个硬件项来具体体现。单词第一、第二、以及第三等的使用不表示任何顺序。可将这些单词解释为名称。It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.
本发明公开了：A1、一种基于LSM-tree结构的数据处理方法，包括：The invention discloses: A1, a data processing method based on LSM-tree structure, comprising:
接收数据操作请求，确定与所述数据操作请求所对应的待操作数据条目；receiving a data operation request, and determining a data item to be operated corresponding to the data operation request;
在缓存中查找所述待操作数据条目对应的元信息条目；Finding the metadata entry corresponding to the data entry to be operated in the cache;
若在所述缓存中未查找到所述待操作数据条目对应的元信息条目，则在数据文件中查找所述待操作数据条目对应的元信息条目，并将在数据文件中查找到的所述待操作数据条目对应的元信息条目写入缓存；If the meta information entry corresponding to the data item to be operated is not found in the cache, the data file is searched for the meta information item corresponding to the data item to be operated, and the data file found in the data file is The metadata entry corresponding to the data entry to be operated is written into the cache;
根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应。Respond to the data operation request according to the meta information item corresponding to the data item to be operated.
A2、根据A1所述的方法，其中，所述待操作数据条目的数据键和与所述待操作数据条目对应的元信息条目的数据键的至少一部分相同。A2. The method according to A1, wherein at least a part of the data key of the data item to be operated is the same as the data key of the meta information item corresponding to the data item to be operated.
A3、根据A1或A2所述的方法，所述数据操作请求为数据删除操作请求、数据读取操作请求或者数据写入操作请求。A3. According to the method described in A1 or A2, the data operation request is a data deletion operation request, a data read operation request or a data write operation request.
A4、根据A3所述的方法，当所述数据操作请求为数据删除操作请求时，所述根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应进一步包括：A4. According to the method described in A3, when the data operation request is a data deletion operation request, responding to the data operation request according to the metadata entry corresponding to the data entry to be operated further includes:
修改所述待操作数据条目对应的元信息条目中的版本信息。Modify the version information in the meta information item corresponding to the data item to be operated.
A5、根据A3所述的方法，当所述数据操作请求为数据读取操作请求时，所述根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应进一步包括：A5. According to the method described in A3, when the data operation request is a data read operation request, responding to the data operation request according to the metadata entry corresponding to the data entry to be operated further includes:
判断所述待操作数据条目中的版本信息是否与所述待操作数据条目对应的元信息条目中的版本信息相符；若否，则展现数据读取失败的提示信息。Judging whether the version information in the to-be-operated data item matches the version information in the meta-information item corresponding to the to-be-operated data item; if not, displaying a prompt message that data reading fails.
A6、根据A3所述的方法，当所述数据操作请求为数据读取操作请求时，所述根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应进一步包括：A6. According to the method described in A3, when the data operation request is a data read operation request, responding to the data operation request according to the metadata entry corresponding to the data entry to be operated further includes:
根据所述待操作数据条目对应的元信息条目中的过期时间信息，判断所述待操作数据条目是否为过期数据条目；若是，则展现数据读取失败的提示信息。According to the expiration time information in the meta-information item corresponding to the data item to be operated, it is judged whether the data item to be operated is an expired data item; if so, a prompt message of data reading failure is displayed.
A7、根据A3所述的方法，当所述数据操作请求为数据写入操作请求时，所述根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应进一步包括：A7. According to the method described in A3, when the data operation request is a data write operation request, responding to the data operation request according to the metadata entry corresponding to the data entry to be operated further includes:
根据所述待操作数据条目对应的元信息条目中的版本信息，修改所述待操作数据条目中的版本信息。Modify the version information in the to-be-operated data item according to the version information in the meta-information item corresponding to the to-be-operated data item.
本发明还公开了：B8、一种基于LSM-tree结构的数据处理装置，包括：The present invention also discloses: B8, a data processing device based on an LSM-tree structure, comprising:
确定模块，适于接收数据操作请求，确定与所述数据操作请求所对应的待操作数据条目；A determination module, adapted to receive a data operation request, and determine a data item to be operated corresponding to the data operation request;
第一查找模块，适于在缓存中查找所述待操作数据条目对应的元信息条目；The first search module is adapted to search the cache for the metadata entry corresponding to the data entry to be operated;
第二查找模块，适于若所述第一查找模块在所述缓存中未查找到所述待操作数据条目对应的元信息条目，则在数据文件中查找所述待操作数据条目对应的元信息条目；The second search module is adapted to search the data file for the metadata entry corresponding to the data entry to be operated if the first search module does not find the metadata entry corresponding to the data entry to be operated in the cache. entry;
写入模块，适于将所述第二查找模块在数据文件中查找到的所述待操作数据条目对应的元信息条目写入缓存；A writing module, adapted to write the metadata entry corresponding to the data entry to be operated found in the data file by the second searching module into the cache;
响应模块，适于根据所述待操作数据条目对应的元信息条目，对所述数据操作请求作出响应。The response module is adapted to respond to the data operation request according to the meta information item corresponding to the data item to be operated.
B9、根据B8所述的装置，其中，所述待操作数据条目的数据键和与所述待操作数据条目对应的元信息条目的数据键的至少一部分相同。B9. The device according to B8, wherein at least a part of the data key of the data item to be operated is the same as the data key of the meta information item corresponding to the data item to be operated.
B10、根据B8或B9所述的装置，所述数据操作请求为数据删除操作请求、数据读取操作请求或者数据写入操作请求。B10. The device according to B8 or B9, wherein the data operation request is a data deletion operation request, a data read operation request or a data write operation request.
B11、根据B10所述的装置，当所述数据操作请求为数据删除操作请求时，所述响应模块进一步适于：B11. According to the device described in B10, when the data operation request is a data deletion operation request, the response module is further adapted to:
修改所述待操作数据条目对应的元信息条目中的版本信息。Modify the version information in the meta information item corresponding to the data item to be operated.
B12、根据B10所述的装置，当所述数据操作请求为数据读取操作请求时，所述响应模块进一步适于：B12. According to the device described in B10, when the data operation request is a data read operation request, the response module is further adapted to:
判断所述待操作数据条目中的版本信息是否与所述待操作数据条目对应的元信息条目中的版本信息相符；若否，则展现数据读取失败的提示信息。Judging whether the version information in the to-be-operated data item matches the version information in the meta-information item corresponding to the to-be-operated data item; if not, displaying a prompt message that data reading fails.
B13、根据B10所述的装置，当所述数据操作请求为数据读取操作请求时，所述响应模块进一步适于：B13. According to the device described in B10, when the data operation request is a data read operation request, the response module is further adapted to:
根据所述待操作数据条目对应的元信息条目中的过期时间信息，判断所述待操作数据条目是否为过期数据条目；若是，则展现数据读取失败的提示信息。According to the expiration time information in the meta-information item corresponding to the data item to be operated, it is judged whether the data item to be operated is an expired data item; if so, a prompt message of data reading failure is displayed.
B14、根据B10所述的装置，当所述数据操作请求为数据写入操作请求时，所述响应模块进一步适于：B14. According to the device described in B10, when the data operation request is a data write operation request, the response module is further adapted to:
根据所述待操作数据条目对应的元信息条目中的版本信息，修改所述待操作数据条目中的版本信息。Modify the version information in the to-be-operated data item according to the version information in the meta-information item corresponding to the to-be-operated data item.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611179316.7ACN106777146A (en) | 2016-12-19 | 2016-12-19 | Data processing method and device based on LSM tree structures |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611179316.7ACN106777146A (en) | 2016-12-19 | 2016-12-19 | Data processing method and device based on LSM tree structures |
| Publication Number | Publication Date |
|---|---|
| CN106777146Atrue CN106777146A (en) | 2017-05-31 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611179316.7APendingCN106777146A (en) | 2016-12-19 | 2016-12-19 | Data processing method and device based on LSM tree structures |
| Country | Link |
|---|---|
| CN (1) | CN106777146A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105224546A (en)* | 2014-06-04 | 2016-01-06 | 阿里巴巴集团控股有限公司 | Data storage and query method and apparatus |
| CN105224237A (en)* | 2014-05-26 | 2016-01-06 | 华为技术有限公司 | A kind of date storage method and device |
| CN105787093A (en)* | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105224237A (en)* | 2014-05-26 | 2016-01-06 | 华为技术有限公司 | A kind of date storage method and device |
| CN105224546A (en)* | 2014-06-04 | 2016-01-06 | 阿里巴巴集团控股有限公司 | Data storage and query method and apparatus |
| CN105787093A (en)* | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| Title |
|---|
| 段翰聪等: "MUSE:一种面向云存储系统的高性能元数据存储引擎", 《电子科技大学学报》* |
| Publication | Publication Date | Title |
|---|---|---|
| US10191856B2 (en) | Method of managing web browser cache size using logical relationships and clustering | |
| CN109086388B (en) | Block chain data storage method, device, equipment and medium | |
| CN108459826B (en) | Method and device for processing IO (input/output) request | |
| US10649852B1 (en) | Index metadata for inode based backups | |
| US9710283B2 (en) | System and method for pre-storing small data files into a page-cache and performing reading and writing to the page cache during booting | |
| WO2020211236A1 (en) | Read-write conflict resolution method and apparatus employing b+ tree and storage medium | |
| CN106294603A (en) | File memory method and device | |
| CN103530420B (en) | The dynamic updating method and device of data file | |
| WO2016016734A1 (en) | Managing web browser cache for offline browsing | |
| CN106021335A (en) | A database accessing method and device | |
| US10521611B2 (en) | Method and system for emergency data destruction | |
| WO2017107833A1 (en) | Method and apparatus for switching page in browser label | |
| CN105447167A (en) | Processing method and apparatus for node cache data in distributed system | |
| CN105608143A (en) | Detection method and device of consistency of multi-copy data | |
| CN106599247B (en) | Method and device for merging data files in LSM-tree structure | |
| CN103049393B (en) | Memory headroom management method and device | |
| CN105653635A (en) | Database management method and apparatus | |
| CN106844676A (en) | Data storage method and device | |
| CN106227769B (en) | Date storage method and device | |
| CN103019956B (en) | A kind of to data cached method of operating and device | |
| CN107992763B (en) | Power failure protection method and device for file system | |
| CN112559913A (en) | Data processing method and device, computing equipment and readable storage medium | |
| CN106528876B (en) | The information processing method and distributed information processing system of distributed system | |
| CN109992535B (en) | A storage control method, device and system | |
| WO2018176811A1 (en) | Hard disk write-in method and apparatus |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20170531 | |
| RJ01 | Rejection of invention patent application after publication |