CN115114232A - Method, device and medium for enumerating historical version objects - Google Patents
Method, device and medium for enumerating historical version objectsDownload PDFInfo
- Publication number
- CN115114232A CN115114232ACN202210832787.2ACN202210832787ACN115114232ACN 115114232 ACN115114232 ACN 115114232ACN 202210832787 ACN202210832787 ACN 202210832787ACN 115114232 ACN115114232 ACN 115114232A
- Authority
- CN
- China
- Prior art keywords
- historical version
- version
- key
- name
- enumeration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及对象存储技术领域，特别是涉及一种历史版本对象列举方法、装置及其介质。The present application relates to the technical field of object storage, and in particular, to a method, apparatus and medium for enumerating historical version objects.
背景技术Background technique
随着互联网技术的发展，互联网数据量，尤其是非结构化数据的数据量增长的越来越块，分布式对象存储应运而生。这种新兴的存储方式，适用于图片、视频等非结构化数据的存储。它同时又具有块存储高速直接访问磁盘的特点以及文件存储分布式共享的特点，因此一经推出就得到广泛的发展与应用。分布式对象存储具有多版本功能，当向同一个存储桶内重复上传同名文件时，为每一个同名文件设置随机生成的版本id，用于唯一的标识不同版本的对象，同时区别于同名文件覆盖机制，多版本不会删除之前的同名文件，而是将同名文件指定了版本id后存储在系统中。With the development of Internet technology, the amount of Internet data, especially the amount of unstructured data, has grown in blocks, and distributed object storage has emerged as the times require. This emerging storage method is suitable for storage of unstructured data such as pictures and videos. At the same time, it has the characteristics of high-speed direct access to disk of block storage and the characteristics of distributed sharing of file storage, so it has been widely developed and applied once it is launched. Distributed object storage has a multi-version function. When uploading files with the same name repeatedly to the same bucket, a randomly generated version id is set for each file with the same name, which is used to uniquely identify objects of different versions, and is different from the overwriting of files with the same name. Mechanism, multi-version will not delete the previous file with the same name, but store the file with the same name in the system after specifying the version id.
目前，在用户请求列举分布式对象存储系统中保存数据的历史版本时，需要遍历存储桶中的每一索引分片，每一索引分片又存储着大量键值对(key-value)，其中，key包括对象名和版本id，value包括对象的基本信息。遍历所有的索引分片、以及每一索引分片下的所有键值对后，将带有历史版本标记的对象作为列举结果返回。这种遍历所有存储对象的方式耗时长、效率低，当存储对象数量巨大时，还会严重影响存储系统的性能。At present, when a user requests to enumerate the historical versions of data stored in a distributed object storage system, it is necessary to traverse each index shard in the bucket, and each index shard stores a large number of key-value pairs, among which , the key includes the object name and version id, and the value includes the basic information of the object. After traversing all index shards and all key-value pairs under each index shard, return the object marked with the historical version as the enumeration result. This method of traversing all storage objects is time-consuming and inefficient. When the number of storage objects is huge, it will seriously affect the performance of the storage system.
所以，现在本领域的技术人员亟需要一种历史版本对象列举方法，解决目前的列举历史版本对象的方法效率低下且对存储系统性能影响大的问题。Therefore, those skilled in the art now urgently need a method for enumerating historical version objects, which solves the problems that the current method for enumerating historical version objects is inefficient and has a large impact on the performance of the storage system.
发明内容SUMMARY OF THE INVENTION
本申请的目的是提供一种历史版本对象列举方法、装置及其介质，以解决目前的列举历史版本对象的方法效率低下且对存储系统性能影响大的问题。The purpose of the present application is to provide a method, apparatus and medium for enumerating historical version objects, so as to solve the problems that the current method for enumerating historical version objects is inefficient and has a large impact on the performance of a storage system.
为解决上述技术问题，本申请提供一种历史版本对象列举方法，包括：In order to solve the above-mentioned technical problems, the present application provides a method for enumerating historical version objects, including:
当接收到历史版本对象列举请求时，根据历史版本前缀名获取目标键值对；其中，历史版本前缀名为当有同名对象新增时，添加并存储在历史版本对象键值对中的前缀名；When a historical version object enumeration request is received, the target key-value pair is obtained according to the historical version prefix name; the historical version prefix name is the prefix name added and stored in the historical version object key-value pair when an object with the same name is added. ;
遍历获取到的目标键值对，以获取历史版本对象的对象信息；Traverse the obtained target key-value pair to obtain the object information of the historical version object;
将历史版本对象的对象信息作为列举结果返回。Returns the object information of the historical version object as the enumeration result.
优选的，还包括：Preferably, it also includes:
接收由用户端上传的对象文件，并保存在对应的存储桶中；Receive the object file uploaded by the client and save it in the corresponding bucket;
若对象文件为存储桶中已有对象的同名对象文件时，为已有对象新增键值对记录；其中，键值对中的key包括：历史版本前缀名、对象名和版本id。If the object file is an object file with the same name as an existing object in the bucket, a new key-value pair record is added for the existing object; the key in the key-value pair includes the historical version prefix name, object name, and version id.
优选的，历史版本列举请求还包括：请求对象名；Preferably, the historical version enumeration request further includes: the request object name;
对应的，还包括：Correspondingly, it also includes:
对获取到的目标键值对根据请求对象名进行筛选，滤除对象名不为请求对象名的键值对，以获取新的目标键值对。The obtained target key-value pairs are filtered according to the request object name, and the key-value pairs whose object names are not the request object name are filtered out to obtain new target key-value pairs.
优选的，历史版本列举请求还包括：请求版本id；Preferably, the historical version enumeration request further includes: a request version id;
对应的，还包括：Correspondingly, it also includes:
对获取到的目标键值对根据请求版本id进行筛选，滤除版本id不为请求版本id的键值对，以获取新的目标键值对。The obtained target key-value pairs are filtered according to the request version id, and the key-value pairs whose version id is not the request version id are filtered out, so as to obtain a new target key-value pair.
优选的，在为已有对象新增键值对记录之后，还包括：Preferably, after adding a key-value pair record for an existing object, the method further includes:
将已有对象的原键值对记录删除。Delete the original key-value pair record of the existing object.
优选的，保存在对应的存储桶中包括：Preferably, the storage in the corresponding storage bucket includes:
根据对象文件的对象名进行哈希计算，以获取哈希值；Perform hash calculation based on the object name of the object file to obtain the hash value;
根据哈希值确定对应的存储桶。Determine the corresponding bucket according to the hash value.
优选的，在将历史版本对象的对象信息作为列举结果返回之后，还包括：Preferably, after returning the object information of the historical version object as the enumeration result, the method further includes:
返回提示信息至用户端。Return prompt information to the client.
为解决上述技术问题，本申请还提供一种历史版本对象列举装置，包括：In order to solve the above technical problems, the present application also provides a historical version object enumeration device, including:
请求接收模块，用于当接收到历史版本对象列举请求时，根据历史版本前缀名获取目标键值对；其中，历史版本前缀名为当有同名对象新增时，添加并存储在历史版本对象键值对中的前缀名；The request receiving module is used to obtain the target key-value pair according to the historical version prefix name when receiving the historical version object enumeration request; the historical version prefix name is added and stored in the historical version object key when an object with the same name is added. the prefix name in the value pair;
对象遍历模块，用于遍历获取到的目标键值对，以获取历史版本对象的对象信息；The object traversal module is used to traverse the obtained target key-value pairs to obtain the object information of the historical version object;
结果生成模块，用于将历史版本对象的对象信息作为列举结果返回。The result generation module is used to return the object information of the historical version object as the enumeration result.
优选的，还包括：Preferably, it also includes:
对象新增模块，用于接收由用户端上传的对象文件，并保存在对应的存储桶中；若对象文件为存储桶中已有对象的同名对象文件时，为已有对象新增键值对记录；其中，键值对中的key包括：历史版本前缀名、对象名和版本id。The object adding module is used to receive the object file uploaded by the client and save it in the corresponding bucket; if the object file is an object file with the same name as the existing object in the bucket, add a key-value pair for the existing object record; among them, the key in the key-value pair includes: historical version prefix name, object name and version id.
对象筛选模块，用于对获取到的目标键值对根据请求对象名进行筛选，滤除对象名不为请求对象名的键值对，以获取新的目标键值对。The object filtering module is used to filter the obtained target key-value pairs according to the request object name, and filter out the key-value pairs whose object name is not the request object name, so as to obtain new target key-value pairs.
版本id筛选模块，用于对获取到的目标键值对根据请求版本id进行筛选，滤除版本id不为请求版本id的键值对，以获取新的目标键值对。The version id filtering module is used to filter the obtained target key-value pairs according to the request version id, and filter out the key-value pairs whose version id is not the request version id, so as to obtain new target key-value pairs.
空间释放模块，用于将已有对象的原键值对记录删除。The space release module is used to delete the original key-value pair records of existing objects.
结果提示模块，用于返回提示信息至用户端。The result prompt module is used to return prompt information to the client.
为解决上述技术问题，本申请还提供一种历史版本对象列举装置，包括：In order to solve the above technical problems, the present application also provides a historical version object enumeration device, including:
存储器，用于存储计算机程序；memory for storing computer programs;
处理器，用于执行计算机程序时实现如上述的历史版本对象列举方法的步骤。The processor is configured to implement the steps of the above-mentioned historical version object enumeration method when executing the computer program.
为解决上述技术问题，本申请还提供一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，计算机程序被处理器执行时实现如上述的历史版本对象列举方法的步骤。To solve the above technical problems, the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned historical version object enumeration method are implemented.
本申请提供的一种历史版本对象列举方法，由于在对象存储的底层机制中，支持根据指定的前缀名来获取索引分片上的特定记录，所以本方法通过在新增同名对象文件存储时，对已有对象进行键值对记录的新增，且新增的键值对的key值以历史版本前缀名+对象名+版本id来表示，历史版本前缀名为一个特定的前缀名，仅由历史版本的对象使用，所以在进行历史版本对象的列举时，可以利用对象存储的底层机制，获取包括历史版本前缀名的所有键值对记录，进而遍历获取对象的基本信息，将其作为列举结果返回，完成历史版本对象列举过程。通过本申请所提供的列举方法，无需遍历每一索引分片中的所有键值对，仅需通过前缀名筛选出历史版本的对象后，在进行遍历以获取对象的基本信息即可，避免了无效遍历的时间消耗，同时也减轻了存储系统进行历史版本对象列举时的工作量，降低对其性能的影响。In a method for enumerating historical version objects provided by this application, since in the underlying mechanism of object storage, it is supported to obtain specific records on an index shard according to a specified prefix name. The existing object is added to the key-value pair record, and the key value of the new key-value pair is represented by the historical version prefix name + object name + version id. Version objects are used, so when enumerating historical version objects, you can use the underlying mechanism of object storage to obtain all key-value pair records including the prefix name of the historical version, and then traverse to obtain the basic information of the object, and return it as the enumeration result , to complete the historical version object enumeration process. With the enumeration method provided in this application, there is no need to traverse all the key-value pairs in each index shard. It is only necessary to filter out the objects of the historical version by the prefix name, and then traverse to obtain the basic information of the objects. The time consumption of invalid traversal also reduces the workload of the storage system when enumerating objects of historical versions, and reduces the impact on its performance.
本申请提供的历史版本对象列举装置、及计算机可读存储介质，与上述方法对应，效果同上。The historical version object enumeration device and the computer-readable storage medium provided by the present application correspond to the above method, and the effects are the same as above.
附图说明Description of drawings
为了更清楚地说明本申请实施例，下面将对实施例中所需要使用的附图做简单的介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to describe the embodiments of the present application more clearly, the following will briefly introduce the drawings that are used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, which are not relevant to ordinary skills in the art. As far as personnel are concerned, other drawings can also be obtained from these drawings on the premise of no creative work.
图1为本发明提供的一种历史版本对象列举方法的流程图；Fig. 1 is the flow chart of a kind of historical version object enumeration method provided by the present invention;
图2为本发明提供的一种历史版本对象列举装置的结构图；2 is a structural diagram of a device for enumerating historical version objects provided by the present invention;
图3为本发明提供的另一种历史版本对象列举装置的结构图。FIG. 3 is a structural diagram of another historical version object enumeration device provided by the present invention.
具体实施方式Detailed ways
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下，所获得的所有其他实施例，都属于本申请保护范围。The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative work fall within the protection scope of the present application.
本申请的核心是提供一种历史版本对象列举方法、装置及其介质。The core of this application is to provide a historical version object enumeration method, apparatus and medium thereof.
为了使本技术领域的人员更好地理解本申请方案，下面结合附图和具体实施方式对本申请作进一步的详细说明。In order to make those skilled in the art better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
在分布式对象存储系统的默认存储情况中，对象文件在存储桶中以键值对的形式进行存储，其中，key为对象文件的对象名，value为对象文件的基本信息。当向同一个存储桶内上传同名的对象文件时，由于桶内同名文件会发生覆盖，且对象存储没有回收站功能，所以当用户某些错误操作导致对象存储上存储的原文件被覆盖后，无法恢复被覆盖的原文件，致使文件丢失、无法恢复。因此，为了避免这种由于误操作造成的数据丢失的意外情况，人们为对象存储系统开发了多版本功能。当同一个桶内重复上传同名文件时，为每一个同名文件设置随机生成的版本身份标识号(identity document，id)，用于唯一标识不同版本的对象。同时区别于同名文件覆盖机制，多版本不会删除之前的同名文件，而是将同名文件指定了版本id后存储在系统中。新增了版本id后的历史版本对象的键值对value无变化，还是存储对象文件的基本信息，但key则变为对象名+版本id的形式。In the default storage situation of the distributed object storage system, object files are stored in the bucket in the form of key-value pairs, where the key is the object name of the object file, and the value is the basic information of the object file. When uploading an object file with the same name to the same bucket, the file with the same name in the bucket will be overwritten, and the object storage does not have a recycle bin function. The overwritten original file cannot be recovered, resulting in the file being lost and unrecoverable. Therefore, in order to avoid this unexpected situation of data loss due to misoperation, people have developed multi-version functions for object storage systems. When files with the same name are uploaded repeatedly in the same bucket, a randomly generated version identity number (identity document, id) is set for each file with the same name, which is used to uniquely identify objects of different versions. At the same time, different from the file overwriting mechanism of the same name, multi-version will not delete the previous file of the same name, but store the file of the same name in the system after specifying the version id. After the version id is added, the key-value pair value of the historical version object does not change, and the basic information of the object file is still stored, but the key changes to the form of object name + version id.
但是在分布式对象系统的应用场景中，多为对大量数据进行存储，用户通常在存储桶内存储上万乃至百万的对象，对于如此大规模的数据，如果用户开启了存储桶的多版本功能，且重复上传了多次同名文件，那么存储的数据量就会更加庞大，如何提高列举所有历史版本文件的性能以及速度就显得尤为重要了。However, in the application scenario of distributed object system, it is mostly for storing a large amount of data. Users usually store tens of thousands or even millions of objects in the bucket. For such large-scale data, if the user enables multiple versions of the bucket function, and the files with the same name are uploaded multiple times, the amount of data stored will be even larger. How to improve the performance and speed of listing all historical version files is particularly important.
目前的对象存储多版本功能在存储同名文件时，是通过在数据池中存储所有版本的文件的数据部分，同时在索引池中的索引分片上，记录了所有同名对象文件的基本信息，此时索引分片上记录的key与默认的存储情况有所不同，为对象名+版本id的形式，value为对象的基本信息。在列举存储桶历史版本时，需要遍历所有的桶索引分片上的所有键值对记录，并根据索引分片上记录的版本id，来区分是否为历史版本，并将历史版本对象返回至客户端，完成历史版本对象的列举过程。在这个过程中，首先需要遍历所有索引分片上的所有对象，然后将带有版本id的对象返回客户端。如果存储桶内对象文件数量达到百万甚至千万，且部分文件均有历史版本时，就会导致遍历耗时过长，且对存储系统的性能造成严重的影响。When the current object storage multi-version function stores files with the same name, it stores the data parts of all versions of the files in the data pool, and records the basic information of all object files with the same name on the index shards in the index pool. The key recorded on the index shard is different from the default storage. It is in the form of object name + version id, and value is the basic information of the object. When enumerating the historical versions of a bucket, it is necessary to traverse all key-value pair records on all bucket index shards, distinguish whether it is a historical version according to the version id recorded on the index shard, and return the historical version object to the client. Complete the enumeration process of historical version objects. In this process, all objects on all index shards need to be traversed first, and then the object with version id is returned to the client. If the number of object files in the bucket reaches one million or even ten million, and some files have historical versions, the traversal time will be too long, and the performance of the storage system will be seriously affected.
为解决上述的问题，如图1所示，本申请提供一种历史版本对象列举方法，包括：In order to solve the above-mentioned problems, as shown in FIG. 1, the present application provides a method for enumerating historical version objects, including:
S11：当接收到历史版本对象列举请求时，根据历史版本前缀名获取目标键值对。S11: When a historical version object enumeration request is received, the target key-value pair is obtained according to the historical version prefix name.
其中，历史版本前缀名为当有同名对象新增时，添加并存储在历史版本对象键值对中的前缀名。Among them, the prefix name of the historical version is the prefix name that is added and stored in the key-value pair of the historical version object when an object with the same name is added.
具体的，包含历史版本前缀名的键值对中的key为历史版本前缀名+对象名+版本id的形式。历史版本前缀名为分布式存储系统底层机制支持的前缀名的一种，本质上为一种特定的前缀名，用于表示存在该前缀名的键值对对应的对象为历史版本对象。所以通过历史版本前缀名便可以快速获取所有包含历史版本前缀名的键值对，也即目标键值对。Specifically, the key in the key-value pair including the historical version prefix name is in the form of the historical version prefix name + object name + version id. The historical version prefix name is one of the prefix names supported by the underlying mechanism of the distributed storage system, which is essentially a specific prefix name, which is used to indicate that the object corresponding to the key-value pair with the prefix name is a historical version object. Therefore, all the key-value pairs containing the prefix name of the historical version, that is, the target key-value pair, can be quickly obtained through the prefix name of the historical version.
S12：遍历获取到的目标键值对，以获取历史版本对象的对象信息。S12: Traverse the acquired target key-value pair to acquire object information of the historical version object.
具体的，也即通过遍历目标键值对的value，以获取对象的基本信息。容易理解的是，由于通过历史版本前缀名筛除一大部分非历史版本的对象键值对，所以在进行遍历时，减少了大量无效遍历，所需遍历的数据量也减少了很大一部分，从而实现列举效率的提升。Specifically, that is, by traversing the value of the target key-value pair, the basic information of the object is obtained. It is easy to understand that since a large number of object key-value pairs of non-historical versions are filtered out by the prefix name of the historical version, a large number of invalid traversals are reduced during traversal, and the amount of data to be traversed is also reduced by a large part. Thus, the efficiency of enumeration can be improved.
S13：将历史版本对象的对象信息作为列举结果返回。S13: Return the object information of the historical version object as the enumeration result.
根据用户端发送的列举需求不同，上述的对象信息可以是保存在键值对value中的对象的基本信息，也可以是通过对象基本信息从数据池中找到的对象的数据部分，本申请对此不做限制，可根据实际情况自由选择合适的实施方式。According to the different enumeration requirements sent by the client, the above-mentioned object information can be the basic information of the object stored in the key-value pair value, or the data part of the object found from the data pool through the basic information of the object. There is no restriction, and an appropriate implementation manner can be freely selected according to the actual situation.
另外，容易理解的是，虽然通过本申请所提供的一种历史版本对象列举方法，可以有效地降低无效遍历的次数，从而提高列举效率，但当存储系统中存储的对象数量过大时，仍需要一定的处理时间。所以为提高用户的使用体验，可以采用异步的方式实现存储系统和用户端之间的通信，当存储系统接收到用户端发送的列举请求后，返回提示信息以告知用户请求成功接收、正在进行处理；而后可以断开连接，在后台运行完成列举任务的进程或程序，避免被用户所感知到；当列举任务完成、生成列举结果后，在将列举结果返回至用户端，进一步的，还可以通过邮件、推送、短信、语音播报等方式，返回提示信息，以告知用户列举任务完成，请查收列举结果；从而进一步提高用户的使用体验。In addition, it is easy to understand that although the historical version object enumeration method provided by the present application can effectively reduce the number of invalid traversals, thereby improving the enumeration efficiency, when the number of objects stored in the storage system is too large, the number of objects stored in the storage system is still too large. Some processing time is required. Therefore, in order to improve the user experience, the communication between the storage system and the client can be implemented in an asynchronous manner. When the storage system receives the enumeration request sent by the client, it returns a prompt message to inform the user that the request has been successfully received and is being processed. ; Then you can disconnect and run the process or program that completes the enumeration task in the background to avoid being perceived by the user; when the enumeration task is completed and the enumeration result is generated, the enumeration result is returned to the user side, and further, you can also pass Email, push, text message, voice broadcast, etc., return prompt information to inform the user that the enumeration task is completed, please check the enumeration result, thereby further improving the user experience.
本申请所提供的一种历史版本对象列举方法，利用了分布式存储系统底层机制中的前缀名机制，支持根据前缀名对存储桶中保存的键值对进行筛选。所以在新增同名对象时，在其键值对中加入一个特定的前缀名，用于表示当前键值对对应的对象为历史版本对象。进而，在需要进行历史版本对象的列举时，便可以通过前缀名筛选的方式获取所有包含历史版本前缀名的键值对，进而对上述获取到的键值对进行遍历，以获取对象信息作为列举结果返回。上述列举过程中，在遍历阶段省去了大量的无效遍历，减少了所需进行遍历的数据量，使得遍历过程更简洁、直接、响应更快，从而提高遍历效率，并且减少对分布式对象存储系统性能的影响，更好地满足用户的历史版本对象列举需求。The method for enumerating historical version objects provided by the present application utilizes the prefix name mechanism in the underlying mechanism of a distributed storage system, and supports filtering of key-value pairs stored in a storage bucket according to the prefix name. Therefore, when adding an object with the same name, a specific prefix name is added to its key-value pair to indicate that the object corresponding to the current key-value pair is a historical version object. Furthermore, when it is necessary to enumerate the historical version objects, all key-value pairs containing the historical version prefix names can be obtained by filtering the prefix name, and then the obtained key-value pairs can be traversed to obtain the object information as an enumeration. The result is returned. In the above enumeration process, a large number of invalid traversals are omitted in the traversal phase, which reduces the amount of data to be traversed, making the traversal process more concise, direct, and responsive, thereby improving the traversal efficiency and reducing the need for distributed object storage. The impact of system performance can better meet the user's historical version object enumeration requirements.
由上述可知，本申请所提供的一种历史版本对象列举方法为实现减少无效遍历、提高列举性能、降低对分布式对象存储系统的性能的影响的有益效果，所采用的技术方案之一为当有同名对象新增时，需要在其键值对记录中添加历史版本前缀名。为进一步说明本申请提供的列举方法，下面对于存储系统中新增对象的步骤进行进一步说明：It can be seen from the above that a historical version object enumeration method provided by the present application achieves the beneficial effects of reducing invalid traversal, improving enumeration performance, and reducing the impact on the performance of the distributed object storage system. One of the technical solutions adopted is when When an object with the same name is added, it is necessary to add the historical version prefix name to its key-value pair record. To further illustrate the enumeration method provided by this application, the steps for adding an object in the storage system are further described below:
当存储系统中新增对象时，接收由客户端上传的对象文件，并保存在对应的存储桶中，一般来说，为对象文件分配存储桶的方式主要是根据对象文件的对象名来实现的，本实施例对此提供一种优选的实施方案，包括：When an object is added to the storage system, the object file uploaded by the client is received and stored in the corresponding bucket. Generally speaking, the method of allocating a bucket for an object file is mainly based on the object name of the object file. , this embodiment provides a preferred implementation for this, including:
根据对象文件的对象名进行哈希计算，以获取哈希值；Perform hash calculation based on the object name of the object file to obtain the hash value;
根据哈希值确定对应的存储桶。Determine the corresponding bucket according to the hash value.
通过对象文件的对象名可以以哈希计算的方式确定唯一的哈希值，进而通过哈希值和存储桶的对应关系确定对象文件所存储的存储桶，本实施例所提供的实施方案简单方便，容易实施且不会对存储系统带来过大的负担。The unique hash value can be determined by hash calculation through the object name of the object file, and then the bucket in which the object file is stored can be determined through the corresponding relationship between the hash value and the bucket. The implementation provided by this embodiment is simple and convenient , which is easy to implement and does not impose excessive burden on the storage system.
此时，存在两种情况，其一，该对象文件为新的对象，存储桶中无同名对象，对于该对象文件通过正常的存储方式即可，也即增加键值对记录以进行存储，其中key为对象名的形式。At this time, there are two situations. First, the object file is a new object, and there is no object with the same name in the bucket. The object file can be stored in a normal way, that is, adding a key-value pair record for storage, where The key is the form of the object name.
其二，该对象文件为存储桶中已有对象的同名对象，此时若开启分布式存储系统的多版本功能，应会新增键值对以保存已有对象的索引记录，新增键值对的key以历史版本前缀名+对象名+版本id的形式进行存储，而对于新增对象，仍以默认的形式进行存储即可。Second, the object file is an object with the same name as the existing object in the bucket. If the multi-version function of the distributed storage system is enabled at this time, a key-value pair should be added to save the index record of the existing object, and the new key-value pair should be added. The right key is stored in the form of historical version prefix name + object name + version id, and for newly added objects, it can still be stored in the default form.
基于上述步骤完成新增对象的存储后，当用户端发送历史版本对象的列举请求时，便可以通过历史版本快速确定目标键值对，后续仅需通过对目标键值对进行遍历，即可获取历史版本对象的信息，从而作为列举结果返回，完成历史版本对象列举任务。After completing the storage of the newly added objects based on the above steps, when the client sends an enumeration request for the historical version objects, the target key-value pair can be quickly determined through the historical version, and then the target key-value pair can be obtained by traversing the target key-value pair. The information of the historical version object is returned as the enumeration result to complete the historical version object enumeration task.
进一步的，在为已有对象新增键值对作为索引记录后，原有的键值对则随之失去效用，所以，为进一步提高存储系统存储能力，本实施例还提供一种优选的实施方案，在为已有对象新增键值对记录之后，还包括：Further, after a key-value pair is added as an index record for an existing object, the original key-value pair will lose its effectiveness. Therefore, in order to further improve the storage capacity of the storage system, this embodiment also provides a preferred implementation. The scheme, after adding a key-value pair record for an existing object, also includes:
将已有对象的原键值对记录删除。Delete the original key-value pair record of the existing object.
由于通过上述步骤，对已有对象的索引记录进行了新增的处理方式，新增的键值对包括历史版本前缀名，以此实现提高列举效率的效果。所以，已有对象的原有索引记录内容可由新的键值对来体现，旧索引记录无用，为节省存储系统的存储空间，本实施例所以提供的优选方案将其删除，以释放存储空间，进一步提高分布式对象存储系统的存储能力。Through the above steps, a new processing method is performed on the index records of the existing objects, and the newly added key-value pair includes the prefix name of the historical version, so as to achieve the effect of improving the enumeration efficiency. Therefore, the original index record content of an existing object can be represented by a new key-value pair, and the old index record is useless. In order to save the storage space of the storage system, the preferred solution provided in this embodiment deletes it to free up the storage space. Further improve the storage capacity of the distributed object storage system.
本实施例所提供的一种优选方案，通过存储系统在新增同名对象时，对已有对象重新新增键值对以作为索引记录保存，新增的键值对除去按照现有的多版本功能要求的格式之外，还包括历史版本前缀名，所以可以利用分布式对象存储系统的现有底层机制，通过前缀名筛选出所需的键值对，相应的，在进行遍历时也仅需遍历包括历史版本前缀名的键值对即可，减少了大量的无效遍历，从而提高列举效率，降低对存储系统性能的影响。In a preferred solution provided by this embodiment, when an object with the same name is added through the storage system, a key-value pair is newly added to an existing object to be stored as an index record, and the newly-added key-value pair is removed according to the existing multi-version In addition to the format required by the function, it also includes the prefix name of the historical version, so the existing underlying mechanism of the distributed object storage system can be used to filter out the required key-value pairs through the prefix name. Correspondingly, when traversing, only It is enough to traverse the key-value pairs including the prefix name of the historical version, which reduces a large number of invalid traversals, thereby improving the enumeration efficiency and reducing the impact on the performance of the storage system.
容易理解的是，上述实施例所针对的列举应用场景，列举需求多为对存储系统的所有历史版本对象进行列举。但在实际使用中，也存在个性化的列举需求，例如，针对某些对象的历史版本，或针对某些对象的某些历史版本的列举需求。所以，为解决上述问题，本实施例提供一种优选的实施方案，历史版本列举请求还包括：请求对象名；It is easy to understand that, in the enumeration application scenarios targeted by the above embodiments, the enumeration requirements are mostly enumeration of all historical version objects of the storage system. However, in actual use, there are also personalized enumeration requirements, for example, the historical versions of certain objects, or the enumeration requirements of certain historical versions of certain objects. Therefore, in order to solve the above problem, this embodiment provides a preferred implementation, and the historical version enumeration request further includes: the request object name;
对应的，本方法还包括：Correspondingly, the method further includes:
对获取到的目标键值对根据请求对象名进行筛选，滤除对象名不为请求对象名的键值对，以获取新的目标键值对。The obtained target key-value pairs are filtered according to the request object name, and the key-value pairs whose object names are not the request object name are filtered out to obtain new target key-value pairs.
上述的根据请求对象名滤除不需要的对象可在遍历目标键值对的步骤中实现，当遍历目标键值对时，自然可获取每一键值对的key，进而得知当前对象的对象名和版本id。从而实现根据请求对象名对列举结果进行过滤，使返回的列举结果更符合用户的实际需求。The above-mentioned filtering out unwanted objects according to the request object name can be implemented in the step of traversing the target key-value pair. When traversing the target key-value pair, the key of each key-value pair can naturally be obtained, and then the object of the current object can be known. name and version id. In this way, the enumeration results are filtered according to the request object name, so that the returned enumeration results are more in line with the actual needs of users.
进一步的，用户的列举需求若还包括请求版本id；Further, if the user's enumeration requirements also include the request version id;
对应的，上述方法还包括：Correspondingly, the above method further includes:
对获取到的目标键值对根据请求版本id进行筛选，滤除版本id不为请求版本id的键值对，以获取新的目标键值对。The obtained target key-value pairs are filtered according to the request version id, and the key-value pairs whose version id is not the request version id are filtered out, so as to obtain a new target key-value pair.
由上述可知，当开启分布式存储系统的多版本功能后，当有同名对象新增时，会以对象名+版本id的形式新增键值对存储，且版本id为随机生成的，用于唯一标识不同版本的对象。所以根据版本id，可以过滤去不需要的历史版本对象，该用于滤去不需要历史版本对象的版本id，即为上述的请求版本id。It can be seen from the above that when the multi-version function of the distributed storage system is enabled, when an object with the same name is added, the key-value pair storage will be added in the form of object name + version id, and the version id is randomly generated for use. Uniquely identifies different versions of an object. Therefore, according to the version id, unnecessary historical version objects can be filtered out, and the version id used to filter out unnecessary historical version objects is the above-mentioned request version id.
需要说明的是，上述实施例所提供的方法主要针对用户对某些特定对象有历史版本列举需求、以及对某些对象的具体某些版本有列举需求等情况所提出的。原因为在实际应用中，用户常见的需求为上述两种，对于其他情况例如不针对特定对象，而是针对特定历史版本的对象进行列举的需求在实际应用中不常见，故本实施例在此不对其具体的筛选方式进行赘述，但具体的实施方案与上述同理。It should be noted that, the methods provided by the above embodiments are mainly proposed for the situation that the user has a requirement for enumerating historical versions of some specific objects, and a requirement for enumerating specific versions of some objects. The reason is that in practical applications, the common requirements of users are the above two. For other cases, for example, the requirements for enumerating objects of specific historical versions, not for specific objects, are not common in practical applications, so this embodiment is here. The specific screening method will not be repeated, but the specific implementation is the same as the above.
本实施例所提供的优选方案，通过列举请求中的请求对象名以及请求版本id，来对目标键值对进行筛选，滤除掉用户所不需要的对象，从而使列举结果精简化，且更符合用户的列举需求。In the preferred solution provided by this embodiment, the target key-value pair is filtered by enumerating the request object name and the request version id in the request, and the objects that the user does not need are filtered out, thereby simplifying the enumeration results and improving the Meet the user's enumeration needs.
为进一步说明本申请所提供的一种历史版本对象列举方法，下面结合实例，对上述实施例所提供的优选方案做出清楚、详实的说明：In order to further illustrate a historical version object enumeration method provided by the present application, the preferred solutions provided by the above-described embodiments are described in a clear and detailed manner below in conjunction with examples:
用户在通过控制指令开启存储桶的多版本功能之后，上传对象文件；从而根据哈希计算的方式在索引池中确定到某一索引分片，在该索引分片上记录一条正常的索引记录。正常的索引记录也即key为对象名+版本id，value为对象基本信息的键值对记录。After enabling the multi-version function of the bucket through the control command, the user uploads the object file; thus, a certain index shard is determined in the index pool according to the hash calculation method, and a normal index record is recorded on the index shard. The normal index record is that the key is the object name + version id, and the value is the key-value pair record of the basic information of the object.
而当用户重复上传一个同名的对象文件到同一个存储桶内时，由于对象名相同，因此通过哈希计算的方式应划分在同一个索引分片上。此时需为上一个同名对象新增一条键值对记录，其中，key为历史版本前缀名+对象名+版本id，value保存为上一个同名对象的基本信息。When a user repeatedly uploads an object file with the same name to the same bucket, since the object name is the same, it should be divided into the same index shard by hash calculation. At this point, you need to add a key-value pair record for the last object with the same name, where the key is the historical version prefix name + object name + version id, and the value is saved as the basic information of the last object with the same name.
删除原有的上一个同名对象的索引记录，即key为对象名+版本id，value为对象基本信息的键值对记录。Delete the original index record of the previous object with the same name, that is, the key is the object name + version id, and the value is the key-value pair record of the basic information of the object.
为新写入的同名对象在索引分片上新增一条正常的索引记录，key为对象名+版本id，value为对象基本信息。Add a normal index record to the index shard for the newly written object with the same name. The key is the object name + version id, and the value is the basic information of the object.
重复上传同名对象时，重复步骤上述步骤，以完成索引记录的存储。When uploading objects with the same name repeatedly, repeat the above steps to complete the storage of index records.
当分布式对象存储系统接收到来自用户端发送的列举历史版本对象请求时，首先遍历索引池中的所有索引分片，然后在遍历到每一个索引分片时指定历史版本的前缀名，根据前缀名过滤出所有的历史版本对象的索引记录，也即键值对，然后根据键值对读取对象的基本信息，根据对象基本信息中的历史版本标识，将符合的历史版本对象返回至客户端，完成列举所有历史版本对象的过程。When the distributed object storage system receives a request to enumerate historical version objects sent from the client, it first traverses all index shards in the index pool, and then specifies the prefix name of the historical version when traversing each index shard, according to the prefix Filter out the index records of all historical version objects by name, that is, key-value pairs, and then read the basic information of the object according to the key-value pair, and return the matching historical version objects to the client according to the historical version identifier in the basic information of the object. , completes the process of enumerating all historical version objects.
本申请提出了一种改进对象存储多版本功能列举历史版本对象的方案，提供了一种历史版本对象列举方法，在桶索引分片上为历史版本对象添加专属的索引记录，即在桶索引分片上记录历史版本对象时，key记为历史版本前缀名+对象名+版本id，value仍为对象的基本信息。其中，历史版本前缀名为历史版本对象专有的前缀名，在对象存储底层机制中支持指定前缀名来获取索引分片上的特定记录。在列举多版本对象时可以通过指定历史版本前缀名获取索引分片上历史版本对应的记录，读取记录的对象的基本信息后将历史版本对象信息返回到客户端，完成列举过程。在添加新增历史版本时，虽然在索引分片上新增了专属记录，在对象数量较多时增加了索引分片的空间占用，但是避免了遍历所有索引分片上所有对象的时间开销，减少了无效遍历，同时通过指定的专属前缀在索引分片上过滤出需要的历史版本对象记录，查找更准确，响应更快。相比于之前每次列举请求发送，对象存储网关系统(RGW)服务端都要遍历所有桶索引分片上所有对象的方法。本申请一方面在遍历桶索引分片时可以直接通过指定前缀获取历史版本对象的索引记录，可以减少桶索引分片上的无效遍历，提高遍历的准确性；另一方面，直接从桶索引分片上读取到历史版本对象，相比于遍历全部对象再筛选出历史版本对象的方式，可以显著提高列举的速度、降低对存储系统性能的影响。The present application proposes a scheme for improving the multi-version function of object storage to enumerate historical version objects, and provides a method for enumerating historical version objects, adding exclusive index records for historical version objects on bucket index shards, that is, on bucket index shards When recording a historical version object, the key is recorded as the historical version prefix name + object name + version id, and the value is still the basic information of the object. Among them, the prefix name of the historical version is the unique prefix name of the historical version object, and the underlying mechanism of the object storage supports specifying the prefix name to obtain specific records on the index shard. When enumerating multi-version objects, you can obtain the records corresponding to the historical version on the index shard by specifying the prefix name of the historical version, and then read the basic information of the recorded object and return the historical version object information to the client to complete the enumeration process. When adding a new historical version, although an exclusive record is added to the index shard, which increases the space occupation of the index shard when the number of objects is large, it avoids the time overhead of traversing all objects on all index shards and reduces invalidation. Traverse, and filter out the required historical version object records on the index shard through the specified exclusive prefix, so that the search is more accurate and the response is faster. Compared with the previous method of sending each enumeration request, the object storage gateway system (RGW) server needs to traverse all objects on all bucket index shards. On the one hand, when traversing bucket index shards, the present application can directly obtain the index records of historical version objects by specifying a prefix, which can reduce invalid traversal on bucket index shards and improve the accuracy of traversal; on the other hand, directly from bucket index shards When reading historical version objects, compared to traversing all objects and then filtering out historical version objects, the enumeration speed can be significantly improved and the impact on storage system performance can be reduced.
在上述实施例中，对于一种历史版本对象列举方法进行了详细描述，本申请还提供一种历史版本对象列举装置对应的实施例。需要说明的是，本申请从两个角度对装置部分的实施例进行描述，一种是基于功能模块的角度，另一种是基于硬件的角度。In the above embodiments, a method for enumerating historical version objects is described in detail, and the present application also provides an embodiment corresponding to an apparatus for enumerating historical version objects. It should be noted that this application describes the embodiments of the device part from two perspectives, one is based on the perspective of functional modules, and the other is based on the perspective of hardware.
基于功能模块的角度，如图2所示，本实施例提供一种历史版本对象列举装置，包括：From the perspective of functional modules, as shown in FIG. 2 , this embodiment provides a device for enumerating historical version objects, including:
请求接收模块21，用于当接收到历史版本对象列举请求时，根据历史版本前缀名获取目标键值对；其中，历史版本前缀名为当有同名对象新增时，添加并存储在历史版本对象键值对中的前缀名；The
对象遍历模块22，用于遍历获取到的目标键值对，以获取历史版本对象的对象信息；The
结果生成模块23，用于将历史版本对象的对象信息作为列举结果返回。The
优选的，还包括：Preferably, it also includes:
对象新增模块，用于接收由用户端上传的对象文件，并保存在对应的存储桶中；若对象文件为存储桶中已有对象的同名对象文件时，为已有对象新增键值对记录；其中，键值对中的key包括：历史版本前缀名、对象名和版本id。The object adding module is used to receive the object file uploaded by the client and save it in the corresponding bucket; if the object file is an object file with the same name as the existing object in the bucket, add a key-value pair for the existing object record; among them, the key in the key-value pair includes: historical version prefix name, object name and version id.
对象筛选模块，用于对获取到的目标键值对根据请求对象名进行筛选，滤除对象名不为请求对象名的键值对，以获取新的目标键值对。The object filtering module is used to filter the obtained target key-value pairs according to the request object name, and filter out the key-value pairs whose object name is not the request object name, so as to obtain new target key-value pairs.
版本id筛选模块，用于对获取到的目标键值对根据请求版本id进行筛选，滤除版本id不为请求版本id的键值对，以获取新的目标键值对。The version id filtering module is used to filter the obtained target key-value pairs according to the request version id, and filter out the key-value pairs whose version id is not the request version id, so as to obtain new target key-value pairs.
空间释放模块，用于将已有对象的原键值对记录删除。The space release module is used to delete the original key-value pair records of existing objects.
结果提示模块，用于返回提示信息至用户端。The result prompt module is used to return prompt information to the client.
由于装置部分的实施例与方法部分的实施例相互对应，因此装置部分的实施例请参见方法部分的实施例的描述，这里暂不赘述。Since the embodiment of the apparatus part corresponds to the embodiment of the method part, for the embodiment of the apparatus part, please refer to the description of the embodiment of the method part, which will not be repeated here.
本实施例所提供的一种历史版本对象列举装置，利用了分布式存储系统底层机制中的前缀名机制，由于该机制支持根据前缀名对存储桶中保存的键值对进行筛选。所以在新增同名对象时，在其键值对中加入一个特定的前缀名，用于表示当前键值对对应的对象为历史版本对象。进而，在需要进行历史版本对象的列举时，便可以通过前缀名筛选的方式获取所有包含历史版本前缀名的键值对，进而通过对象遍历模块对上述获取到的键值对进行遍历，以获取对象信息作为列举结果返回。上述列举过程中，在遍历阶段省去了大量的无效遍历，使得遍历过程更简洁、直接、响应更快，减少了所需进行遍历的数据量，从而提高遍历效率，并且减少对分布式对象存储系统性能的影响，更好地满足用户的历史版本对象列举需求。The apparatus for enumerating historical version objects provided in this embodiment utilizes the prefix name mechanism in the underlying mechanism of the distributed storage system, because the mechanism supports filtering the key-value pairs stored in the storage bucket according to the prefix name. Therefore, when adding an object with the same name, a specific prefix name is added to its key-value pair to indicate that the object corresponding to the current key-value pair is a historical version object. Furthermore, when it is necessary to enumerate historical version objects, all key-value pairs containing the prefix name of the historical version can be obtained by means of prefix name filtering, and then the above-obtained key-value pairs can be traversed through the object traversal module to obtain Object information is returned as the enumeration result. In the above enumeration process, a large number of invalid traversals are omitted in the traversal phase, which makes the traversal process more concise, direct, and responds faster, reduces the amount of data that needs to be traversed, thereby improves the traversal efficiency and reduces the need for distributed object storage. The impact of system performance can better meet the user's historical version object enumeration requirements.
图3为本申请另一实施例提供的一种历史版本对象列举装置的结构图，如图3所示，一种历史版本对象列举装置包括：存储器30，用于存储计算机程序；FIG. 3 is a structural diagram of an apparatus for enumerating objects of historical versions provided by another embodiment of the present application. As shown in FIG. 3 , an apparatus for enumerating objects of historical versions includes: a memory 30 for storing computer programs;
处理器31，用于执行计算机程序时实现如上述实施例一种历史版本对象列举方法的步骤。The processor 31 is configured to implement the steps of the method for enumerating historical version objects according to the above embodiment when executing the computer program.
本实施例提供的一种历史版本对象列举装置可以包括但不限于智能手机、平板电脑、笔记本电脑或台式电脑等。A historical version object enumeration device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer.
其中，处理器31可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器31可以采用数字信号处理器(Digital Signal Processor，DSP)、现场可编程门阵列(Field－Programmable Gate Array，FPGA)、可编程逻辑阵列(Programmable LogicArray，PLA)中的至少一种硬件形式来实现。处理器31也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称中央处理器(CentralProcessing Unit，CPU)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器31可以集成有图像处理器(Graphics Processing Unit，GPU)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器31还可以包括人工智能(Artificial Intelligence，AI)处理器，该AI处理器用于处理有关机器学习的计算操作。The processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 31 may use at least one hardware form of a digital signal processor (Digital Signal Processor, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA). accomplish. The processor 31 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake-up state, also called a central processing unit (CPU); A low-power processor for processing data in a standby state. In some embodiments, the processor 31 may be integrated with a graphics processor (Graphics Processing Unit, GPU), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 31 may further include an artificial intelligence (Artificial Intelligence, AI) processor for processing computing operations related to machine learning.
存储器30可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器30还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。本实施例中，存储器30至少用于存储以下计算机程序301，其中，该计算机程序被处理器31加载并执行之后，能够实现前述任一实施例公开的一种历史版本对象列举方法的相关步骤。另外，存储器30所存储的资源还可以包括操作系统302和数据303等，存储方式可以是短暂存储或者永久存储。其中，操作系统302可以包括Windows、Unix、Linux等。数据303可以包括但不限于一种历史版本对象列举方法等。Memory 30 may include one or more computer-readable storage media, which may be non-transitory. Memory 30 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash storage devices. In this embodiment, the memory 30 is at least used to store the following computer program 301 , where, after the computer program is loaded and executed by the processor 31 , it can implement the relevant steps of a method for enumerating historical version objects disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 30 may also include an operating system 302, data 303, etc., and the storage mode may be short-term storage or permanent storage. The operating system 302 may include Windows, Unix, Linux, and the like. The data 303 may include, but is not limited to, a historical version object enumeration method and the like.
在一些实施例中，一种历史版本对象列举装置还可包括有显示屏32、输入输出接口33、通信接口34、电源35以及通信总线36。In some embodiments, a historical version object enumeration device may further include a display screen 32 , an input/output interface 33 , a communication interface 34 , a power supply 35 and a communication bus 36 .
本领域技术人员可以理解，图3中示出的结构并不构成对一种历史版本对象列举装置的限定，可以包括比图示更多或更少的组件。Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation to a historical version object enumeration device, and may include more or less components than those shown in the drawings.
本申请实施例提供的一种历史版本对象列举装置，包括存储器和处理器，处理器在执行存储器存储的程序时，能够实现如下方法：一种历史版本对象列举方法。An apparatus for enumerating historical version objects provided by an embodiment of the present application includes a memory and a processor. When the processor executes a program stored in the memory, the processor can implement the following method: a method for enumerating historical version objects.
本实施例所提供的一种历史版本对象列举装置，利用了分布式存储系统底层机制中的前缀名机制，由于该机制支持根据前缀名对存储桶中保存的键值对进行筛选。所以在新增同名对象时，在其键值对中加入一个特定的前缀名，用于表示当前键值对对应的对象为历史版本对象。进而，当处理器执行保存在存储器中的计算机程序时，便可以通过前缀名筛选的方式获取所有包含历史版本前缀名的键值对，进而通过对象遍历模块对上述获取到的键值对进行遍历，以获取对象信息作为列举结果返回。上述列举过程中，在遍历阶段省去了大量的无效遍历，使得遍历过程更简洁、直接、响应更快，减少了所需进行遍历的数据量，从而提高遍历效率，并且减少对分布式对象存储系统性能的影响，更好地满足用户的历史版本对象列举需求。The apparatus for enumerating historical version objects provided in this embodiment utilizes the prefix name mechanism in the underlying mechanism of the distributed storage system, because the mechanism supports filtering the key-value pairs stored in the storage bucket according to the prefix name. Therefore, when adding an object with the same name, a specific prefix name is added to its key-value pair to indicate that the object corresponding to the current key-value pair is a historical version object. Furthermore, when the processor executes the computer program stored in the memory, it can obtain all key-value pairs containing the prefix name of the historical version by means of prefix name filtering, and then traverse the obtained key-value pairs through the object traversal module. , to get the object information and return it as the enumeration result. In the above enumeration process, a large number of invalid traversals are omitted in the traversal phase, which makes the traversal process more concise, direct, and responds faster, reduces the amount of data that needs to be traversed, thereby improves the traversal efficiency and reduces the need for distributed object storage. The impact of system performance can better meet the user's historical version object enumeration requirements.
最后，本申请还提供一种计算机可读存储介质对应的实施例。计算机可读存储介质上存储有计算机程序，计算机程序被处理器执行时实现如上述方法实施例中记载的步骤。Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps described in the foregoing method embodiments are implemented.
可以理解的是，如果上述实施例中的方法以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。It can be understood that, if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .
本实施例所提供的一种计算机可读取存储介质，利用了分布式存储系统底层机制中的前缀名机制，由于该机制支持根据前缀名对存储桶中保存的键值对进行筛选。所以在新增同名对象时，在其键值对中加入一个特定的前缀名，用于表示当前键值对对应的对象为历史版本对象。进而，当其中存储的计算机程序被执行时，可以实现通过前缀名筛选的方式获取所有包含历史版本前缀名的键值对，进而通过对象遍历模块对上述获取到的键值对进行遍历，以获取对象信息作为列举结果返回。上述列举过程中，在遍历阶段省去了大量的无效遍历，使得遍历过程更简洁、直接、响应更快，减少了所需进行遍历的数据量，从而提高遍历效率，并且减少对分布式对象存储系统性能的影响，更好地满足用户的历史版本对象列举需求。The computer-readable storage medium provided in this embodiment utilizes the prefix name mechanism in the underlying mechanism of the distributed storage system, because the mechanism supports screening of key-value pairs stored in the storage bucket according to the prefix name. Therefore, when adding an object with the same name, a specific prefix name is added to its key-value pair to indicate that the object corresponding to the current key-value pair is a historical version object. Furthermore, when the computer program stored therein is executed, all key-value pairs containing the prefix name of the historical version can be obtained by means of prefix name filtering, and then the above-obtained key-value pairs can be traversed through the object traversal module to obtain Object information is returned as the enumeration result. In the above enumeration process, a large number of invalid traversals are omitted in the traversal phase, which makes the traversal process more concise, direct, and responds faster, reduces the amount of data that needs to be traversed, thereby improves the traversal efficiency and reduces the need for distributed object storage. The impact of system performance can better meet the user's historical version object enumeration requirements.
以上对本申请所提供的一种历史版本对象列举方法、装置及其介质进行了详细介绍。说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请权利要求的保护范围内。A method, apparatus and medium for enumerating historical version objects provided by the present application have been described above in detail. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method. It should be pointed out that for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can also be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.
还需要说明的是，在本说明书中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations. There is no such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.
Claims (10)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210832787.2ACN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210832787.2ACN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115114232Atrue CN115114232A (en) | 2022-09-27 |
Family
ID=83332365
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210832787.2APendingCN115114232A (en) | 2022-07-15 | 2022-07-15 | Method, device and medium for enumerating historical version objects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115114232A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en)* | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN117033317A (en)* | 2023-09-28 | 2023-11-10 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102193925A (en)* | 2010-03-05 | 2011-09-21 | 新奥特（北京）视频技术有限公司 | On-line multi-version management method and device for manuscripts in manuscript system |
| US20130226955A1 (en)* | 2012-02-28 | 2013-08-29 | International Business Machines Corporation | Bi-temporal key value cache system |
| CN107832470A (en)* | 2017-11-29 | 2018-03-23 | 郑州云海信息技术有限公司 | A kind of object storage method and device based on storage system |
| US20210034674A1 (en)* | 2019-08-02 | 2021-02-04 | EMC IP Holding Company LLC | Cuckoo tree with duplicate key support |
| CN114138716A (en)* | 2021-11-22 | 2022-03-04 | 杭州宏杉科技股份有限公司 | Method, device, electronic device and storage medium for storing objects |
- 2022
- 2022-07-15CNCN202210832787.2Apatent/CN115114232A/enactivePending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102193925A (en)* | 2010-03-05 | 2011-09-21 | 新奥特（北京）视频技术有限公司 | On-line multi-version management method and device for manuscripts in manuscript system |
| US20130226955A1 (en)* | 2012-02-28 | 2013-08-29 | International Business Machines Corporation | Bi-temporal key value cache system |
| CN107832470A (en)* | 2017-11-29 | 2018-03-23 | 郑州云海信息技术有限公司 | A kind of object storage method and device based on storage system |
| US20210034674A1 (en)* | 2019-08-02 | 2021-02-04 | EMC IP Holding Company LLC | Cuckoo tree with duplicate key support |
| CN114138716A (en)* | 2021-11-22 | 2022-03-04 | 杭州宏杉科技股份有限公司 | Method, device, electronic device and storage medium for storing objects |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116150093A (en)* | 2023-03-04 | 2023-05-23 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN116150093B (en)* | 2023-03-04 | 2023-11-03 | 北京大道云行科技有限公司 | Method for realizing object storage enumeration of objects and electronic equipment |
| CN117033317A (en)* | 2023-09-28 | 2023-11-10 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
| CN117033317B (en)* | 2023-09-28 | 2024-02-02 | 苏州元脑智能科技有限公司 | Enumeration information acquisition method, system, device, communication equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, apparatus and system for storing data | |
| US11237761B2 (en) | Management of multiple physical function nonvolatile memory devices | |
| CN105631035B (en) | Date storage method and device | |
| CN113535068B (en) | Data reading method and system | |
| CN110737682A (en) | cache operation method, device, storage medium and electronic equipment | |
| US20180075159A1 (en) | Efficient property graph storage for streaming / multi-versioning graphs | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN109804359A (en) | For the system and method by write back data to storage equipment | |
| US11977785B2 (en) | Non-volatile memory device-assisted live migration of virtual machine data | |
| CN113297267B (en) | Data caching and task processing method, device, equipment and storage medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN102971698A (en) | Snapshot data-processing method and system, storage system and snapshot agency | |
| US11520818B2 (en) | Method, apparatus and computer program product for managing metadata of storage object | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN105808451B (en) | Data caching method and related device | |
| JP2024509198A (en) | Soft deletion of data in a sharded database | |
| WO2024198872A1 (en) | Data management system, and data updating method and apparatus | |
| US11720551B1 (en) | Method and system for streaming data from portable storage devices | |
| CN114356949B (en) | A method, device and medium for maintaining cache data consistency | |
| CN114968114A (en) | Object storage method, device and medium | |
| CN115510016A (en) | A client response method, device and medium based on directory fragmentation | |
| CN115309701A (en) | A file migration method, device and medium of a distributed file system | |
| CN115525219A (en) | Object data storage method, device and medium | |
| CN114911410A (en) | Data storage method, device, equipment and storage medium | |
| CN112835967A (en) | Data processing method, apparatus, device and medium based on distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |