技术领域technical field
本发明涉及多客户端并发访问的技术领域，特别是涉及一种多客户端并发数据读写加速方法及装置。The invention relates to the technical field of concurrent access by multiple clients, in particular to a method and device for accelerating concurrent data reading and writing by multiple clients.
背景技术Background technique
随着科技的进一步发展，大数据时代的逐步来临，存储系统应用越来越广泛。存储应用场景越来越广泛，其中海量数据的处理场景非常多。尤其是多客户端并发访问，对存储的IO响应速度要求极高。With the further development of science and technology and the gradual advent of the era of big data, the application of storage systems is becoming more and more extensive. Storage application scenarios are becoming more and more extensive, and there are many scenarios for processing massive data. In particular, concurrent access by multiple clients requires extremely high IO response speed of the storage.
目前，当多客户端并发访问海量数据时，每个客户端都会使用缓存来提高读写速度。最简单的情况是每个客户端分别管理自己的缓存数据，每个客户端在读操作时会将读到的内容放入缓存，写入的数据也是先写到缓存，这样如果再有对相同数据的读写操作，会直接操作缓存中的数据，提高速度，然后客户端会定期执行缓存数据下刷操作，具体缓存数据下刷选择算法此处不再赘述。在这种情况下，如果多个客户端操作相同的数据，会出现数据不一致的情况。所以为了解决这种问题，采用对数据加锁的方式，当有一个客户端对一个数据块（往往是连续多个数据块）进行读写操作时，会通知存储系统当前有客户端要对某一段数据进行读写，为了防止数据不一致，要求对该数据段进行加锁，然后使当前客户端独占该数据段的缓存，进行读写操作。当有其他客户端同样读写相同数据段时，新客户端获取到数据段的加锁后，之前独占该数据段的客户端需要将数据下刷到磁盘。这样对于一个数据段，同一时刻只有一个客户端独占数据缓存，这样可以防止出现数据不一致的情况。但是对高并发读写场景会降低性能。Currently, when multiple clients access massive amounts of data concurrently, each client uses a cache to improve read and write speed. The simplest case is that each client manages its own cache data separately. Each client puts the read content into the cache during the read operation, and writes the data to the cache first, so that if there is another pair of the same data The read and write operations will directly operate the data in the cache to increase the speed, and then the client will periodically perform cache data refresh operations. The specific cache data refresh selection algorithm will not be described here. In this case, if multiple clients operate on the same data, data inconsistency will occur. Therefore, in order to solve this problem, the method of locking data is adopted. When a client reads and writes a data block (often multiple consecutive data blocks), it will notify the storage system that there is currently a client that wants to lock a certain data block. A piece of data is read and written. In order to prevent data inconsistency, it is required to lock the data section, and then let the current client monopolize the cache of the data section to perform read and write operations. When other clients also read and write the same data segment, after the new client obtains the lock of the data segment, the client that previously exclusively occupied the data segment needs to flush the data to disk. In this way, for a data segment, only one client has an exclusive data cache at the same time, which can prevent data inconsistency. However, for high-concurrency read and write scenarios, performance will be reduced.
在多客户端并发访问的场景中，大多数的访问模式是客户端频繁进行读操作，但是写操作会比较少，如果能提高读操作对并发的影响，这种高并发读写的场景IO读写速度会极大提高。In the scenario of multi-client concurrent access, most of the access modes are that the client performs frequent read operations, but the write operations will be relatively small. If the impact of read operations on concurrency can be improved, this high-concurrency read and write scenario IO read Write speed will be greatly improved.
发明内容Contents of the invention
本发明目的是提供一种多客户端并发数据读写加速方法及装置，本发明通过在多客户端加入缓存临时占有机制，使多客户端在并发处理读操作请求时能够极大提升IO速度。The purpose of the present invention is to provide a multi-client concurrent data reading and writing acceleration method and device. The present invention adds a cache temporary occupation mechanism to the multi-client, so that the multi-client can greatly increase the IO speed when concurrently processing read operation requests.
为了实现上述目的，本发明采用以下的技术方案：In order to achieve the above object, the present invention adopts the following technical solutions:
一种多客户端并发数据读写加速方法，包括以下步骤：A multi-client concurrent data reading and writing acceleration method, comprising the following steps:
获取数据访问请求信息；Obtain data access request information;
任一数据，若数据访问请求中存在至多一个写操作请求，则将该数据放入缓存中并通过缓存中数据执行各访问操作；若数据访问请求中存在至少两个写操作请求，则将缓存中数据按时序下刷至存储设备中。For any data, if there is at most one write operation request in the data access request, put the data into the cache and perform each access operation through the data in the cache; if there are at least two write operation requests in the data access request, then the cache The data in the storage is flashed to the storage device in time sequence.
优选地，若数据访问请求中存在至少两个写操作请求，则将缓存中数据按时序下刷至存储设备中，进一步包括：在存储设备中经过写操作后的数据下刷至各个缓存，并通过缓存中数据执行各访问操作。Preferably, if there are at least two write operation requests in the data access request, then flushing the data in the cache to the storage device in time sequence, further comprising: flushing the data after the write operation in the storage device to each cache, and Each access operation is performed through the data in the cache.
优选地，若数据访问请求中均是读操作请求，则将该数据放入缓存中并通过缓存中数据执行各访问操作。Preferably, if all of the data access requests are read operation requests, the data is put into the cache and each access operation is performed through the data in the cache.
优选地，若数据访问请求中存在一个写操作请求，则将该数据放入发起写操作请求的客户端缓存中，该客户端对缓存中数据执行各访问操作。Preferably, if there is a write operation request in the data access request, the data is put into the cache of the client that initiated the write operation request, and the client performs various access operations on the data in the cache.
本发明还提供一种多客户端并发数据读写加速装置，包括：The present invention also provides a multi-client concurrent data reading and writing acceleration device, including:
访问请求信息获取模块，用于获取数据访问请求信息；An access request information acquisition module, configured to acquire data access request information;
访问请求执行模块，用于对任一数据，若数据访问请求中存在至多一个写操作请求，则将该数据放入缓存中并通过缓存中数据执行各访问操作；若数据访问请求中存在至少两个写操作请求，则将缓存中数据按时序下刷至存储设备中，在存储设备中经过写操作后的数据下刷至各个缓存，并通过缓存中数据执行各访问操作。The access request execution module is used for any data, if there is at most one write operation request in the data access request, put the data into the cache and perform each access operation through the data in the cache; if there are at least two write operations in the data access request If there is a write operation request, the data in the cache is flushed to the storage device in time sequence, and the data after the write operation in the storage device is flushed to each cache, and each access operation is performed through the data in the cache.
与现有技术相比，本发明具有以下优点：Compared with the prior art, the present invention has the following advantages:
本发明通过在多客户端加入缓存临时占有机制，即多个客户端对同一个数据进行访问时，如果除了某一客户端以外的其他客户端没有对该数据进行写操作请求，这一客户端就可以临时占有该数据，放在缓存中，这一客户端读写该数据只需要读写缓存中的数据；如果除了某一客户端以外的其他客户端，有任一客户端对数据进行写操作请求，上述某一客户端将经过写操作后的数据刷写到存储设备，则其他客户端的缓存进行数据更新，使得同一数据在多个客户端保持一致。通过以上方法，同一数据在多个客户端缓存中能够保持一致；多个客户端从各自的缓存中对数据进行并发读写，提升了多客户端并发数据读写的速度。The present invention adds a cache temporary possession mechanism to multiple clients, that is, when multiple clients access the same data, if other clients except a certain client do not request to write the data, the client will You can temporarily occupy the data and put it in the cache. This client only needs to read and write the data in the cache; if any client other than a certain client writes the data For an operation request, one of the above-mentioned clients will flush the data after the write operation to the storage device, and then the caches of other clients will update the data so that the same data is consistent across multiple clients. Through the above method, the same data can be kept consistent in multiple client caches; multiple clients read and write data concurrently from their respective caches, which improves the speed of concurrent data read and write by multiple clients.
附图说明Description of drawings
图1是本发明一种多客户端并发数据读写加速方法的流程示意图；Fig. 1 is a schematic flow diagram of a multi-client concurrent data reading and writing acceleration method of the present invention;
图2是本发明一种多客户端并发数据读写加速装置的结构示意图。FIG. 2 is a schematic structural diagram of a multi-client concurrent data reading and writing acceleration device according to the present invention.
具体实施方式detailed description
下面结合附图和实施例，对本发明的具体实施方式作进一步详细描述：Below in conjunction with accompanying drawing and embodiment, the specific embodiment of the present invention is described in further detail:
本发明提供一种多客户端并发数据读写加速方法，包括以下步骤：The present invention provides a multi-client concurrent data reading and writing acceleration method, comprising the following steps:
获取数据访问请求信息；Obtain data access request information;
任一数据，若数据访问请求中存在至多一个写操作请求，则将该数据放入缓存中并通过缓存中数据执行各访问操作；若数据访问请求中存在至少两个写操作请求，则将缓存中数据按时序下刷至存储设备中。For any data, if there is at most one write operation request in the data access request, put the data into the cache and perform each access operation through the data in the cache; if there are at least two write operation requests in the data access request, then the cache The data in the storage is flashed to the storage device in time sequence.
请参考图1，图1为本发明实施例所提供的多客户端并发数据读写加速方法的流程示意图；本实施例提供一种多客户端并发数据读写加速方法，包括以下步骤：Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a multi-client concurrent data reading and writing acceleration method provided by an embodiment of the present invention; this embodiment provides a multi-client concurrent data reading and writing acceleration method, including the following steps:
步骤S101，对多个客户端划分缓存，每个客户端对应至少一个缓存；Step S101, divide caches for multiple clients, each client corresponds to at least one cache;
在本实施例中，每个客户端对应一个缓存。In this embodiment, each client corresponds to a cache.
作为一种实施方式，每个客户端对应两个或两个以上缓存。As an implementation manner, each client corresponds to two or more caches.
步骤S102，多个客户端并发访问同一个数据；Step S102, multiple clients concurrently access the same data;
步骤S103，存储设备获取数据访问请求信息；Step S103, the storage device acquires data access request information;
步骤S104，通过缓存中数据执行各访问操作。In step S104, each access operation is performed through the data in the cache.
本实施例中，多个客户端的数据访问请求分为以下三种情况：In this embodiment, the data access requests of multiple clients are divided into the following three situations:
情况一，当多个客户端的数据访问请求均是读操作请求，每个客户端可以临时占有该数据，将该数据放入缓存中，客户端可以直接从对应的缓存中对该数据进行读操作，不需要进行存储设备的读操作。Case 1, when the data access requests of multiple clients are read operation requests, each client can temporarily occupy the data, put the data in the cache, and the client can directly read the data from the corresponding cache , no read operation of the storage device is required.
情况二，当除了某一客户端，例如客户端A，以外的其他客户端没有对该数据的写操作请求，客户端A就可以临时占有该数据，放入缓存中,客户端A读写该数据只需要读写缓存中的数据，不涉及存储设备数据的下刷。Situation 2, when other clients except a certain client, such as client A, do not have a write operation request for the data, client A can temporarily occupy the data and put it in the cache, and client A reads and writes the data. The data only needs to read and write the data in the cache, and does not involve the flashing of the storage device data.
情况三，当除了某一客户端，例如客户端A，以外的任一客户端，例如客户端B，对该数据进行写操作，客户端A先将经过写操作后的数据刷写到存储设备中，其他客户端的缓存根据存储设备中更改后的数据进行数据更新，然后客户端B再对缓存中更改后的数据进行写操作。Case 3, when any client other than a certain client, such as client A, such as client B, writes the data, client A first flushes the data after the write operation to the storage device , the caches of other clients update data according to the changed data in the storage device, and then client B writes the changed data in the cache.
本发明通过将数据存储在每个客户端的缓存中，而且只要有数据发生变化，每个客户端的缓存就会进行数据更新，使同一数据在每个客户端保持一致，以便多个客户端可以从对应的缓存中对数据进行读写操作，提升了多客户端并发数据IO读写速度。The present invention stores the data in the cache of each client, and as long as there is a change in the data, the cache of each client will update the data, so that the same data can be kept consistent in each client, so that multiple clients can read from The data is read and written in the corresponding cache, which improves the concurrent data IO read and write speed of multiple clients.
本实施例还提供一种多客户端并发数据读写加速装置，如图2所示，包括：This embodiment also provides a multi-client concurrent data reading and writing acceleration device, as shown in Figure 2, including:
访问请求信息获取模块21，用于获取数据访问请求信息；An access request information acquisition module 21, configured to acquire data access request information;
访问请求执行模块22，用于对任一数据，若数据访问请求中存在至多一个写操作请求，则将该数据放入缓存中并通过缓存中数据执行各访问操作；若数据访问请求中存在至少两个写操作请求，则将缓存中数据按时序下刷至存储设备中，在存储设备中经过写操作后的数据下刷至各个缓存，并通过缓存中数据执行各访问操作。Access request execution module 22, for any data, if there is at most one write operation request in the data access request, then put the data into the cache and perform each access operation through the data in the cache; if there is at least one write operation request in the data access request For two write operation requests, the data in the cache is flushed to the storage device in time sequence, the data after the write operation in the storage device is flushed to each cache, and each access operation is performed through the data in the cache.
以上所示仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。What is shown above is only a preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, some improvements and modifications can also be made without departing from the principles of the present invention. It should be regarded as the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611240006.1ACN106599301A (en) | 2016-12-29 | 2016-12-29 | Multi-client concurrent data read-write accelerating method and device |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611240006.1ACN106599301A (en) | 2016-12-29 | 2016-12-29 | Multi-client concurrent data read-write accelerating method and device |
| Publication Number | Publication Date |
|---|---|
| CN106599301Atrue CN106599301A (en) | 2017-04-26 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611240006.1APendingCN106599301A (en) | 2016-12-29 | 2016-12-29 | Multi-client concurrent data read-write accelerating method and device |
| Country | Link |
|---|---|
| CN (1) | CN106599301A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291404A (en)* | 2017-08-02 | 2017-10-24 | 郑州云海信息技术有限公司 | The lower brush method and system of one kind caching |
| CN107608627A (en)* | 2017-08-21 | 2018-01-19 | 云宏信息科技股份有限公司 | A kind of remote data classification storage method, electronic equipment and storage medium |
| CN108595109A (en)* | 2017-12-29 | 2018-09-28 | 郑州云海信息技术有限公司 | A method of the multi-client based on Ceph distributed memory systems is write |
| CN110691219A (en)* | 2019-09-11 | 2020-01-14 | 视联动力信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2133783A1 (en)* | 2008-06-09 | 2009-12-16 | Deutsche Thomson OHG | Data storage server, stored set of instructions, and method of controlling data access |
| CN103294786A (en)* | 2013-05-17 | 2013-09-11 | 华中科技大学 | Metadata organization and management method and system of distributed file system |
| CN104580437A (en)* | 2014-12-30 | 2015-04-29 | 创新科存储技术（深圳）有限公司 | Cloud storage client and high-efficiency data access method thereof |
| CN105224255A (en)* | 2015-10-14 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | A kind of storage file management method and device |
| CN105701158A (en)* | 2015-12-30 | 2016-06-22 | 浪潮(北京)电子信息产业有限公司 | File system read-write optimization method and framework |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2133783A1 (en)* | 2008-06-09 | 2009-12-16 | Deutsche Thomson OHG | Data storage server, stored set of instructions, and method of controlling data access |
| CN103294786A (en)* | 2013-05-17 | 2013-09-11 | 华中科技大学 | Metadata organization and management method and system of distributed file system |
| CN104580437A (en)* | 2014-12-30 | 2015-04-29 | 创新科存储技术（深圳）有限公司 | Cloud storage client and high-efficiency data access method thereof |
| CN105224255A (en)* | 2015-10-14 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | A kind of storage file management method and device |
| CN105701158A (en)* | 2015-12-30 | 2016-06-22 | 浪潮(北京)电子信息产业有限公司 | File system read-write optimization method and framework |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291404A (en)* | 2017-08-02 | 2017-10-24 | 郑州云海信息技术有限公司 | The lower brush method and system of one kind caching |
| CN107291404B (en)* | 2017-08-02 | 2020-08-21 | 苏州浪潮智能科技有限公司 | Method and system for flushing cache |
| CN107608627A (en)* | 2017-08-21 | 2018-01-19 | 云宏信息科技股份有限公司 | A kind of remote data classification storage method, electronic equipment and storage medium |
| CN107608627B (en)* | 2017-08-21 | 2020-10-02 | 云宏信息科技股份有限公司 | Remote data hierarchical storage method, electronic equipment and storage medium |
| CN108595109A (en)* | 2017-12-29 | 2018-09-28 | 郑州云海信息技术有限公司 | A method of the multi-client based on Ceph distributed memory systems is write |
| CN108595109B (en)* | 2017-12-29 | 2021-06-29 | 郑州云海信息技术有限公司 | A method of multi-client writing based on Ceph distributed storage system |
| CN110691219A (en)* | 2019-09-11 | 2020-01-14 | 视联动力信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| US9996542B2 (en) | Cache management in a computerized system | |
| TWI335512B (en) | Technique for using memory attributes | |
| KR102398912B1 (en) | Method and processor for processing data | |
| US10073649B2 (en) | Storing metadata | |
| US11210228B2 (en) | Method, device and computer program product for cache management | |
| CN105224255B (en) | A kind of storage file management method and device | |
| CN105446888B (en) | The method of mobile data, controller and storage system between storage device | |
| CN103412822B (en) | Operation Nonvolatile memory and the method and relevant apparatus of data manipulation | |
| CN105224444A (en) | Daily record generation method and device | |
| CN109791541B (en) | Log serial number generation method and device and readable storage medium | |
| US9836223B2 (en) | Changing storage volume ownership using cache memory | |
| CN106599301A (en) | Multi-client concurrent data read-write accelerating method and device | |
| US20190057030A1 (en) | Method and device for cache management | |
| CN107888687B (en) | Proxy client storage acceleration method and system based on distributed storage system | |
| CN112346659A (en) | Storage method, equipment and storage medium for distributed object storage metadata | |
| CN114785662B (en) | A storage management method, device, equipment, and machine-readable storage medium | |
| CN106527974A (en) | Data writing method, device and system | |
| US10901914B2 (en) | Method for writing multiple copies into storage device, and storage device | |
| CN115951844B (en) | File lock management method, device and medium of distributed file system | |
| CN111488293A (en) | Access method and device for data visitor directory in multi-core system | |
| US10073851B2 (en) | Fast new file creation cache | |
| CN107451070A (en) | The processing method and server of a kind of data | |
| CN107003932B (en) | Cache directory processing method and directory controller of multi-core processor system | |
| JP2018536230A (en) | Cache access | |
| CN107102898B (en) | Memory management and data structure construction method and device based on NUMA (non Uniform memory Access) architecture |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20170426 |