技术领域technical field
本发明涉及一种混合云结构中热度数据缓存的云存储方法。The invention relates to a cloud storage method for thermal data cache in a hybrid cloud structure.
背景技术Background technique
随着云存储、云计算的快速发展，国内外都涌现出了各式各样的云存储服务。公有云和私有云是云存储服务的两种方式。公有云可以通过Internet使用。企业保护自己的数据安全而构建自己的私有云服务。企业会将安全要求高的数据放在私有云，将一些公共数据放在公有云，这就是通常所说的混合云。With the rapid development of cloud storage and cloud computing, various cloud storage services have emerged at home and abroad. Public cloud and private cloud are two ways of cloud storage services. Public clouds are available over the Internet. Enterprises build their own private cloud services to protect their data security. Enterprises will place data with high security requirements in private clouds and some public data in public clouds, which is commonly referred to as hybrid cloud.
传统的混合云结构中，用户要访问数据时需要先访问服务端的元数据数据库，该数据库中存放着数据文件生成的元数据。因此，得到了元数据就可以获取到文件存放的位置。当访问的文件在私有云时，用户就从私有云获取；访问文件在公有云时，用户根据系统算出速度最快的下载方案进行下载。In the traditional hybrid cloud structure, when users want to access data, they first need to access the metadata database on the server side, which stores metadata generated by data files. Therefore, once the metadata is obtained, the location where the file is stored can be obtained. When the accessed file is in the private cloud, the user obtains it from the private cloud; when the accessed file is in the public cloud, the user downloads it according to the fastest download scheme calculated by the system.
在实际应用中，由于私有云和公有云在不同的网络上，而且系统在访问文档前并不会自主选择访问速度与成本最优的下载数据的方式，所以如果公有云上存放着内网用户经常访问的数据，则内网用户访问这部分数据会消耗大量不必要的时间。由此，开发一种更加优化的存储方式便成为一个重要的需求，特别是数据存储量极大的现代企业。In practical applications, since the private cloud and the public cloud are on different networks, and the system does not independently choose the way to download data with the best access speed and cost before accessing the document, if there are intranet users stored on the public cloud Frequently accessed data, it will consume a lot of unnecessary time for intranet users to access this part of data. Therefore, developing a more optimized storage method has become an important requirement, especially for modern enterprises with a huge amount of data storage.
发明内容Contents of the invention
有鉴于此，本发明的目的在于提供一种混合云结构中热度数据缓存的云存储方法，以便用户快速高效地访问下载公有云中的常用数据。In view of this, the object of the present invention is to provide a cloud storage method for thermal data caching in a hybrid cloud structure, so that users can quickly and efficiently access and download common data in public clouds.
为实现上述目的，本发明采用如下技术方案：一种混合云结构中热度数据缓存的云存储方法，其特征在于，包括以下步骤：In order to achieve the above object, the present invention adopts the following technical solution: a cloud storage method for thermal data cache in a hybrid cloud structure, characterized in that it includes the following steps:
步骤S1：云存储代理系统在私有云中开辟了一个固定大小的数据交换区，用于存放公有云上内网用户访问的数据；Step S1: The cloud storage proxy system opens up a fixed-size data exchange area in the private cloud for storing data accessed by intranet users on the public cloud;
步骤S2：在所述数据交换区中建立一张表，用于记录所述数据交换区中的文件被访问的频率即热度，并按照热度由高到低来排列文件，同样，公有云中数据也按热度排序；Step S2: Create a table in the data exchange area to record the frequency of access to the files in the data exchange area, that is, the popularity, and arrange the files according to the popularity from high to low. Similarly, the data in the public cloud Also sorted by popularity;
步骤S3：将公有云上的部分数据按热度由高到低依次拷贝到私有云的数据交换区中；Step S3: Copy part of the data on the public cloud to the data exchange area of the private cloud in descending order of popularity;
步骤S4：按更新规则定期对数据交换区里的数据进行比较和替换，以此达到混合云环境中数据互动的目的；Step S4: regularly compare and replace the data in the data exchange area according to the update rules, so as to achieve the purpose of data interaction in the hybrid cloud environment;
步骤S5：当用户申请访问或下载文件时，系统自动根据不同网络区域用户采取不同搜索数据的方式来访问文件。Step S5: When a user applies for accessing or downloading a file, the system automatically accesses the file in different ways of searching data according to different network areas.
进一步的，所述步骤S4中，先计算获得私有云的数据交换区的中间热度，即取数据交换区中数据热度由大到小顺序排列的中位数；然后每隔t分钟，公有云最高数据热度与私有云的数据交换区的中间热度进行比较，如果公有云数据的最高热度超过所述中间热度，则对私有云中的数据进行维护，将公有云中数据热度高于交换区中间热度的数据替换进所述数据交换区。Further, in the step S4, first calculate and obtain the median heat of the data exchange area of the private cloud, that is, take the median of the data heat in the data exchange area from large to small; then every t minutes, the public cloud has the highest The data heat is compared with the middle heat of the data exchange area of the private cloud. If the highest heat of the public cloud data exceeds the middle heat, the data in the private cloud is maintained, and the data heat in the public cloud is higher than the middle heat of the exchange area. The data is replaced into the data exchange area.
进一步的，所述数据交换区中数据热度的定义为私有云用户下载次数与该数据在数据交换区中存放天数的比值。Further, the data popularity in the data exchange area is defined as the ratio of the number of downloads by private cloud users to the number of days the data is stored in the data exchange area.
进一步的，将私有云的数据交换区中数据热度低于中间热度的所有数据所占空间求和，获得私有云的数据交换区的中间热度以下的数据所占的空间总和设为S，即待替换空间大小；公有云中数据热度大于私有云的数据交换区中间热度的数据所占空间总量求和，其结果设为m，即用于替换的候选数据大小总和。Further, the sum of the space occupied by all the data whose data heat is lower than the middle heat in the data exchange area of the private cloud is summed, and the sum of the space occupied by the data below the middle heat in the data exchange area of the private cloud is set as S, that is, to be The size of the replacement space; the sum of the total space occupied by the data whose data heat in the public cloud is greater than the middle heat of the data exchange area of the private cloud, and the result is set to m, which is the sum of the size of the candidate data for replacement.
进一步的，比较S与m的大小；如果m大于或等于S，则删除私有云的数据交换区中热度从低到高数据量总和等于S的全部数据，然后，将公有云中热度从高到低数据量总和小于或等于S的数据拷贝进来；如果m小于S，删除私有云数据交换区中热度从低到高数据量总和等于或刚好大于m的全部数据，然后，将公有云中热度从高到低数据量总和为m的数据拷贝过来。Further, compare the size of S and m; if m is greater than or equal to S, delete all the data in the data exchange area of the private cloud whose heat is from low to high and the sum of the data volume is equal to S, and then change the heat from high to high in the public cloud The data with the sum of the low data volume less than or equal to S is copied in; if m is less than S, delete all the data in the private cloud data exchange area from low to high and the sum of the data volume is equal to or just greater than m, and then, the heat in the public cloud is changed from Copy the data whose total amount of data from high to low is m.
进一步的，所述步骤S5中，根据用户的请求，系统先判断用户是私有云网络区域用户还是公有云网络区域用户；若是私有云网络区域用户，则系统先在私有云的非数据交换区内匹配是否有用户请求的文件，若没有匹配到，则到数据交换区中查找，若仍没找到，最后再到公有云中查找；对于公有云用户来说，系统则直接在公有云上查询所需数据。Further, in the step S5, according to the user's request, the system first judges whether the user is a user in a private cloud network area or a user in a public cloud network area; Match whether there is a file requested by the user. If there is no match, it will search in the data exchange area. If it is still not found, it will finally search in the public cloud; Data required.
本发明与现有技术相比具有以下有益效果：本发明在私有云中开辟一个数据交换区，存放公有云上的常用数据，并依据数据热度的变化对交换区的数据进行实时动态的更新，加快内网用户访问原先公有云上常用数据的速度，降低了公有云上常用数据丢失的风险，同时减少带宽和流量的访问成本，以便用户快递高效的访问下载公有云中的常用数据。Compared with the prior art, the present invention has the following beneficial effects: the present invention opens up a data exchange area in the private cloud, stores commonly used data on the public cloud, and performs real-time dynamic update of the data in the exchange area according to the change of data heat, Accelerates the speed for intranet users to access common data on the original public cloud, reduces the risk of common data loss on the public cloud, and reduces access costs for bandwidth and traffic, so that users can quickly and efficiently access and download common data in the public cloud.
附图说明Description of drawings
图1是本发明的混合云中数据动态存储流程图。Fig. 1 is a flow chart of dynamic storage of data in a hybrid cloud according to the present invention.
图2是本发明的数据交换具体流程图。Fig. 2 is a specific flow chart of data exchange in the present invention.
图3是本发明一实施例的不同网络区域用于访问数据方式的示意图。Fig. 3 is a schematic diagram of the ways in which different network areas are used to access data according to an embodiment of the present invention.
具体实施方式Detailed ways
下面结合附图及实施例对本发明做进一步说明。The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
本发明提供一种混合云结构中热度数据缓存的云存储方法，其特征在于，包括以下步骤：The present invention provides a cloud storage method for temperature data caching in a hybrid cloud structure, which is characterized in that it comprises the following steps:
步骤S1：基于云存储代理系统（私有云和公有云的空间大小各为1000G），在私有云中开辟了一个固定大小的数据交换区，用于存放公有云上内网用户访问的数据，开辟的数据交换区的容量大小定为占私有云固定阈值大小的容量，于本实施例中，阈值为30%；Step S1: Based on the cloud storage proxy system (the private cloud and the public cloud each have a space size of 1000G), a fixed-size data exchange area is opened in the private cloud to store data accessed by intranet users on the public cloud, and open up The capacity of the data exchange area is defined as the capacity that accounts for the fixed threshold size of the private cloud. In this embodiment, the threshold is 30%;
步骤S2：在所述数据交换区中建立一张表，用于记录所述数据交换区中的文件被访问的频率（下文称热度），并按照热度由高到低来排列文件，同样，公有云中数据也按热度排序；假设首次私有云中30%的容量可容纳10份数据量，数据热度不等，所占空间百分比不同，情况如下表1（交换区数据状况表）所示：Step S2: Create a table in the data exchange area to record the frequency (hereinafter referred to as popularity) of the files in the data exchange area, and arrange the files according to the popularity from high to low. Similarly, public The data in the cloud is also sorted by popularity; assuming that 30% of the capacity in the private cloud for the first time can accommodate 10 copies of data, the data popularity varies, and the percentage of space occupied is different. The situation is shown in Table 1 (exchange area data status table):
表1：Table 1:
将其热度由高到低进行排列得到下表2（交换区热度顺序数据状况表）：Arrange their heat from high to low to get the following table 2 (table of heat order data in the exchange area):
表2：Table 2:
步骤S3：请参照图1，将公有云上的部分数据按热度由高到低依次拷贝到私有云的数据交换区中；特别的，若该次访问为第一次访问数据，则以下载频率作为依据判断数据热度，下载频率越高的数据视为热度越高的数据，拷贝公有云热点数据到所述数据交换区，由于是第一次将公有云数据存入数据交换区，可将数据交换区填满。Step S3: Please refer to Figure 1, and copy some data on the public cloud to the data exchange area of the private cloud in descending order of popularity; especially, if this access is the first access to data, the download frequency As a basis for judging data popularity, data with higher download frequency is regarded as data with higher popularity. Copy public cloud hotspot data to the data exchange area. Since it is the first time that public cloud data is stored in the data exchange area, the data can be The swap area is full.
步骤S4：请继续参照图1，按更新规则定期对数据交换区里的数据进行比较和替换，以此达到混合云环境中数据互动的目的；Step S4: Please continue to refer to Figure 1, and regularly compare and replace the data in the data exchange area according to the update rules, so as to achieve the purpose of data interaction in the hybrid cloud environment;
将数据交换区中数据热度的定义为私有云用户下载次数与该数据在数据交换区中存放天数的比值。The heat of data in the data exchange area is defined as the ratio of the number of downloads by private cloud users to the number of days the data is stored in the data exchange area.
先计算获得私有云的数据交换区的中间热度，即取数据交换区中数据热度由大到小顺序排列的中位数设为a；然后每隔t分钟，公有云最高数据热度与私有云的数据交换区的中间热度进行比较，如果公有云数据的最高热度超过所述中间热度，则对私有云中的数据进行维护，将公有云中数据热度高于交换区中间热度的数据替换进所述数据交换区。First calculate the median heat of the data exchange area of the private cloud, that is, take the median of the data heat in the data exchange area in descending order as a; then every t minutes, the highest data heat of the public cloud and the private cloud Compared with the middle heat of the data exchange area, if the highest heat of the public cloud data exceeds the middle heat, the data in the private cloud is maintained, and the data in the public cloud is replaced by the data whose heat is higher than the middle heat of the exchange area. data exchange area.
下表3和下表4分别是一实施例的公有云数据状况表及公有云热度顺序数据状况表The following table 3 and the following table 4 are the public cloud data status table and the public cloud heat order data status table of an embodiment respectively
表3：table 3:
表4：Table 4:
请参照图2，将私有云的数据交换区中数据热度低于中间热度的所有数据所占空间求和，获得私有云的数据交换区的中间热度以下的数据所占的空间总和设为S，即待替换空间大小；由表2可得出中间热度为4，中间热度以下所占热度为数据10，数据2，数据4和数据1，所占空间总和为1000G*44%=440G。公有云中数据热度大于私有云的数据交换区中间热度的数据所占空间总量求和，其结果设为m，即用于替换的候选数据大小总和。Please refer to Figure 2, sum the space occupied by all the data whose data heat is lower than the middle heat in the data exchange area of the private cloud, and obtain the sum of the space occupied by the data below the middle heat in the data exchange area of the private cloud as S, That is, the size of the space to be replaced; from Table 2, it can be concluded that the middle heat is 4, and the heat below the middle heat is data 10, data 2, data 4 and data 1, and the total occupied space is 1000G*44%=440G. The sum of the total space occupied by the data whose data heat in the public cloud is greater than the middle heat of the data exchange area of the private cloud is summed, and the result is set to m, which is the sum of the candidate data sizes for replacement.
在此，公有云数据热度定义为：公有云用户访问量与首次访问该数据至本次更新时所经历时间（天）的比值。Here, the popularity of public cloud data is defined as the ratio of public cloud user visits to the time (days) elapsed from the first access to the data to this update.
将公有云中热度超过私有云数据交换区中间热度的数据拷贝过来。拷贝的过程中采用以下方案：Copy the data in the public cloud whose heat exceeds the middle heat of the private cloud data exchange area. The following schemes are used in the process of copying:
计算公有云中热度大于a的数据量总和，设为m。在步骤5表4中则表示为数据3, 数据6, 数据10, 数据8, 数据1, 数据2, 数据7等数据所占大小的容量之和，即m=1000G*80%=800G。Calculate the sum of the amount of data whose popularity is greater than a in the public cloud, and set it to m. In step 5 and table 4, it is expressed as the sum of the capacity of data 3, data 6, data 10, data 8, data 1, data 2, data 7, etc., that is, m=1000G*80%=800G.
·如果m大于S，剔除私有云交换区中热度从低到高数据量总和等于s的全部数据。然后，拷贝公有云中热度从高到低数据量总和等于或小于s的数据过来。·If m is greater than S, delete all the data in the private cloud exchange area whose heat from low to high data volume sum is equal to s. Then, copy the data in the public cloud whose heat is from high to low and the total amount of data is equal to or less than s.
·如果m小于S，剔除私有云交换区中热度从低到高数据量总和大于或等于m的全部数据。然后，将拷贝公有云中热度从高到低数据量总和为m的数据过来。具体过程可参考附图图1。·If m is less than S, remove all data in the private cloud exchange area whose heat from low to high data volume sum is greater than or equal to m. Then, the data in the public cloud with a total data volume of m from high to low will be copied. The specific process can refer to Figure 1 of the attached drawing.
本例为m>S，则按已有规则，将交换区中的数据10，数据2，数据4，数据1删除，而由公有云中的数据3，数据6，数据10进行替代。In this example, m>S, then according to the existing rules, delete data 10, data 2, data 4, and data 1 in the exchange area, and replace them with data 3, data 6, and data 10 in the public cloud.
步骤S5：当用户申请访问或下载文件时，系统自动根据不同网络区域用户采取不同搜索数据的方式来访问文件。Step S5: When a user applies for accessing or downloading a file, the system automatically accesses the file in different ways of searching data according to different network areas.
进一步的，所述步骤S5中，在用户登录的系统中设置获取用于IP功能，根据用户的请求的IP网段，系统先判断用户是私有云网络区域用户还是公有云网络区域用户；若是私有云网络区域用户，则系统先在私有云的非数据交换区内匹配是否有用户请求的文件，若没有匹配到，则到数据交换区中查找，若仍没找到，最后再到公有云中查找；对于公有云用户来说，系统则直接在公有云上查询所需数据，具体过程可参考附图3。Further, in the step S5, the IP function is set in the system where the user logs in, and according to the IP network segment requested by the user, the system first judges whether the user is a user in a private cloud network area or a user in a public cloud network area; For users in the cloud network area, the system will first check whether there is a file requested by the user in the non-data exchange area of the private cloud. If there is no match, it will search in the data exchange area. If it is still not found, it will finally search in the public cloud ; For public cloud users, the system directly queries the required data on the public cloud, the specific process can refer to Figure 3.
以上所述仅为本发明的较佳实施例，凡依本发明申请专利范围所做的均等变化与修饰，皆应属本发明的涵盖范围。The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610519867.7ACN106210015B (en) | 2016-07-05 | 2016-07-05 | A cloud storage method for temperature data cache in hybrid cloud structure |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610519867.7ACN106210015B (en) | 2016-07-05 | 2016-07-05 | A cloud storage method for temperature data cache in hybrid cloud structure |
| Publication Number | Publication Date |
|---|---|
| CN106210015A CN106210015A (en) | 2016-12-07 |
| CN106210015Btrue CN106210015B (en) | 2019-12-31 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610519867.7AExpired - Fee RelatedCN106210015B (en) | 2016-07-05 | 2016-07-05 | A cloud storage method for temperature data cache in hybrid cloud structure |
| Country | Link |
|---|---|
| CN (1) | CN106210015B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107291393A (en)* | 2017-06-23 | 2017-10-24 | 郑州云海信息技术有限公司 | A kind of caching method and device based on mixing cloud storage |
| CN108418871B (en)* | 2018-02-09 | 2022-02-11 | 国家电网公司 | Cloud storage performance optimization method and system |
| CN109639778A (en)* | 2018-11-29 | 2019-04-16 | 威富通科技有限公司 | A kind of data transmission method, private clound and server |
| CN110929317B (en)* | 2019-10-17 | 2023-04-14 | 广联达科技股份有限公司 | Method, system and computer readable storage medium for automatically complementing user component modeling information |
| CN113487278B (en)* | 2021-07-02 | 2023-05-09 | 深圳市顿泽慧科技有限公司 | An enterprise collaborative office system based on the Internet of Things |
| CN113553306B (en)* | 2021-07-27 | 2023-07-21 | 重庆紫光华山智安科技有限公司 | Data processing method and data storage management system |
| CN113742304B (en)* | 2021-11-08 | 2022-02-15 | 杭州雅观科技有限公司 | Data storage method of hybrid cloud |
| CN117171794B (en)* | 2023-09-08 | 2024-07-05 | 之江实验室 | Mixed cloud data secure storage system and method based on resource classification |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103067406A (en)* | 2013-01-14 | 2013-04-24 | 暨南大学 | Access control system and access control method between public cloud and private cloud |
| CN103095805A (en)* | 2012-12-20 | 2013-05-08 | 江苏辰云信息科技有限公司 | Cloud storage system of data intelligent and decentralized management |
| US8732182B2 (en)* | 2004-12-02 | 2014-05-20 | Desktopsites Inc. | System and method for launching a resource in a network |
| CN104462240A (en)* | 2014-11-18 | 2015-03-25 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing hierarchical storage and management in cloud storage |
| CN105447062A (en)* | 2014-09-30 | 2016-03-30 | 中国电信股份有限公司 | Hot spot data identification method and device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105376218B (en)* | 2015-10-21 | 2020-11-13 | 上海思华科技股份有限公司 | Streaming media system and method for quickly responding to user request |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8732182B2 (en)* | 2004-12-02 | 2014-05-20 | Desktopsites Inc. | System and method for launching a resource in a network |
| CN103095805A (en)* | 2012-12-20 | 2013-05-08 | 江苏辰云信息科技有限公司 | Cloud storage system of data intelligent and decentralized management |
| CN103067406A (en)* | 2013-01-14 | 2013-04-24 | 暨南大学 | Access control system and access control method between public cloud and private cloud |
| CN105447062A (en)* | 2014-09-30 | 2016-03-30 | 中国电信股份有限公司 | Hot spot data identification method and device |
| CN104462240A (en)* | 2014-11-18 | 2015-03-25 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing hierarchical storage and management in cloud storage |
| Publication number | Publication date |
|---|---|
| CN106210015A (en) | 2016-12-07 |
| Publication | Publication Date | Title |
|---|---|---|
| CN106210015B (en) | A cloud storage method for temperature data cache in hybrid cloud structure | |
| JP6584575B2 (en) | Distributed cache of graph data | |
| US11561930B2 (en) | Independent evictions from datastore accelerator fleet nodes | |
| US10467188B2 (en) | In-line policy management with multi-level object handle | |
| CN103139301B (en) | Be applied to the internet access accelerated method in content distribution network system and device | |
| US9043372B2 (en) | Metadata subsystem for a distributed object store in a network storage system | |
| CN106484820B (en) | A renaming method, access method and device | |
| US20180067657A1 (en) | Data storage space recovery | |
| US20040030731A1 (en) | System and method for accessing files in a network | |
| CN103150394A (en) | Distributed file system metadata management method facing to high-performance calculation | |
| US20180145983A1 (en) | Distributed data storage system using a common manifest for storing and accessing versions of an object | |
| CN102984280A (en) | Data backup system and method for social cloud storage network application | |
| CN104317669A (en) | Cloud heterogeneous storage system and data copy management method thereof | |
| CN104426838B (en) | A kind of internet buffer scheduling method and system | |
| JP2018049653A (en) | Cache management | |
| CN105704218A (en) | Data storage and management method in cloud computing platform | |
| CN105915619B (en) | Take the cyberspace information service high-performance memory cache method of access temperature into account | |
| CN109388651B (en) | A data processing method and device | |
| Gopalakrishnan et al. | Cache me if you can: capacitated selfish replication games | |
| CN104424316B (en) | A kind of date storage method, data query method, relevant apparatus and system | |
| Xu et al. | Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems | |
| Cha et al. | Adaptive metadata rebalance in exascale file system | |
| CN109495525B (en) | Network component, method of resolving content identification, and computer-readable storage medium | |
| CN114238420B (en) | Method, apparatus and electronic device for metadata usage based on multi-tenant architecture | |
| Anitha et al. | CBF: Metadata management in cloud computing |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20191231 |