技术领域technical field
本发明涉及文件访问领域，特别涉及一种小文件和大文件的存储及访问方法。The invention relates to the field of file access, in particular to a method for storing and accessing small files and large files.
背景技术Background technique
随着社会的发展，人们日常生活和工作中产生了大量的数据，这些数据以文本、文档、Exce前缀等小文件为主，但也包括部分大文件如视频文件、高清图片文件等。根据习惯来看，人们访问最多的通常是小文件和部分大文件。因此如何高效地存储这两类文件并使用户能快速访问这两类文件成为当前文件访问领域存在的难题。With the development of society, a large amount of data is generated in people's daily life and work. These data are mainly small files such as text, documents, and Excel prefixes, but also include some large files such as video files and high-definition picture files. According to the habit, people usually access the most small files and some large files. Therefore, how to efficiently store these two types of files and enable users to quickly access these two types of files has become a difficult problem in the current file access field.
当前的文件访问方式研究集中在海量小文件存储和快速访问领域，如淘宝的TFS能提供针对小文件的快速访问，却无法快速访问大文件，又比如HDFS能高效快速访问大文件，却把小文件当作大文件一样存储在数据节点中，这导致了针对小文件的访问速度较慢。实际操作中，用户通常经常访问多个小文件和少量大文件，因此如何将这部分经常访问的大文件与一般大文件区分开来并在确保高效访问小文件的前提下快速访问经常访问的大文件成为目前需要解决的技术问题。Current research on file access methods focuses on the storage of massive small files and fast access. For example, Taobao’s TFS can provide fast access to small files, but cannot quickly access large files. Files are stored in data nodes as if they were large files, which results in slower access for small files. In actual operation, users usually access multiple small files and a small number of large files, so how to distinguish these frequently accessed large files from general large files and quickly access frequently accessed large files under the premise of ensuring efficient access to small files Documentation becomes a technical issue that needs to be resolved now.
发明内容Contents of the invention
本发明的目的在于克服现有技术的缺点与不足，提供一种小文件和大文件的存储方法。The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a storage method for small files and large files.
本发明的第二目的在于提供一种快速的小文件和大文件的访问方法。该访问方法保证既能对小文件的快速访问又能对使用频率较高的大文件实现快速访问。The second object of the present invention is to provide a fast method for accessing small files and large files. The access method ensures fast access to small files and fast access to large files with high frequency of use.
本发明的第一目的通过下述技术方案实现：一种小文件和大文件的存储方法，步骤如下：The first object of the present invention is achieved through the following technical solutions: a storage method for small files and large files, the steps are as follows:
S1、用户通过用户设备创建新文件，根据第一阈值确定所要创建的新文件是大文件还是小文件，并设置文件名，在文件名上添加用于识别大文件和小文件的标识性前缀；S1. The user creates a new file through the user device, determines whether the new file to be created is a large file or a small file according to the first threshold, sets the file name, and adds an identifying prefix for identifying the large file and the small file to the file name;
S2、用户设备将创建新文件请求指令发送给缓存服务器，缓存服务器收到用户设备发送的创建新文件请求指令后，识别所要创建的新文件的文件名标识性前缀，判断所要创建的新文件是大文件还是小文件；S2. The user device sends the command to create a new file to the cache server. After receiving the command to create a new file from the user device, the cache server identifies the file name identifier prefix of the new file to be created, and determines whether the new file to be created is Large files or small files;
若所要创建的新文件为小文件，则在缓存服务器中创建该文件并保存其元数据；If the new file to be created is a small file, create the file in the cache server and save its metadata;
若所要创建的新文件为大文件，则缓存器将用户设备发送的创建新文件请求指令发送给管理服务器；If the new file to be created is a large file, the cache sends the request instruction for creating a new file sent by the user equipment to the management server;
S3、管理服务器收到来自于缓存服务器的创建新文件请求指令后，管理服务器创建该大文件并将其进行分块，得到数据块，并且保存该大文件的元数据；S3. After the management server receives the command to create a new file from the cache server, the management server creates the large file and divides it into blocks, obtains data blocks, and saves the metadata of the large file;
S4、管理服务器将创建的大文件划分的数据块发送给数据服务器进行保存，同时缓存服务器根据大文件请求访问次数确定大文件是否为常用大文件，若是，则从数据服务器端将该大文件下载并保存在缓存服务器中。S4. The management server sends the data blocks of the created large file to the data server for storage, and at the same time, the cache server determines whether the large file is a commonly used large file according to the number of accesses requested by the large file, and if so, downloads the large file from the data server and stored in the cache server.
优选的，在所述步骤S2中，若所要创建的新文件为小文件时，则缓存服务器首先查询缓存服务器当前存储的小文件元数据，判断所要创建的新文件的文件名是否与缓存服务器中当前已有小文件的文件名重名；Preferably, in said step S2, if the new file to be created is a small file, the cache server first inquires the metadata of the small file currently stored by the cache server to determine whether the file name of the new file to be created is the same as that in the cache server The file name of the existing small file is the same;
若是，则缓存服务器发送“文件已存在”信息给用户设备，并且丢弃用户发送的创建新文件请求指令；If so, the cache server sends the "file already exists" message to the user device, and discards the request instruction for creating a new file sent by the user;
若否，在缓存服务器中创建该文件并保存其元数据。If not, create the file and save its metadata in the cache server.
优选的，在所述步骤S3中，若所要创建的新文件为大文件，管理服务器收到来自于缓存服务器的创建新文件请求指令后，判断所要创建的新文件的文件名是否与管理服务器中当前已有大文件的文件名重名；Preferably, in said step S3, if the new file to be created is a large file, after the management server receives the request instruction for creating a new file from the cache server, it judges whether the file name of the new file to be created is the same as that in the management server. The file name of the existing large file is the same;
若是，则管理服务器发送“文件已存在”信息给用户设备，并且丢弃用户发送的创建新文件请求指令；If so, the management server sends the "file already exists" message to the user device, and discards the request instruction for creating a new file sent by the user;
若否，在管理服务器中创建该文件并保存其元数据。If not, create the file and save its metadata in the management server.
优选的，所述第一阈值为1M，当文件的大小超过1M时，则将该文件判断为大文件，否则判断为小文件。Preferably, the first threshold is 1M, and when the size of the file exceeds 1M, the file is judged as a large file, otherwise it is judged as a small file.
优选的，所述大文件和小文件的标识性前缀分别为L和S；即用户设备在创建的大文件的文件名前添加L，用户设备在创建的小文件的文件名前添加S。Preferably, the identifying prefixes of the large file and the small file are L and S respectively; that is, the user equipment adds L before the file name of the created large file, and the user equipment adds S before the file name of the created small file.
优选的，所述缓存服务器包括存储区和缓存区；小文件及小文件元数据存储在缓存服务器的存储区，常用大文件以及常用大文件元数据保存在缓存服务器的缓存区。Preferably, the cache server includes a storage area and a cache area; small files and metadata of small files are stored in the storage area of the cache server, and commonly used large files and metadata of frequently used large files are stored in the cache area of the cache server.
本发明的第二目的通过下述技术方案实现：一种基于上述存储方法实现的小文件和大文件的访问方法，其特征在于，步骤如下：The second object of the present invention is achieved through the following technical solutions: a method for accessing small files and large files based on the above storage method, characterized in that the steps are as follows:
Sa、用户在用户设备中输入文件名，通过用户设备提出访问文件请求，用户设备搜索日志确定用户输入的文件名对应的文件是大文件还是小文件，然后在用户访问请求指令中的文件名中添加大文件或小文件标识性前缀，用户设备发送用户访问请求指令给缓存服务器；Sa, the user inputs the file name in the user equipment, puts forward the access file request through the user equipment, the user equipment search log determines whether the file corresponding to the file name input by the user is a large file or a small file, and then in the file name in the user access request instruction Add a large file or small file identification prefix, and the user device sends a user access request instruction to the cache server;
Sb、缓存服务器接收到用户设备发送的用户访问请求指令，根据用户访问请求指令中文件名的标识性前缀判断用户请求访问的是大文件还是小文件；Sb. The cache server receives the user access request instruction sent by the user equipment, and judges whether the user requests to access a large file or a small file according to the identifying prefix of the file name in the user access request instruction;
若是小文件，则缓存服务器搜索本地小文件元数据，判断缓存服务器中是否存在用户要求访问的小文件；若存在，则缓存服务器引导用户对该小文件进行的访问；否则缓存服务器返回未找到信息指令给用户设备；If it is a small file, the cache server searches the metadata of the local small file to determine whether there is a small file requested by the user in the cache server; if it exists, the cache server guides the user to access the small file; otherwise, the cache server returns the not found information Instructions to user equipment;
若是大文件，则缓存服务器搜索本地常用大文件元数据，判断缓存服务器中是否存在用户要求访问的大文件；若是，则缓存服务器引导用户对该大文件进行的访问；若否，则缓存服务器将用户设备发送的用户访问请求指令发送给管理服务器，进入步骤Sc，同时缓存服务器记录通过其请求访问的该大文件的请求访问次数；If it is a large file, the cache server searches the local common large file metadata to determine whether there is a large file requested by the user in the cache server; if so, the cache server guides the user to access the large file; if not, the cache server will The user access request instruction sent by the user equipment is sent to the management server, and enters step Sc, and the cache server records the number of request visits of the large file accessed through its request;
Sc、管理服务器接收到来自于缓存服务器的用户访问请求指令后，搜索本地大文件元数据，判断管理服务器中是否存在用户要求访问的大文件；若存在，则引导用户对大文件访问；否则返回未找到信息给用户设备。Sc. After the management server receives the user access request instruction from the cache server, it searches the metadata of the local large file to determine whether there is a large file requested by the user in the management server; if it exists, guide the user to access the large file; otherwise return No information was found for the user device.
优选的，所述步骤Sb中，缓存服务器引导用户对小文件进行访问的具体过程如下：Preferably, in the step Sb, the specific process of the cache server guiding the user to access the small file is as follows:
首先缓存服务器判断用户要求访问的小文件是否处于可编辑状态；First, the cache server judges whether the small file requested by the user is editable;
若是，则允许用户对该小文件实施读或写操作；同时在用户对小文件实施读或写操作时，缓存服务器自动锁定该小文件；If so, the user is allowed to perform read or write operations on the small file; at the same time, when the user performs read or write operations on the small file, the cache server automatically locks the small file;
若否，缓存服务器发送“该文件正被其他用户使用”信息给用户设备，并终止用户的访问请求；If not, the cache server sends the information "this file is being used by other users" to the user device, and terminates the user's access request;
所述步骤Sb中，缓存服务器引导用户对常用大文件进行访问的具体过程如下：In the step Sb, the specific process of the caching server guiding users to access frequently used large files is as follows:
首先缓存服务器判断用户要求访问的常用大文件是否处于可编辑状态；First, the cache server judges whether the commonly used large files requested by the user are editable;
若是，则允许用户对该大文件实施读或写操作；同时在用户对文件实施读或写操作时，缓存服务器自动锁定该大文件；在用户操作完成后，缓存服务器发送更新日志给管理服务器，管理服务器根据收到的更新日志更新对应的大文件；If so, the user is allowed to perform read or write operations on the large file; at the same time, when the user performs read or write operations on the file, the cache server automatically locks the large file; after the user operation is completed, the cache server sends the update log to the management server, The management server updates the corresponding large file according to the received update log;
若否，则缓存服务器发送用户访问请求指令给管理服务器，然后进入步骤Sc。If not, the cache server sends the user access request command to the management server, and then enters step Sc.
优选的，所述步骤Sc中，管理服务器引导用户对文件进行访问的具体过程如下：Preferably, in the step Sc, the specific process of the management server guiding the user to access the file is as follows:
首先管理服务器判断用户要求访问的大文件是否处于可编辑状态；First, the management server judges whether the large file requested by the user is editable;
若是，则允许用户对该常用大文件实施读或写操作；同时在用户对大文件实施读或写操作时，管理服务器自动锁定该大文件；在用户操作完成后，管理服务器发送更新日志给缓存服务器，缓存服务器根据接收到的更新日志查询本地是否缓存了该大文件；若是，则更新该大文件；若否，则丢弃管理服务器发送的该更新日志；If so, the user is allowed to perform read or write operations on the commonly used large files; at the same time, when the user performs read or write operations on the large files, the management server automatically locks the large files; after the user operation is completed, the management server sends the update log to the cache server, the cache server queries whether the large file is cached locally according to the received update log; if so, updates the large file; if not, discards the update log sent by the management server;
若否，管理服务器发送“该文件正被其他用户使用”信息给用户设备，并终止用户的访问请求。If not, the management server sends the information "the file is being used by other users" to the user device, and terminates the user's access request.
优选的，所述大文件和小文件的标识性前缀分别为L和S；所述步骤Sa中当所要访问文件名对应的文件是大文件时，用户设备在用户访问请求指令的文件名前添加L；当所要访问文件名对应的文件是小文件时，用户设备在用户访问请求指令的文件名中添加S。Preferably, the identifying prefixes of the large file and the small file are L and S respectively; in the step Sa, when the file corresponding to the file name to be accessed is a large file, the user device adds L before the file name of the user access request instruction ; When the file corresponding to the file name to be accessed is a small file, the user device adds S to the file name of the user access request instruction.
本发明相对于现有技术具有如下的优点及效果：Compared with the prior art, the present invention has the following advantages and effects:
本发明引入用于判定大文件和小文件的文件名标识性前缀，将小文件、大文件中的访问频率较高的常用大文件以及一般大文件进行分开存储，其中小文件和常用大文件存储于访问时优先搜索的缓存服务器，大文件存储于访问时后搜索的管理服务器中，在访问过程中通过缓存服务器能够优先访问到小文件和常用大文件，大大提高了小文件以及常用大文件的访问速度，有效降低了文件访问复杂度，在确保高效访问小文件的前提下保证能够快速访问常用大文件，减少了用户访问等待时间。The present invention introduces the file name identification prefix for judging large files and small files, and stores small files, commonly used large files with high access frequency among large files and general large files separately, among which small files and commonly used large files are stored The cache server is searched first when accessing, and the large files are stored in the management server that is searched after the access. During the access process, the cache server can preferentially access small files and frequently used large files, which greatly improves the efficiency of small files and commonly used large files. The access speed effectively reduces the complexity of file access. On the premise of ensuring efficient access to small files, it can quickly access commonly used large files and reduce the waiting time for user access.
附图说明Description of drawings
图1是本发明中实现存储和访问方法的系统架构图。Fig. 1 is a system architecture diagram for implementing storage and access methods in the present invention.
图2是本发明小文件和大文件的存储方法流程图。Fig. 2 is a flow chart of the method for storing small files and large files in the present invention.
图3是本发明小文件和大文件的访问方法流程图。Fig. 3 is a flow chart of the method for accessing small files and large files in the present invention.
具体实施方式Detailed ways
下面结合实施例及附图对本发明作进一步详细的描述，但本发明的实施方式不限于此。The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.
实施例Example
实施例1Example 1
本实施例公开一种小文件和大文件的存储方法，该方法是在基于如图1所示的用户设备、缓存服务器、管理服务器和数据服务器的基础上实现的；其中This embodiment discloses a method for storing small files and large files, which is implemented on the basis of the user equipment, cache server, management server, and data server as shown in Figure 1; wherein
用户设备：用于创建文件、记录日志以及搜索日志；根据第一阈值确定所要创建的文件是大文件还是小文件，在创建的文件名上添加分别用于识别大文件和小文件的标识性前缀L和S，其中用户设备在创建的大文件的文件名前添加L，用户设备在创建的小文件的文件名前添加S，并发送创建新文件请求指令给缓存服务器。用于根据搜索日志确定所要访问文件名对应的文件是大文件还是小文件，然后在用户访问请求指令中的文件名中添加用于识别大文件和小文件的标识性前缀，当所要访问文件名对应的文件是大文件时，用户设备在用户访问请求指令的文件名前添加L；当所要访问文件名对应的文件是小文件时，用户设备在用户访问请求指令的文件名中添加S。用于发送用户访问请求指令。在本实施例中确定文件为大文件还是小文件的第一阈值为1M，即当文件的大小超过1M时，则将该文件判断为大文件，否则判断为小文件。User equipment: used to create files, record logs, and search logs; determine whether the file to be created is a large file or a small file according to the first threshold, and add identifying prefixes for identifying large files and small files to the created file name L and S, where the user equipment adds L before the file name of the large file to be created, and the user equipment adds S before the file name of the small file to be created, and sends a new file creation request instruction to the cache server. It is used to determine whether the file corresponding to the file name to be accessed is a large file or a small file according to the search log, and then add an identifying prefix used to identify the large file and small file to the file name in the user access request instruction. When the file name to be accessed When the corresponding file is a large file, the user device adds L before the file name of the user access request instruction; when the file corresponding to the file name to be accessed is a small file, the user device adds S to the file name of the user access request instruction. Used to send user access request instructions. In this embodiment, the first threshold for determining whether a file is a large file or a small file is 1M, that is, when the file size exceeds 1M, the file is judged as a large file, otherwise it is judged as a small file.
缓存服务器：用于存储小文件、小文件元数据、常用大文件和常用大文件元数据，其中所述常用大文件指的是访问频率大于第二阈值的大文件；用于接收用户设备发送的创建新文件请求指令；用于接收用户设备发送的用户访问请求指令，然后根据用户访问请求指令中文件名的标识性前缀判断用户需要访问的文件为大文件还是小文件；用于引导用户访问小文件和常用大文件；用于发送用户访问指令请求到管理服务器；缓存服务器还用于计算未保存在其中的大文件访问次数，将请求访问次数大于第二阈值的大文件认定为常用大文件，然后从数据服务器端将该大文件下载并保存在本地缓存区中。Cache server: used to store small files, small file metadata, commonly used large files and commonly used large file metadata, wherein the commonly used large files refer to large files with an access frequency greater than the second threshold; used to receive Create a new file request command; used to receive the user access request command sent by the user device, and then judge whether the file that the user needs to access is a large file or a small file according to the identifying prefix of the file name in the user access request command; used to guide the user to access the small file Files and common large files; used to send user access instruction requests to the management server; the cache server is also used to calculate the number of accesses of large files not stored therein, and identify large files with the number of requested accesses greater than the second threshold as frequently used large files, Then download the large file from the data server and save it in the local buffer.
管理服务器，用于存储大文件元数据；用于负责响应缓存服务器发送的用户访问请求指令；用于引导用户访问大文件。The management server is used to store metadata of large files; it is responsible for responding to user access request instructions sent by the cache server; it is used to guide users to access large files.
数据服务器，用于存储大文件数据块。Data server for storing large file data blocks.
如图2所示，本实施例中小文件和大文件的存储方法具体步骤如下：As shown in Figure 2, the specific steps of the method for storing small files and large files in this embodiment are as follows:
S1、用户通过用户设备创建新文件，根据第一阈值确定所要创建的新文件是大文件还是小文件，并设置文件名，在文件名上添加用于识别大文件和小文件的标识性前缀L和S；在本实施例中确定文件为大文件还是小文件的第一阈值为1M，即当文件的大小超过1M时，则将该文件判断为大文件，否则判断为小文件。S1. The user creates a new file through the user device, determines whether the new file to be created is a large file or a small file according to the first threshold, sets the file name, and adds an identifying prefix L for identifying large files and small files to the file name and S; in this embodiment, the first threshold for determining whether a file is a large file or a small file is 1M, that is, when the size of the file exceeds 1M, the file is judged as a large file, otherwise it is judged as a small file.
S2、用户设备将创建新文件请求指令发送给缓存服务器，缓存服务器收到用户设备发送的创建新文件请求指令后，通过创建新文件请求指令识别所要创建的新文件的文件名标识性前缀，判断所要创建的新文件是大文件还是小文件，若缓存服务器识别到所要创建的新文件的文件名标识性前缀为L，则判定要创建的新文件为大文件，若识别到所要创建的新文件的文件名标识性前缀为S，则判定要创建的新文件为小文件；S2. The user device sends the command to create a new file to the cache server. After the cache server receives the command to create a new file sent by the user device, the cache server recognizes the file name identifier prefix of the new file to be created through the command to create a new file, and judges Whether the new file to be created is a large file or a small file, if the cache server recognizes that the file name of the new file to be created has an identifying prefix of L, it will determine that the new file to be created is a large file, and if it recognizes the new file to be created The identifying prefix of the file name is S, then it is determined that the new file to be created is a small file;
若所要创建的新文件为小文件，则缓存服务器首先查询当前存储的小文件元数据，判断所要创建的新文件的文件名是否与缓存服务器中当前已有小文件的文件名重名；If the new file to be created is a small file, the cache server first queries the metadata of the currently stored small file to determine whether the file name of the new file to be created is the same as the file name of the currently existing small file in the cache server;
若是，则缓存服务器发送“文件已存在”信息给用户设备，并且丢弃用户发送的创建新文件请求指令；If so, the cache server sends the "file already exists" message to the user device, and discards the request instruction for creating a new file sent by the user;
若否，在缓存服务器中创建该文件并保存其元数据。If not, create the file and save its metadata in the cache server.
若所要创建的新文件为大文件，则缓存器将用户设备发送的创建新文件请求指令发送给管理服务器。If the new file to be created is a large file, the buffer sends the command for creating a new file sent by the user equipment to the management server.
S3、管理服务器收到来自于缓存服务器的创建新文件请求指令后，判断所要创建的新文件的文件名是否与管理服务器中当前已有大文件的文件名重名；S3. After the management server receives the command to create a new file from the cache server, it judges whether the file name of the new file to be created is the same as the file name of the existing large file in the management server;
若是，则管理服务器发送“文件已存在”信息给用户设备，并且丢弃用户发送的创建新文件请求指令；If so, the management server sends the "file already exists" message to the user device, and discards the request instruction for creating a new file sent by the user;
若否，在管理存服务器中创建该大文件并将其进行分块，得到数据块，并且保存该大文件的元数据；If not, create the large file in the management storage server and divide it into blocks to obtain data blocks, and save the metadata of the large file;
S4、管理服务器将创建的大文件划分的数据块发送给数据服务器进行保存；同时缓存服务器根据大文件请求访问次数确定大文件是否为常用大文件，若是，则从数据服务器端将该大文件下载并保存在缓存服务器中。S4. The management server sends the data blocks of the created large file to the data server for storage; at the same time, the cache server determines whether the large file is a commonly used large file according to the number of accesses requested by the large file, and if so, downloads the large file from the data server and stored in the cache server.
在本实施例中缓存服务器包括存储区和缓存区；小文件及小文件元数据存储在缓存服务器的存储区，以永久方式保存小文件元数据；常用大文件以及常用大文件元数据保存在缓存服务器的缓存区，以更新方式保存经常访问的常用大文件元数据。In this embodiment, the cache server includes a storage area and a cache area; small files and small file metadata are stored in the storage area of the cache server, and the small file metadata is permanently saved; commonly used large files and commonly used large file metadata are stored in the cache The cache area of the server saves the metadata of frequently accessed large files in an updated manner.
如图3所示，本实施例还公开了一种小文件和大文件的访问方法，步骤如下：As shown in Figure 3, this embodiment also discloses a method for accessing small files and large files, the steps are as follows:
Sa、用户在用户设备中输入文件名，通过用户设备提出访问文件请求，用户设备搜索日志确定用户输入的文件名对应的文件是大文件还是小文件，然后在用户访问请求指令中的文件名中添加大文件或小文件标识性前缀，当所要访问文件名对应的文件是大文件时，用户设备在用户访问请求指令的文件名前添加L；当所要访问文件名对应的文件是小文件时，用户设备在用户访问请求指令的文件名中添加S，用户设备发送用户访问请求指令给缓存服务器；Sa, the user inputs the file name in the user equipment, puts forward the access file request through the user equipment, the user equipment search log determines whether the file corresponding to the file name input by the user is a large file or a small file, and then in the file name in the user access request instruction Add an identifying prefix for large files or small files. When the file corresponding to the file name to be accessed is a large file, the user device adds L before the file name of the user access request instruction; when the file corresponding to the file name to be accessed is a small file, the user The device adds S to the file name of the user access request command, and the user device sends the user access request command to the cache server;
Sb、缓存服务器接收到用户设备发送的用户访问请求指令，根据用户访问请求指令中文件名的标识性前缀判断用户请求访问的是大文件还是小文件；若用户访问请求指令中文件名的标识性前缀为L时，则用户请求访问的是大文件，若用户访问请求指令中文件名的标识性前缀为S时，则用户请求访问的是小文件；Sb. The cache server receives the user access request instruction sent by the user device, and judges whether the user requests access to a large file or a small file according to the identifying prefix of the file name in the user access request instruction; if the identification of the file name in the user access request instruction When the prefix is L, the user requests to access a large file; if the identifying prefix of the file name in the user access request instruction is S, the user requests to access a small file;
若是小文件，则缓存服务器搜索本地存储区的小文件元数据，判断缓存服务器中是否存在用户要求访问的小文件；If it is a small file, the cache server searches the metadata of the small file in the local storage area to determine whether there is a small file requested by the user in the cache server;
若不存在，则缓存服务器返回“请求文件不存在”的未找到信息指令给用户设备；If it does not exist, the cache server returns a not-found information command of "the requested file does not exist" to the user device;
若存在，则缓存服务器引导用户对该小文件进行的访问，具体过程如下：If it exists, the cache server guides the user to access the small file. The specific process is as follows:
首先缓存服务器判断用户要求访问的小文件是否处于可编辑状态；First, the cache server judges whether the small file requested by the user is editable;
若是，则允许用户对该小文件实施读或写操作；同时在用户对小文件实施读或写操作时，缓存服务器自动锁定该小文件；If so, the user is allowed to perform read or write operations on the small file; at the same time, when the user performs read or write operations on the small file, the cache server automatically locks the small file;
若否，缓存服务器发送“该文件正被其他用户使用”信息给用户设备，并终止用户的访问请求；If not, the cache server sends the information "this file is being used by other users" to the user device, and terminates the user's access request;
若是大文件，则缓存服务器搜索本地缓存区的常用大文件元数据，判断缓存服务器中是否存在用户要求访问的大文件；If it is a large file, the cache server searches the common large file metadata in the local cache area to determine whether there is a large file requested by the user in the cache server;
若不存在，则缓存服务器将用户设备发送的用户访问请求指令发送给管理服务器，然后进入步骤Sc；同时缓存服务器器记录通过其请求访问的该大文件的请求访问次数，若访问次数大于第二阈值，则将该大文件认定为常用大文件，在存储过程中，从数据服务器端将该大文件下载并保存在本地缓存区中，其中本实施例中第二阈值的大小根据实际访问需求进行设定；If it does not exist, then the caching server sends the user access request command sent by the user equipment to the management server, and then enters step Sc; simultaneously, the caching server records the request access times of this large file accessed by its request, if the number of accesses is greater than the second threshold, the large file is identified as a commonly used large file, and in the storage process, the large file is downloaded from the data server and stored in the local cache area, wherein the size of the second threshold in this embodiment is determined according to actual access requirements. set up;
若存在，则缓存服务器引导用户对该大文件进行的访问；具体过程如下：If it exists, the cache server guides the user to access the large file; the specific process is as follows:
首先缓存服务器判断用户要求访问的常用大文件是否处于可编辑状态；First, the cache server judges whether the commonly used large files requested by the user are editable;
若是，则允许用户对该常用大文件实施读或写操作；同时在用户对常用大文件实施读或写操作时，缓存服务器自动锁定该常用大文件；在用户操作完成后，缓存服务器发送更新日志给管理服务器，管理服务器根据收到的更新日志更新对应的大文件；If so, the user is allowed to perform read or write operations on the commonly used large files; at the same time, when the user performs read or write operations on the commonly used large files, the cache server automatically locks the frequently used large files; after the user operation is completed, the cache server sends an update log To the management server, the management server updates the corresponding large file according to the received update log;
若否，则缓存服务器发送用户访问请求指令给管理服务器，然后进入步骤Sc。If not, the cache server sends the user access request command to the management server, and then enters step Sc.
Sc、管理服务器接收到来自于缓存服务器的用户访问请求指令后，搜索本地大文件元数据，判断管理服务器中是否存在用户要求访问的大文件；Sc. After the management server receives the user access request instruction from the cache server, it searches the metadata of the local large file, and determines whether there is a large file requested by the user in the management server;
若不存在，则管理服务器返回“请求文件不存在”的未找到信息指令给用户设备；If it does not exist, the management server returns the not found information instruction of "the requested file does not exist" to the user equipment;
若存在，则引导用户对大文件访问；具体过程如下：If it exists, guide the user to access the large file; the specific process is as follows:
首先管理服务器判断用户要求访问的大文件是否处于可编辑状态；First, the management server judges whether the large file requested by the user is editable;
若是，则允许用户对该常用大文件实施读或写操作；同时在用户对大文件实施读或写操作时，管理服务器自动锁定该大文件；在用户操作完成后，管理服务器发送更新日志给缓存服务器，缓存服务器根据接收到的更新日志查询本地是否缓存了该大文件；若是，则更新该大文件；若否，则丢弃管理服务器发送的该更新日志；If so, the user is allowed to perform read or write operations on the commonly used large files; at the same time, when the user performs read or write operations on the large files, the management server automatically locks the large files; after the user operation is completed, the management server sends the update log to the cache server, the cache server queries whether the large file is cached locally according to the received update log; if so, updates the large file; if not, discards the update log sent by the management server;
若否，管理服务器发送“该文件正被其他用户使用”信息给用户设备，并终止用户的访问请求。If not, the management server sends the information "the file is being used by other users" to the user device, and terminates the user's access request.
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410559955.0ACN104331428B (en) | 2014-10-20 | 2014-10-20 | The storage of a kind of small documents and big file and access method |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410559955.0ACN104331428B (en) | 2014-10-20 | 2014-10-20 | The storage of a kind of small documents and big file and access method |
| Publication Number | Publication Date |
|---|---|
| CN104331428Atrue CN104331428A (en) | 2015-02-04 |
| CN104331428B CN104331428B (en) | 2017-07-04 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410559955.0AActiveCN104331428B (en) | 2014-10-20 | 2014-10-20 | The storage of a kind of small documents and big file and access method |
| Country | Link |
|---|---|
| CN (1) | CN104331428B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105404645A (en)* | 2015-10-27 | 2016-03-16 | 北京乐动卓越科技有限公司 | File management method in file server system and file server system |
| CN105608193A (en)* | 2015-12-23 | 2016-05-25 | 深圳市深信服电子科技有限公司 | Data management method and apparatus for distributed file system |
| CN106020713A (en)* | 2015-09-16 | 2016-10-12 | 展视网（北京）科技有限公司 | File storage method based on buffer area |
| CN106230971A (en)* | 2016-08-29 | 2016-12-14 | 无锡华云数据技术服务有限公司 | Big document distribution method based on CDN |
| CN106453474A (en)* | 2015-07-17 | 2017-02-22 | 生物辐射实验室股份有限公司 | Network transfer of large files in an unstable network environment |
| WO2017028688A1 (en)* | 2015-08-14 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method, device and system for reading and writing files |
| CN107341267A (en)* | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN108174136A (en)* | 2018-03-14 | 2018-06-15 | 成都创信特电子技术有限公司 | Cloud disk video coding and storage method |
| CN108491167A (en)* | 2018-03-29 | 2018-09-04 | 重庆大学 | A kind of quick random distribution storage method of industrial process floor data |
| CN108667939A (en)* | 2018-05-21 | 2018-10-16 | 北京五八信息技术有限公司 | A kind of method, apparatus of file download, terminal and storage medium |
| CN109002260A (en)* | 2018-07-02 | 2018-12-14 | 深圳市茁壮网络股份有限公司 | A kind of data cached processing method and processing system |
| CN110750505A (en)* | 2019-08-31 | 2020-02-04 | 苏州浪潮智能科技有限公司 | A large file reading optimization method, device, device and storage medium |
| CN111324581A (en)* | 2020-02-14 | 2020-06-23 | 苏州浪潮智能科技有限公司 | A file writing method, device, device and storage medium |
| CN113590550A (en)* | 2021-07-30 | 2021-11-02 | 展讯通信(天津)有限公司 | File creation method and device and electronic equipment |
| CN116800733A (en)* | 2023-08-18 | 2023-09-22 | 荣耀终端有限公司 | A differential package downloading method and server |
| US12399868B1 (en)* | 2023-03-20 | 2025-08-26 | Amazon Technologies, Inc. | Managed file compaction for distributed storage systems |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070078914A1 (en)* | 2005-09-30 | 2007-04-05 | International Business Machines Corporation | Method, apparatus and program storage device for providing a centralized policy based preallocation in a distributed file system |
| CN101075241A (en)* | 2006-12-26 | 2007-11-21 | 腾讯科技（深圳）有限公司 | Method and system for processing buffer |
| CN102855239A (en)* | 2011-06-28 | 2013-01-02 | 清华大学 | Distributed geographical file system |
| CN103500089A (en)* | 2013-09-18 | 2014-01-08 | 北京航空航天大学 | Small file storage system suitable for Mapreduce calculation model |
| CN104079600A (en)* | 2013-03-27 | 2014-10-01 | 中兴通讯股份有限公司 | File storage method, file storage device, file access client and metadata server system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070078914A1 (en)* | 2005-09-30 | 2007-04-05 | International Business Machines Corporation | Method, apparatus and program storage device for providing a centralized policy based preallocation in a distributed file system |
| CN101075241A (en)* | 2006-12-26 | 2007-11-21 | 腾讯科技（深圳）有限公司 | Method and system for processing buffer |
| CN102855239A (en)* | 2011-06-28 | 2013-01-02 | 清华大学 | Distributed geographical file system |
| CN104079600A (en)* | 2013-03-27 | 2014-10-01 | 中兴通讯股份有限公司 | File storage method, file storage device, file access client and metadata server system |
| CN103500089A (en)* | 2013-09-18 | 2014-01-08 | 北京航空航天大学 | Small file storage system suitable for Mapreduce calculation model |
| Title |
|---|
| 贾瑞勇 等: ""基于网络存储的异构集群文件系统研究与实现"", 《西北工业大学学报》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453474B (en)* | 2015-07-17 | 2021-02-05 | 生物辐射实验室股份有限公司 | Network transmission of large files in unstable network environments |
| CN106453474A (en)* | 2015-07-17 | 2017-02-22 | 生物辐射实验室股份有限公司 | Network transfer of large files in an unstable network environment |
| WO2017028688A1 (en)* | 2015-08-14 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method, device and system for reading and writing files |
| CN106469150A (en)* | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | File read/write method, device and system |
| CN106469150B (en)* | 2015-08-14 | 2019-10-08 | 阿里巴巴集团控股有限公司 | File read/write method, device and system |
| US10338917B2 (en) | 2015-08-14 | 2019-07-02 | Alibaba Group Holding Limited | Method, apparatus, and system for reading and writing files |
| CN106020713A (en)* | 2015-09-16 | 2016-10-12 | 展视网（北京）科技有限公司 | File storage method based on buffer area |
| CN105404645A (en)* | 2015-10-27 | 2016-03-16 | 北京乐动卓越科技有限公司 | File management method in file server system and file server system |
| CN105608193A (en)* | 2015-12-23 | 2016-05-25 | 深圳市深信服电子科技有限公司 | Data management method and apparatus for distributed file system |
| CN105608193B (en)* | 2015-12-23 | 2019-03-26 | 深信服科技股份有限公司 | The data managing method and device of distributed file system |
| CN106230971B (en)* | 2016-08-29 | 2019-03-01 | 无锡华云数据技术服务有限公司 | Big document distribution method based on CDN |
| CN106230971A (en)* | 2016-08-29 | 2016-12-14 | 无锡华云数据技术服务有限公司 | Big document distribution method based on CDN |
| CN107341267A (en)* | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN108174136A (en)* | 2018-03-14 | 2018-06-15 | 成都创信特电子技术有限公司 | Cloud disk video coding and storage method |
| CN108491167B (en)* | 2018-03-29 | 2020-12-04 | 重庆大学 | A fast random distribution storage method for industrial process working condition data |
| CN108491167A (en)* | 2018-03-29 | 2018-09-04 | 重庆大学 | A kind of quick random distribution storage method of industrial process floor data |
| CN108667939A (en)* | 2018-05-21 | 2018-10-16 | 北京五八信息技术有限公司 | A kind of method, apparatus of file download, terminal and storage medium |
| CN109002260B (en)* | 2018-07-02 | 2021-08-13 | 深圳市茁壮网络股份有限公司 | Processing method and processing system for cache data |
| CN109002260A (en)* | 2018-07-02 | 2018-12-14 | 深圳市茁壮网络股份有限公司 | A kind of data cached processing method and processing system |
| CN110750505A (en)* | 2019-08-31 | 2020-02-04 | 苏州浪潮智能科技有限公司 | A large file reading optimization method, device, device and storage medium |
| CN111324581A (en)* | 2020-02-14 | 2020-06-23 | 苏州浪潮智能科技有限公司 | A file writing method, device, device and storage medium |
| CN113590550A (en)* | 2021-07-30 | 2021-11-02 | 展讯通信(天津)有限公司 | File creation method and device and electronic equipment |
| US12399868B1 (en)* | 2023-03-20 | 2025-08-26 | Amazon Technologies, Inc. | Managed file compaction for distributed storage systems |
| CN116800733A (en)* | 2023-08-18 | 2023-09-22 | 荣耀终端有限公司 | A differential package downloading method and server |
| CN116800733B (en)* | 2023-08-18 | 2023-11-17 | 荣耀终端有限公司 | A differential package downloading method and server |
| Publication number | Publication date |
|---|---|
| CN104331428B (en) | 2017-07-04 |
| Publication | Publication Date | Title |
|---|---|---|
| CN104331428B (en) | The storage of a kind of small documents and big file and access method | |
| CN107247808B (en) | Distributed NewSQL database system and picture data query method | |
| CN103995855B (en) | Method and device for storing data | |
| CN110147204B (en) | Metadata disk-dropping method, device and system and computer-readable storage medium | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN104156322B (en) | A kind of buffer memory management method and cache management device | |
| CN104679898A (en) | Big data access method | |
| CN104778270A (en) | Storage method for multiple files | |
| CN110502535B (en) | Data access method, device, equipment and storage medium | |
| CN104516974A (en) | Management method and device for file system directory entry | |
| CN109597829B (en) | Middleware method for realizing searchable encryption relational database cache | |
| CN115437579B (en) | A metadata management method, device, computer equipment and readable storage medium | |
| WO2018120876A1 (en) | Method and device for searching for cache update | |
| CN110515979B (en) | Data query method, device, device and storage medium | |
| CN106372266A (en) | Cache and accessing method of cloud operation system based on aspects and configuration documents | |
| CN104899249B (en) | Reliable index upgrade system and method under a kind of mass data | |
| CN104423982A (en) | Request processing method and device | |
| US20080071992A1 (en) | Method and Apparatus for Space Efficient Identification of Candidate Objects for Eviction from a Large Cache | |
| CN109871338B (en) | Data storage method and device and computer equipment | |
| CN107220287A (en) | For the index managing method of log query, device, storage medium and equipment | |
| CN105701233A (en) | Method for optimizing server cache management | |
| CN102724301B (en) | Cloud database system and method and equipment for reading and writing cloud data | |
| CN107633102A (en) | A kind of method, apparatus, system and equipment for reading metadata | |
| WO2020147334A1 (en) | Method and system for data query based on ignite cache architecture | |
| US8135760B1 (en) | Determining the lineage of a content unit on an object addressable storage system |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |