技术领域Technical Field
本发明涉及分布式文件系统技术领域，尤其涉及一种读写混合场景文件大小获取方法、装置和设备。The present invention relates to the technical field of distributed file systems, and in particular to a method, device and equipment for obtaining the size of a file in a read-write mixed scenario.
背景技术Background technique
随着高清视频(例如4K、8K)的普及，视频素材的体积变得越来越大，单个视频素材的文件大小会占用几百GB或者更多，越来越多的媒体客户选择使用分布式文件存储来存储海量文件。在广电领域，除了一个客户端在正常的采集收录素材外，还伴随着其他客户端同时对这个文件进行读取的操作。在分布式文件系统(例如Cephfs)中，多个存储客户端对同一个文件的读写，需要与元数据服务(Meta Data Service，MDS)进行一致性权限(Consistency,Availability,Partition Tolerance，CAP)交互，由MDS统一管控客户端的读写权限，其他客户端在读的同时，还需要去不断的调用获取属性函数(getattr)获取文件大小(size)，这也需要通过MDS切锁获得权限后才能进行，这个操作会导致正常的读写权限的回收，影响文件读写性能。With the popularity of high-definition videos (such as 4K and 8K), the size of video materials is getting larger and larger. The file size of a single video material will occupy hundreds of GB or more. More and more media customers choose to use distributed file storage to store massive files. In the field of radio and television, in addition to the normal collection and recording of materials by a client, there are also other clients reading the file at the same time. In a distributed file system (such as Cephfs), multiple storage clients need to interact with the metadata service (MDS) for consistency permissions (Consistency, Availability, Partition Tolerance, CAP) when reading and writing the same file. The MDS uniformly manages the read and write permissions of the client. While reading, other clients also need to continuously call the getattr function to obtain the file size (size), which also requires the permission to be obtained through the MDS lock. This operation will cause the normal read and write permissions to be recovered, affecting the file read and write performance.
发明内容Summary of the invention
有鉴于此，本发明提供一种读写混合场景文件大小获取方法、装置和设备，用于解决读写混合场景文件大小获取影响正常I/O读写的技术问题。In view of this, the present invention provides a method, device and equipment for obtaining the file size in a read-write hybrid scenario, which are used to solve the technical problem that the file size acquisition in a read-write hybrid scenario affects the normal I/O reading and writing.
基于本发明实施例的一方面，本发明提供了一种读写混合场景文件大小获取方法，该方法应用于分布式文件系统中的元数据服务MDS中，在已为向目标文件写入数据的写客户端分配读写权限，并为从目标文件读取数据的读客户端分配读权限的场景下，该方法包括：Based on one aspect of an embodiment of the present invention, the present invention provides a method for obtaining the file size in a read-write hybrid scenario, which is applied to a metadata service MDS in a distributed file system. In a scenario where a write client that writes data to a target file has been assigned read-write permissions, and a read client that reads data from a target file has been assigned read permissions, the method includes:
接收读客户端获取目标文件大小的第一请求；Receive a first request from a read client to obtain a target file size;
根据第一请求携带的目标文件信息确定写客户端；Determine the writing client according to the target file information carried in the first request;
向写客户端发送获取目标文件大小的第二请求；Sending a second request to the write client to obtain the target file size;
接收写客户端返回的第二请求响应；Receive a second request response returned by the write client;
发送第一请求响应给读客户端，其中携带第二请求响应返回的目标文件大小。The first request response is sent to the read client, carrying the target file size returned by the second request response.
进一步地，根据第一请求携带的目标文件信息确定写客户端的方法为：Furthermore, the method for determining the writing client according to the target file information carried in the first request is:
根据第一请求中携带的目标文件信息从目标文件的元数据或集中式锁服务中查询当前已分配写权限的客户端，将查询到的客户端确定为写客户端。According to the target file information carried in the first request, the client currently assigned with the write permission is queried from the metadata of the target file or the centralized lock service, and the queried client is determined as the write client.
进一步地，向写客户端发送的获取目标文件大小的第二请求中携带第一请求的请求标识及目标文件的文件元数据结构标识(inode号)；Furthermore, the second request for obtaining the target file size sent to the write client carries the request identifier of the first request and the file metadata structure identifier (inode number) of the target file;
第二请求响应中携带目标文件大小及第一请求的请求标识；The second request response carries the target file size and the request identifier of the first request;
根据第一请求的请求标识查询到第一请求实例，根据第一请求实例将第一请求响应发送给读客户端。A first request instance is queried according to the request identifier of the first request, and a first request response is sent to the read client according to the first request instance.
基于本发明实施例的另一方面，本发明还提供一种读写混合场景文件大小获取方法，该方法应用于分布式文件系统中已分配目标文件写权限的写客户端，该方法包括：Based on another aspect of an embodiment of the present invention, the present invention further provides a method for obtaining the file size in a read-write hybrid scenario, the method being applied to a writing client that has been assigned write permission for a target file in a distributed file system, the method comprising:
接收元数据服务MDS发送的获取目标文件大小的第二请求；receiving a second request for obtaining a target file size sent by the metadata service MDS;
从本地获取目标文件大小，并通过第二请求响应将目标文件大小发送给元数据服务。The target file size is obtained locally, and the target file size is sent to the metadata service through a second request response.
进一步地，所述第二请求中携带第一请求的请求标识及目标文件的文件元数据结构标识(inode号)，所述第一请求为已分配目标文件读权限的读客户端向元数据服务MDS发送的获取目标文件大小的请求；Furthermore, the second request carries a request identifier of the first request and a file metadata structure identifier (inode number) of the target file, and the first request is a request for obtaining the size of the target file sent by a read client that has been assigned read permission for the target file to the metadata service MDS;
所述第二请求响应中还携带第一请求的请求标识，以使元数据服务根据第一请求的请求标识查询第一请求实例后，将第一请求响应发送给读客户端。The second request response also carries the request identifier of the first request, so that the metadata service queries the first request instance according to the request identifier of the first request and then sends the first request response to the reading client.
基于本发明实施例，本发明还提供一种读写混合场景文件大小获取装置，该装置应用于分布式文件系统中的元数据服务MDS中，在已为向目标文件写入数据的写客户端分配读写权限，并为从目标文件读取数据的读客户端分配读权限的场景下，该装置包括：Based on the embodiment of the present invention, the present invention also provides a device for obtaining the file size in a read-write hybrid scenario, which is applied to a metadata service MDS in a distributed file system. In a scenario where a write client that writes data to a target file has been assigned read-write permissions, and a read client that reads data from a target file has been assigned read permissions, the device includes:
请求转发模块，用于接收读客户端获取目标文件大小的第一请求；根据第一请求携带的目标文件信息确定写客户端；及向写客户端发送获取目标文件大小的第二请求；A request forwarding module, configured to receive a first request from a read client to obtain a target file size; determine a write client according to the target file information carried in the first request; and send a second request to the write client to obtain the target file size;
文件大小反馈模块，用于接收写客户端返回的第二请求响应；及发送第一请求响应给读客户端，其中携带第二请求响应返回的目标文件大小。The file size feedback module is used to receive the second request response returned by the write client; and send the first request response to the read client, which carries the target file size returned by the second request response.
进一步地，请求转发模块向写客户端发送的获取目标文件大小的第二请求中携带第一请求的请求标识及目标文件的文件元数据结构标识(inode号)；Furthermore, the second request for obtaining the target file size sent by the request forwarding module to the writing client carries the request identifier of the first request and the file metadata structure identifier (inode number) of the target file;
文件大小反馈模块接收的第二请求响应中携带目标文件大小及第一请求的请求标识；The second request response received by the file size feedback module carries the target file size and the request identifier of the first request;
文件大小反馈模块根据第一请求的请求标识查询到第一请求实例，根据第一请求实例将第一请求响应发送给读客户端。The file size feedback module queries the first request instance according to the request identifier of the first request, and sends the first request response to the read client according to the first request instance.
基于本发明实施例，本发明还提供一种读写混合场景文件大小获取装置，该装置应用于分布式文件系统中已分配目标文件写权限的写客户端，该装置包括：Based on the embodiment of the present invention, the present invention also provides a device for obtaining the size of a file in a read-write mixed scenario, which is applied to a writing client that has been assigned write permission for a target file in a distributed file system, and the device includes:
接收单元，用于接收元数据服务MDS发送的获取目标文件大小的第二请求；A receiving unit, configured to receive a second request for obtaining a target file size sent by the metadata service MDS;
响应单元，用于从本地获取目标文件大小，并通过第二请求响应将目标文件大小发送给元数据服务。The response unit is used to obtain the target file size locally and send the target file size to the metadata service through a second request response.
进一步地，所述第二请求中携带第一请求的请求标识及目标文件的文件元数据结构标识(inode号)，所述第一请求为已分配目标文件读权限的读客户端向元数据服务MDS发送的获取目标文件大小的请求；Furthermore, the second request carries a request identifier of the first request and a file metadata structure identifier (inode number) of the target file, and the first request is a request for obtaining the size of the target file sent by a read client that has been assigned read permission for the target file to the metadata service MDS;
所述第二请求响应中还携带第一请求的请求标识，以使元数据服务根据第一请求的请求标识查询第一请求实例后，将第一请求响应发送给读客户端。The second request response also carries the request identifier of the first request, so that the metadata service queries the first request instance according to the request identifier of the first request and then sends the first request response to the reading client.
本发明提供的装置可以以软件、硬件或软硬结合的方式实现。当以软件模块方式实现时，该软件模块的程序代码被加载到设备的存储介质中，由处理器读取存储介质中的程序代码并执行。The device provided by the present invention can be implemented in software, hardware or a combination of software and hardware. When implemented in a software module, the program code of the software module is loaded into the storage medium of the device, and the processor reads the program code in the storage medium and executes it.
本发明实施例中的读客户端向MDS请求获取文件大小时，MDS发请求给写客户端，直接从写客户端获取文件大小，将文件大小再返回给读客户端。通过本发明，MDS不需要频繁切换分布式锁回收写客户端权限，就能拿到准确的文件大小给读客户端，既保证了所获得的文件大小的准确性，又不影响写I/O的性能。When the read client in the embodiment of the present invention requests the MDS to obtain the file size, the MDS sends a request to the write client, obtains the file size directly from the write client, and returns the file size to the read client. Through the present invention, the MDS does not need to frequently switch the distributed lock to reclaim the write client's authority, and can get the accurate file size to the read client, which not only ensures the accuracy of the obtained file size, but also does not affect the performance of the write I/O.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更加清楚地说明本发明实施例或者现有技术中的技术方案，下面将对本发明实施例或者现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明中记载的一些实施例，对于本领域普通技术人员来讲，还可以根据本发明实施例的这些附图获得其他的附图。In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.
图1为一客户端以独占模式写入文件的交互过程；Figure 1 is an interactive process of a client writing a file in exclusive mode;
图2为一客户端写一客户端读场景下锁切换交互过程示意图；FIG2 is a schematic diagram of the lock switching interaction process in a one-client write-one-client read scenario;
图3为一客户端写入一客户端读场景下频繁获取文件大小导致写业务中断的示意图；FIG3 is a schematic diagram of a write service interruption caused by frequent acquisition of file size in a client-write-client-read scenario;
图4为本发明一实施例提供的读写混合场景下文件大小获取方法的消息交互步骤流程示意图；FIG4 is a flow chart showing message interaction steps of a method for obtaining file size in a read-write hybrid scenario provided by an embodiment of the present invention;
图5为本发明一实施例提供的实施读写混合场景文件大小获取方法的电子设备结构示意图。FIG5 is a schematic diagram of the structure of an electronic device for implementing a method for obtaining the file size in a mixed read-write scenario provided by an embodiment of the present invention.
具体实施方式Detailed ways
在本发明实施例使用的术语仅仅是出于描述特定实施例的目的，而非限制本发明实施例。本发明实施例中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其它含义。应当理解，尽管在本发明实施例可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用于区别类似的信息、实体或步骤，而不是用于描述特定的顺序或先后次序。例如，在不脱离本发明实施例范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。此外，所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。本发明中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，其中A，B可以是单数或者复数。并且，在本发明的描述中，除非另有说明，“多个”是指两个或多于两个。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b，或c中的至少一项(个)，可以表示：a，b，c，a-b，a-c，b-c，或a-b-c，其中a，b，c可以是单个，也可以是多个。The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms of "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish similar information, entities or steps, rather than to describe a specific order or sequence. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. In addition, the word "if" used can be interpreted as "at..." or "when..." or "in response to determination". The "and/or" in the present invention is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and/or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or plural.
以下以分布式文件系统cephfs为例，分析说明多机头即多存储客户端读写并获取文件size的场景中，客户端与MDS之间的权限交互过程。The following uses the distributed file system cephfs as an example to analyze and explain the permission interaction process between the client and MDS in the scenario where multiple heads, i.e. multiple storage clients, read and write files and obtain the file size.
图1为一客户端以独占模式写入文件的交互过程。图1的示例中，假设需要收录一个素材文件，一个客户端(client1)需要往存储集群写入文件，它首先需要与MDS交互，打开文件时，传入O_RDWR(读写)标志，MDS处理发现此时只有一个客户端操作此文件，则将其权限锁置为LOCK_EXCL(独占)，同时分配给客户端Fsxwrcb(读写以及缓存)权限，客户端拿到写权限，然后才能去执行写文件操作，将数据写入OSD(对象持久化设备)。Figure 1 shows the interactive process of a client writing a file in exclusive mode. In the example of Figure 1, assuming that a material file needs to be included, a client (client1) needs to write the file to the storage cluster. It first needs to interact with the MDS. When opening the file, the O_RDWR (read-write) flag is passed in. The MDS process finds that only one client is operating this file at this time, so its permission lock is set to LOCK_EXCL (exclusive), and the Fsxwrcb (read-write and cache) permission is assigned to the client. The client obtains the write permission, and then it can perform the file write operation and write the data to the OSD (object persistence device).
图2为一客户端写一客户端读场景下锁切换交互过程示意图。在图1示例的client1写入文件的基础上，图2示例了client2加入后的锁切换过程。客户端(client2)以O_RDONLY(只读方式)打开位于OSD中的同一个文件，来读这个文件，同样的，它需要先与MDS交互，拿到权限后，再去读取，但是由于在第一步MDS已经给客户端(client1)分配了读写缓存权限，现在客户端(client2)来读这个文件，MDS首先要将(client1)的Fcb(文件读写缓存)权限进行回收，回收成功后，MDS将权限锁切到LOCK_MIX(混合读写)状态，表示有多个客户端同时读写，然后再分配给(client2)Fr(文件直读)的权限，分配给(client1)Fwr(文件直读直写)权限。客户端收到权限后，才能再发请求给osd进行读写操作。Figure 2 is a schematic diagram of the lock switching interaction process in a one-client write and one-client read scenario. Based on the example of client1 writing a file in Figure 1, Figure 2 illustrates the lock switching process after client2 joins. The client (client2) opens the same file located in the OSD in O_RDONLY (read-only mode) to read the file. Similarly, it needs to interact with the MDS first and obtain the permission before reading. However, since the MDS has assigned the read and write cache permissions to the client (client1) in the first step, now the client (client2) wants to read the file. The MDS must first reclaim the Fcb (file read and write cache) permission of (client1). After the reclaim is successful, the MDS switches the permission lock to the LOCK_MIX (mixed read and write) state, indicating that multiple clients are reading and writing at the same time, and then assigns the Fr (file direct read) permission to (client2) and the Fwr (file direct read and write) permission to (client1). After the client receives the permission, it can send a request to the osd for read and write operations.
图3为一客户端写入一客户端读场景下频繁获取文件大小导致写业务中断的示意图。图3示例了读写混合场景下文件读写的锁切换过程，假设client1采集一个大的视频文件，client1不断地向文件中写入视频数据，client2在client1写入的同时读取视频数据进行播放，由于文件大小因为不断的写入过程是变化的，client2为了能不断读取到新写入的视频数据，客户端client2在读取文件的过程中，需要不断地通过getattr(读元数据请求)去获取文件的size(大小)，判断具体能读到文件的哪个偏移位置，这个过程中，MDS在强一致性要求的约束下，需要将写权限先回收，将文件变为元数据只读状态，才能返回size给客户端，然而，写权限的收回，会影响正常的写I/O的业务，造成client1写入I/O的波动，降低了文件访问效率。Figure 3 is a schematic diagram of frequent acquisition of file size in a scenario where one client writes and one client reads, resulting in interruption of write services. Figure 3 illustrates the file read and write lock switching process in a mixed read and write scenario. Assuming that client 1 collects a large video file, client 1 continuously writes video data to the file, and client 2 reads the video data for playback while client 1 is writing. Since the file size changes due to the continuous writing process, in order to continuously read the newly written video data, client 2 needs to continuously obtain the file size through getattr (read metadata request) during the process of reading the file to determine which specific offset position of the file can be read. In this process, under the constraint of strong consistency requirements, MDS needs to reclaim the write permission first and change the file to metadata read-only state before returning the size to the client. However, the revocation of the write permission will affect the normal write I/O business, causing fluctuations in client 1's write I/O and reducing file access efficiency.
基于对上述应用场景的分析，本发明的目的是提供一种读写混合场景文件大小获取方案。本发明技术方案的主体思路是：在读文件客户端获取文件尺寸时，MDS不再收回写文件客户端的写入权限，而是由MDS从写文件客户端获取文件大小后将文件大小转发给读文件客户端。本发明方案在读文件客户端获取文件大小期间不需要进行分布式锁的切换，因此提高了边写边读场景下文件的读写I/O效率。Based on the analysis of the above application scenarios, the purpose of the present invention is to provide a file size acquisition solution for a mixed read-write scenario. The main idea of the technical solution of the present invention is: when the file reading client obtains the file size, the MDS no longer revokes the write permission of the file writing client, but the MDS obtains the file size from the file writing client and forwards the file size to the file reading client. The solution of the present invention does not require the switching of distributed locks during the period when the file reading client obtains the file size, thereby improving the read and write I/O efficiency of files in the write-while-read scenario.
基于本发明的基本思路，本发明提供如下实施例，其中，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。Based on the basic idea of the present invention, the present invention provides the following embodiments, wherein it should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
图4为本发明一实施例提供的读写混合场景下文件大小获取方法的消息交互步骤流程示意图。图示的应用场景为读写混合场景，该场景下，写客户端client1在不断向目标文件写入数据，一个或多个读客户端client2在client1不断写入数据的同时也在不断读取目标文件的数据，MDS先后为写客户端client1分配了写权限(Fw)，为读客户端client2分配读权限(Fr)，MDS的一致性权限caps状态处于读写混合(LOCK MIX)状态，在该状态下，获取文件大小size的过程如下：Figure 4 is a flowchart of the message interaction steps of the method for obtaining the file size in a mixed read-write scenario provided by an embodiment of the present invention. The illustrated application scenario is a mixed read-write scenario, in which a write client client1 is continuously writing data to the target file, and one or more read clients client2 are continuously reading data from the target file while client1 is continuously writing data. The MDS has successively assigned write permission (Fw) to the write client client1 and read permission (Fr) to the read client client2. The consistency permission caps state of the MDS is in a mixed read-write (LOCK MIX) state. In this state, the process of obtaining the file size size is as follows:
步骤401.读客户端client2向MDS发送获取目标文件大小的第一请求(getattr)。Step 401: The read client client2 sends a first request (getattr) to obtain the target file size to the MDS.
步骤402.MDS接收到第一请求后，根据第一请求携带的目标文件信息确定写客户端；Step 402: After receiving the first request, the MDS determines the write client according to the target file information carried in the first request;
MDS接收到第一请求后，通过handle_client_getattr函数处理第一请求，通过该函数实现获取目标文件大小的请求由MDS到写客户端的转发。After receiving the first request, the MDS processes the first request through the handle_client_getattr function, and forwards the request for obtaining the target file size from the MDS to the writing client through the function.
MDS可通过元数据或集中式锁服务来查询当前已分配写权限的客户端。MDS can query the clients that are currently assigned write permissions through metadata or centralized lock services.
在进行文件写入操作时，客户端会更新目标文件的元数据，包括写客户端的信息。MDS可通过查询目标文件的元数据来获取当前正在进行写入的客户端信息。When writing a file, the client will update the metadata of the target file, including the information of the writing client. MDS can obtain the information of the client currently writing by querying the metadata of the target file.
另一种确定写客户端的方式是采用集中式锁服务，所有客户端在写入文件之前都需要向该中心服务请求写入锁。在获得写入锁时，客户端将自己的信息注册到锁服务中。MDS可根据目标文件信息查找集中式锁服务以获取当前所有正在进行写入的客户端列表。Another way to determine the writing client is to use a centralized lock service. All clients need to request a write lock from the central service before writing a file. When obtaining a write lock, the client registers its information with the lock service. MDS can search the centralized lock service based on the target file information to obtain a list of all clients currently writing.
例如，MDS根据目标文件的文件元数据结构标识(inode号)从MDS的元数据或集中式锁服务中获取到写客户端标识，例如从client_range(写客户端偏移范围)元数据结构中查找到写客户端标识，client_range里保存了每个写客户端对应的能写到多大的偏移的数值，采用<client_id,range>的存储结构。通过判断range的值大于0，可确定对应的client_id是写客户端。在边采边播高清视频的应用场景下，通常只有一个写客户端在写入大的视频媒体文件，可有多个读客户端同时读取视频文件。For example, MDS obtains the write client identifier from the MDS metadata or centralized lock service based on the file metadata structure identifier (inode number) of the target file. For example, the write client identifier is found from the client_range (write client offset range) metadata structure. The client_range stores the value of the offset that each write client can write to, using the storage structure of <client_id,range>. By judging that the value of range is greater than 0, it can be determined that the corresponding client_id is the write client. In the application scenario of simultaneously recording and broadcasting high-definition video, there is usually only one write client writing a large video media file, and multiple read clients can read the video file at the same time.
文件inode在Unix和类Unix操作系统中用于表示文件或目录的元数据结构，其中包括了文件的属性、权限、所属用户和组、文件大小、时间戳等信息。inode号是inode数据结构的唯一标识。The file inode is used to represent the metadata structure of a file or directory in Unix and Unix-like operating systems, which includes information such as file attributes, permissions, owner and group, file size, timestamp, etc. The inode number is the unique identifier of the inode data structure.
步骤403.生成获取目标文件大小size的消息体MClientGetsize，将目标文件inode号及getattr请求标识tid写入消息体中；Step 403: Generate a message body MClientGetsize for obtaining the target file size, and write the target file inode number and getattr request identifier tid into the message body;
步骤404.向写客户端发送获取目标文件大小的第二请求(MSG_FILE_GET_SIZE消息)，请求中携带消息体MClientGetsize。Step 404: Send a second request (MSG_FILE_GET_SIZE message) to the write client to obtain the target file size. The request carries a message body MClientGetsize.
步骤405.写客户端接收第二请求后，从消息体MClientGetsize中获取目标文件inode号，通过inode号在本地查找出目标文件的inode结构缓存，将inode数据结构中的文件大小size赋值给返回的消息结构体中。Step 405. After receiving the second request, the write client obtains the target file inode number from the message body MClientGetsize, searches the inode structure cache of the target file locally through the inode number, and assigns the file size size in the inode data structure to the returned message structure body.
写客户端收到第二请求后，通过handle_mds_get_file_size函数进行处理，从本地获取目标文件大小并返回给MDS。After receiving the second request, the write client processes it through the handle_mds_get_file_size function, obtains the target file size from the local computer, and returns it to the MDS.
步骤406.写客户端发送第二请求响应(MSG_FILE_GET_SIZE_ACK消息给MDS，其中携带目标文件大小及第一请求标识(getattr tid)；Step 406. The write client sends a second request response (MSG_FILE_GET_SIZE_ACK message to the MDS, which carries the target file size and the first request identifier (getattr tid);
步骤407.MDS接收第二请求响应，从中解析出目标文件大小及第一请求标识；Step 407: The MDS receives the second request response and parses the target file size and the first request identifier therefrom;
步骤408.MDS根据第一请求标识找到对应的读客户端的第一请求实例，将目标文件大小size更新到MDS本地缓存，然后打包第一请求响应消息体。Step 408: The MDS finds the first request instance of the corresponding read client according to the first request identifier, updates the target file size to the MDS local cache, and then packages the first request response message body.
步骤409.MDS将第一响应消息发送给读客户端client2，从而使client2获取目标文件的当前大小size。Step 409: The MDS sends a first response message to the read client client2, so that client2 obtains the current size of the target file.
基于图4的示例可以看出，MDS不再如图3示例的步骤那样，在处理读客户端获取目标文件大小getattr的请求时，暂时收回写客户端的写权限(revoke Fw)，从而避免了写客户端写入文件的中断，减小了对写客户端的影响。Based on the example of FIG. 4 , it can be seen that the MDS no longer temporarily revokes the write permission (revoke Fw) of the write client when processing the read client's request to obtain the target file size getattr as in the steps of the example of FIG. 3 , thereby avoiding the interruption of the write client's writing of the file and reducing the impact on the write client.
图5为本发明一实施例提供的实施读写混合场景文件大小获取方法的电子设备结构示意图，该设备500包括：诸如中央处理单元(CPU)的处理器510、通信总线520、通信接口540以及存储器530。其中，处理器510与存储器530可以通过通信总线520相互通信。存储器530内存储有计算机程序，当该计算机程序被处理器510执行时即可实现本发明提供的读写混合场景文件大小获取方法的步骤功能。5 is a schematic diagram of the structure of an electronic device for implementing a method for obtaining the size of a file in a read-write hybrid scenario provided by an embodiment of the present invention, wherein the device 500 includes: a processor 510 such as a central processing unit (CPU), a communication bus 520, a communication interface 540, and a memory 530. The processor 510 and the memory 530 can communicate with each other via the communication bus 520. The memory 530 stores a computer program, and when the computer program is executed by the processor 510, the steps and functions of the method for obtaining the size of a file in a read-write hybrid scenario provided by the present invention can be implemented.
本发明实施例提供的实施读写混合场景文件大小获取方法的计算机程序可分为位于写客户端的部分和位于MDS的部分，运行位于写客户端计算机程序的如图5示例的设备为客户端设备，运行位于MDS部分的计算机程序的如图5示例的设备为MDS服务器。The computer program for implementing the method for obtaining the file size of a mixed read-write scenario provided in an embodiment of the present invention can be divided into a part located at the writing client and a part located at the MDS. The device running the computer program located at the writing client as shown in the example in Figure 5 is a client device, and the device running the computer program located at the MDS part as shown in the example in Figure 5 is an MDS server.
存储器是指基于某种存储介质用于存储计算机程序和/或数据的装置，它可以是易失性存储器(Volatile Memory，VM，常称为内存)，也可以是非易失性存储器(Non-Volatile Memory，NVM)。内存是指与处理器直接交换数据的内部存储器，它可以随时读写数据，而且速度很快，作为操作系统和其它运行程序的临时数据的存储介质。内存可以是同步动态随机存取内存(Synchronous Dynamic Random Access Memory，SDRAM)、动态随机存取内存(Dynamic Random Access Memory，DRAM)等。非易失性存储器是指采用持久化存储介质的存储器，具有容量大和可持久保存数据的特性，可以是存储级存储器(StorageClass Memory，SCM)、固态硬盘(Solid State Disk，SSD)、NAND闪存、磁盘等。SCM是业界对介于内存与闪存之间的新存储介质的统称，是一种同时结合持久化存储特性与内存特性的复合型储存技术，存取速度慢于DRAM快于SSD硬盘。Memory refers to a device based on a certain storage medium for storing computer programs and/or data. It can be a volatile memory (VM, often called memory) or a non-volatile memory (NVM). Memory refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It is used as a storage medium for temporary data of the operating system and other running programs. Memory can be synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), etc. Non-volatile memory refers to a memory that uses a persistent storage medium. It has the characteristics of large capacity and persistent data storage. It can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. SCM is the industry's general term for new storage media between memory and flash memory. It is a composite storage technology that combines persistent storage characteristics and memory characteristics. The access speed is slower than DRAM and faster than SSD hard disks.
处理器可以是通用处理器，包括中央处理器(Central Processing Unit，CPU)、网络处理器(Network Processor，NP)等；还可以是数字信号处理器(Digital SignalProcessing，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
应当认识到，本发明的实施例可以由计算机硬件、硬件和软件的组合、或者通过存储在非暂时性(或称为非持久性)存储器中的计算机指令来实现或实施。所述方法可以使用标准编程技术，包括配置有计算机程序的非暂时性存储介质在计算机程序中实现，其中如此配置的存储介质使得计算机以特定和预定义的方式操作。每个程序可以以高级过程或面向对象的编程语言来实现以与计算机系统通信。然而，若需要，该程序可以以汇编或机器语言实现。在任何情况下，该语言可以是编译或解释的语言。此外，为此目的该程序能够在编程的专用集成电路上运行。此外，可按任何合适的顺序来执行本发明描述的过程的操作，除非本发明另外指示或以其他方式明显地与上下文矛盾。本发明描述的过程(或变型和/或其组合)可在配置有可执行指令的一个或多个计算机系统的控制下执行，并且可作为共同地在一个或多个处理器上执行的代码(例如，可执行指令、一个或多个计算机程序或一个或多个应用)、由硬件或其组合来实现。所述计算机程序包括可由一个或多个处理器执行的多个指令。It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium configured in this way causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.
进一步，所述方法可以在可操作地连接至合适的任何类型的计算平台中实现，包括但不限于个人电脑、迷你计算机、主框架、工作站、网络或分布式计算环境、单独的或集成的计算机平台、或者与带电粒子工具或其它成像装置通信等等。本发明的各方面可以以存储在非暂时性存储介质或设备上的机器可读代码来实现，无论是可移动的还是集成至计算平台，如硬盘、光学读取和/或写入存储介质、RAM、ROM等，使得其可由可编程计算机读取，当存储介质或设备由计算机读取时可用于配置和操作计算机以执行在此所描述的过程。此外，机器可读代码，或其部分可以通过有线或无线网络传输。当此类媒体包括结合微处理器或其他数据处理器实现上文所述步骤的指令或程序时，本发明所述的发明包括这些和其他不同类型的非暂时性计算机可读存储介质。当根据本发明所述的方法和技术编程时，本发明还包括计算机本身。Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and/or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.
以上所述仅为本发明的实施例而已，并不用于限制本发明。对于本领域技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311695791.XACN117850674A (en) | 2023-12-08 | 2023-12-08 | Method, device and equipment for obtaining file size in mixed read-write scenario |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311695791.XACN117850674A (en) | 2023-12-08 | 2023-12-08 | Method, device and equipment for obtaining file size in mixed read-write scenario |
| Publication Number | Publication Date |
|---|---|
| CN117850674Atrue CN117850674A (en) | 2024-04-09 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311695791.XAPendingCN117850674A (en) | 2023-12-08 | 2023-12-08 | Method, device and equipment for obtaining file size in mixed read-write scenario |
| Country | Link |
|---|---|
| CN (1) | CN117850674A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119311658A (en)* | 2024-09-29 | 2025-01-14 | 北京焱融科技有限公司 | A multi-client additional writing processing method and system |
| CN120315656A (en)* | 2025-06-18 | 2025-07-15 | 苏州元脑智能科技有限公司 | Disk file reading and writing method and device, electronic device and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119311658A (en)* | 2024-09-29 | 2025-01-14 | 北京焱融科技有限公司 | A multi-client additional writing processing method and system |
| CN120315656A (en)* | 2025-06-18 | 2025-07-15 | 苏州元脑智能科技有限公司 | Disk file reading and writing method and device, electronic device and storage medium |
| CN120315656B (en)* | 2025-06-18 | 2025-08-22 | 苏州元脑智能科技有限公司 | Method and device for reading and writing disk file, electronic equipment and storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| CN117850674A (en) | Method, device and equipment for obtaining file size in mixed read-write scenario | |
| CN111475483B (en) | Database migration method and device and computing equipment | |
| CN111309720B (en) | Time sequence data storage and reading method and device, electronic equipment and storage medium | |
| EP3076307A1 (en) | Method and device for responding to a request, and distributed file system | |
| CN110765165B (en) | Method, device and system for synchronously processing cross-system data | |
| KR20180008774A (en) | Method and system for implementing indexing when saving files | |
| CN113051221B (en) | Data storage method, device, medium, equipment and distributed file system | |
| CN108460045A (en) | A kind of processing method and distributed block storage system of snapshot | |
| CN114036195B (en) | A data request processing method, device, server and storage medium | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| CN114860432B (en) | A method and device for determining memory failure information | |
| CN114372064B (en) | Data processing apparatus, method, computer readable medium and processor | |
| CN111930684A (en) | Small file processing method, device and equipment based on HDFS (Hadoop distributed File System) and storage medium | |
| CN115442439A (en) | Distributed cache cluster management method, system, terminal and storage medium | |
| CN113326146B (en) | Message processing method and device, electronic equipment and storage medium | |
| CN108241710A (en) | A file creation method and device, and a file query method and device | |
| US20240370430A1 (en) | Data query methods and apparatuses and database systems | |
| CN119397166A (en) | A data request processing method, device, equipment and storage medium | |
| CN106257447A (en) | The video storage of cloud storage server and search method, video cloud storage system | |
| CN112685329B (en) | Method for processing data and related device | |
| CN113626383A (en) | Data processing method, device and equipment | |
| CN118170728A (en) | File merging method and device, electronic equipment and storage medium | |
| US11855868B2 (en) | Reducing the impact of network latency during a restore operation | |
| CN113301173A (en) | Domain name updating system and method, message forwarding method and server | |
| CN113986133B (en) | Data processing method and device, data storage system and electronic equipment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |