CN112346663A

Movatterモバイル変換

Info

Publication number: CN112346663A
Application number: CN202011352910.8A
Authority: CN
Inventors: 唐俊杰; 蔡晓华; 杨光辉
Original assignee: Shanghai Netis Technologies Co ltd
Current assignee: Shanghai Netis Technologies Co ltd
Priority date: 2020-11-26
Filing date: 2020-11-26
Publication date: 2021-02-09
Anticipated expiration: 2040-11-26
Also published as: CN112346663B

Abstract

The invention provides a data writing and clearing method and a system for multi-partition rotation, which comprises the following steps: preparing a main partition, and mounting the main partition to a first directory; mounting all partitions of the disk to a second directory respectively; selecting a partition in which data is written last as a current partition, and selecting a preset partition as the current partition by a disk started for the first time; detecting the usage amount of the current partition; judging whether the usage amount is larger than a preset threshold value or not, if so, entering a first operation step, and if not, entering a second operation step; a first operation step: formatting and re-mounting a next partition of the current partition to be used as a new current partition; a second operation step: and linking the data needing to be written from the main partition to the current partition in a file soft connection or directory soft connection mode, and redirecting the real position of the written data from the main partition to the current partition. The invention can greatly reduce the disk load caused by deleting the old data.

Description

Multi-partition rotation data writing and clearing method and system

Technical Field

The invention relates to the technical field of data storage, in particular to a multi-partition rotation data writing and clearing method and system.

Background

In many program systems, a large amount of new data needs to be written to disk, while a large amount of old data needs to be deleted in the disk file system. Repeated data write and delete operations can result in fragmentation of the file system, which can result in reduced write performance and increased latency of the disk. In a relatively real-time system, it is also impossible to defragment a file system having a large number of read and write disk requests.

The existing solutions are as follows:

1) and (4) performing partition arrangement at the time of a low flow (such as a non-working period). Obviously, the method has limited application scenes and low system efficiency. Meanwhile, if the partition arrangement time is too long, the normal working time period is influenced.

2) File systems are allocated in advance, and the writing and fragmentation of files are managed by themselves. Obviously, the method has high development cost and poor compatibility. Meanwhile, various optimization mechanisms of OS and RAID card hardware cannot be utilized, and sequential reading and writing and randomness cannot be well processed simultaneously.

Patent document CN103474087A discloses a management method and device for simply preventing disk fragments from generating video files, which formats and fixes the disk in a video recorder to one or several partitions, and uses different formatting methods provided by the operating system for different file systems; establishing a video file index database and providing a video file retrieval service; fixing the size of the video file, and recording as A (Byte); when the disk is not full, adopting a general file creating and writing mode; and after the disk is fully written, finding out the file with the oldest creation time on the disk, taking the file as a new video file, overwriting the file from the initial position of the file, and repeating the steps. The scheme needs to fix the size of the video file, needs an additional video file index database, adopts a mode of repeatedly using the old file to prevent file fragments, and is complex in implementation mode.

Disclosure of Invention

In view of the defects in the prior art, the present invention provides a method and a system for writing and clearing data in a multi-partition rotation.

The invention provides a data writing and clearing method for multi-partition rotation, which comprises the following steps:

main partition preparation: preparing a main partition, and mounting the main partition to a first directory;

a partition preparation step: mounting all partitions of the disk to a second directory respectively;

current partition selection: selecting a partition in which data is written last as a current partition, and selecting a preset partition as the current partition by a disk started for the first time;

and (3) usage amount detection: detecting the usage amount of the current partition;

and (3) judging the dosage: judging whether the usage amount is larger than a preset threshold value or not, if so, entering a first operation step, and if not, entering a second operation step;

a first operation step: formatting and re-mounting a next partition of the current partition to be used as a new current partition;

a second operation step: and linking the data needing to be written from the main partition to the current partition in a file soft connection or directory soft connection mode, and redirecting the real position of the written data from the main partition to the current partition.

Preferably, the main partition is on the same disk as the partition, or on a different disk.

Preferably, after the first operating step or the second operating step is performed, the method further includes performing:

CPU abandoning step: and giving up the CPU preset time, and returning to the usage detection step.

Preferably, the first operating step comprises:

unloading the next partition of the current partition, formatting and then re-mounting;

the next partition of the current partition is designated as the new current partition.

Preferably, the second operating step comprises:

calculating the file name or directory name of data to be written;

linking data needing to be written from a main partition to a current partition in a file soft connection or directory soft connection mode;

the real location of the written data is redirected by the main partition to the current partition.

The invention provides a multi-partition rotating data writing and clearing system, which comprises:

a main partition preparation module: preparing a main partition, and mounting the main partition to a first directory;

a partition preparation module: mounting all partitions of the disk to a second directory respectively;

a current partition selection module: selecting a partition in which data is written last as a current partition, and selecting a preset partition as the current partition by a disk started for the first time;

a usage detection module: detecting the usage amount of the current partition;

a dosage judgment module: judging whether the usage amount is larger than a preset threshold value or not, if so, entering a first operation module, and if not, entering a second operation module;

a first operation module: formatting and re-mounting a next partition of the current partition to be used as a new current partition;

a second operation module: and linking the data needing to be written from the main partition to the current partition in a file soft connection or directory soft connection mode, and redirecting the real position of the written data from the main partition to the current partition.

Preferably, after executing the first operation module or the second operation module, the method further includes executing:

a CPU abandon module: and giving up the CPU preset time, and returning to the usage detection module.

Preferably, the first operating module includes:

Preferably, the second operation module includes:

calculating the file name or directory name of data to be written;

Compared with the prior art, the invention has the following beneficial effects:

1) the invention can greatly reduce the disk load caused by deleting the old data.

2) The invention solves the problems of reduced read-write performance and increased delay of the disk caused by fragmentation of the file system.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:

FIG. 1 is a flow chart of the operation of the present invention.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.

The invention provides a scheme of multi-partition rotation, and old data is cleaned in a formatted partition mode, so that the disk load caused by deleting the old data is greatly reduced, and the problems of reduced disk read-write performance and increased delay caused by fragmentation of a file system are solved. The small partitions are divided and stored on a large whole disk, so that the number of partition inodes can be reduced, and the IO retrieval speed of the file can be increased.

Referring to fig. 1, the data writing and clearing method for multi-partition rotation includes the following steps:

step 1: a main partition (which may or may not be on the same physical disk as the "n disk partitions" described in step 2) is prepared in advance. The main partition is the destination file system to which the external program was originally prepared to write files. Suppose the main partition is mounted to/main/path by mount.

Step 2: n disk partitions with the numbers of 1- > n are divided on a disk in advance, and the n partition mount is placed in a pre-designated directory during starting. Such as/dev/sdb 1,/dev/sdb2 …/dev/sdbn mount to/mnt/dir 1,/mnt/dir2 …/mnt/dirn, respectively; (the main partition may be on the disk or may be another disk).

And step 3: and selecting the disk partition written last time the program is exited last time as the current partition. If started for the first time, partition 1 may be selected as the current partition.

And 4, step 4: and detecting the disk usage amount of the current partition.

And 5: and judging whether the disk usage is larger than or equal to a specified threshold (such as 80%), if so, executing the step 6, otherwise, executing the step 7.

Step 6: executing the operation that the usage of the disk is more than or equal to a specified threshold; after the execution is finished, step 8 is executed.

And 7: executing the operation that the usage of the disk is less than a specified threshold; after the execution is finished, step 8 is executed.

And 8: the program gives up the CPU for several seconds, waits for the next detection operation, and returns to step 4.

Wherein, the step 6 comprises the following steps:

step 6.1: and unloading the next partition of the current partition from the system umount, and mounting the mount again after formatting.

Step 6.2: the next partition of the current partition is designated as the new current partition.

The step 7 comprises the following steps:

step 7.1: and calculating the file name required to be written by the next external program or the directory name required to be written into the path. This step requires that the file names or directory names of the next set of files to be written by the external program are predictable, typically in a video recording system that performs file segmentation according to a fixed time length;

step 7.2: and linking the next batch of files to be written from the main partition to the current partition in a file soft link or directory soft link mode. Assuming that the external program writes the file directory split by hours, i.e. writes/main/path/2019072013, then the/main/path/2019072013 directory is linked to/mnt/dirx/2019072013 in a soft link manner;

step 7.3: the external program writes the actual write disk location of the file, which is redirected to the current partition by the main partition. I.e., from the main partition where/main/path/2019072013 is located, to the/dev/sdbx partition where/mnt/dirx/2019072013 is located.

Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.

In the description of the present application, it is to be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience in describing the present application and simplifying the description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present application.

The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims

1. A multi-partition rotation data writing and clearing method is characterized by comprising the following steps:

2. The multi-partition rotated data write and purge method of claim 1, wherein said main partition is on the same disk as said partition or on a different disk.

3. The method for writing and clearing data of multi-partition rotation according to claim 1, further comprising performing, after the performing the first operating step or the second operating step:

4. The multi-partition round robin data writing and clearing method of claim 1, wherein the first operating step comprises:

5. The multi-partition round robin data writing and clearing method of claim 1, wherein the second operating step comprises:

calculating the file name or directory name of data to be written;

6. A multi-partition rotating data write and purge system, comprising:

a usage detection module: detecting the usage amount of the current partition;

7. The multi-partition rotated data write and purge system of claim 6, wherein the main partition is on the same disk as the partition or on a different disk.

8. The multi-partition rotated data writing and clearing system of claim 6, further comprising performing, after executing the first operation module or the second operation module:

9. The multi-partition rotated data write and purge system of claim 6, wherein the first operation module comprises:

10. The multi-partition rotated data write and purge system of claim 6, wherein said second operation module comprises:

calculating the file name or directory name of data to be written;