CN112667374A - Thread synchronization method, system and medium - Google Patents
Thread synchronization method, system and mediumDownload PDFInfo
- Publication number
- CN112667374A CN112667374ACN202011521685.6ACN202011521685ACN112667374ACN 112667374 ACN112667374 ACN 112667374ACN 202011521685 ACN202011521685 ACN 202011521685ACN 112667374 ACN112667374 ACN 112667374A
- Authority
- CN
- China
- Prior art keywords
- event
- events
- thread
- module
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Hardware Redundancy (AREA)
Abstract
Translated fromChinese
Description
Technical Field
The present invention relates to the field of computer application technologies, and in particular, to a method, a system, and a medium for thread synchronization.
Background
The Rsync is a remote data synchronization tool frequently used in operation and maintenance work, can realize synchronous file transmission in a plurality of hosts, and has the advantages of high safety, rapid backup and the like. In the prior art, an inotify mechanism is added, an inotify is notified by using a message in a Linux kernel, and file synchronization is performed by monitoring the change of file attributes of a file system and mobilizing rsync. Although the technology achieves the effect of real-time synchronization, problems exist, such as that rsync plus inotify is triggered updating, and each file change triggers rsync operation, so that rsync is frequently operated, and bandwidth and server performance are affected.
Disclosure of Invention
The invention mainly solves the technical problems that the operation of the existing file synchronization triggering rsync is frequent and the bandwidth and the server performance are influenced when the operation is frequent.
In order to solve the technical problems, the invention adopts a technical scheme that: a thread synchronization method is provided, which comprises the following steps:
setting an event threshold value and a rsync threshold value;
establishing a monitoring process, monitoring and reading events in a monitoring mode, writing the events in the monitoring mode into an event pool, and judging whether the number of the events in the event pool exceeds the event threshold value;
if the number of the called rsyncs exceeds the threshold value of the rsync, executing synchronous operation, and synchronizing the content in the file system server to the web server;
if the event threshold is not exceeded, the content in the file system server is not synchronized.
Furthermore, a storage mechanism and a filtering mechanism are arranged in the monitoring process;
the monitoring process stores the events in the monitoring mode into an event pool through the storage mechanism;
and the monitoring process creates an exclusion file through the filtering mechanism, filters a filtering event generated in the first time period and sets a filtering priority.
Further, the step of storing the events in the monitoring mode into an event pool comprises:
reading an event in a monitoring mode;
writing the event into an event pool and a log file;
and if the number of the events in the event pool exceeds the event threshold value, sending the events in the event pool to a multithread calling process, and emptying the event pool.
Further, the events in the monitoring mode include a file creation event, a file deletion event, a file modification event, a file movement event, and a file attribute change event.
Further, the setting of the filtering priority is to filter an event before a deletion event of the file; the excluded file is a file which is not synchronized;
the filtering events comprise temporary files and repeated events in the monitoring mode; the first time period is from the beginning of listening to the time when the number of events in the event pool reaches the event threshold.
Furthermore, the multi-thread calling process comprises a main module process and a slave module process;
the main module process creates a sub-thread group, sets the number of sub-threads, receives the events in the event pool, distributes tasks to the sub-threads, the sub-threads call the rsync to execute synchronous operation, and writes an execution result into a log file after the execution is finished, so as to perform thread recovery;
before the synchronous operation is finished, if new synchronous operation needs to be executed, the main module process continues to call the sub-threads to execute the synchronous operation, if all the sub-threads in the sub-thread group are in a synchronous state, the new synchronous operation enters a waiting state, and when any one of the sub-threads is in an asynchronous state, the multi-thread calling process distributes the synchronous operation in the waiting state to the sub-threads in the asynchronous state.
And the slave module process judges whether a record of synchronization failure exists in the log file, if the record of synchronization failure exists, the record of synchronization failure in the log file is read, the rsync is called to execute the synchronization operation again, and if the synchronization failure exists, a record is generated and a failure log file is recorded.
Further, the thread recovery is to release the child thread when the child thread completes one synchronization operation and there is no next synchronization operation allocation.
The present invention also provides a thread synchronization system, comprising: the system comprises a threshold setting module, an event monitoring module, a judging module and a thread scheduling module;
the threshold setting module sets an event threshold and a rsync threshold;
the monitoring event module monitors and reads events in a monitoring mode and stores the monitored events in the monitoring mode;
the judging module judges whether the number of the stored events exceeds the event threshold value;
if the event threshold is exceeded, triggering a thread scheduling module;
and the thread scheduling module calls rsync processes, and when the number of the called rsync processes exceeds the threshold value of the rsync, the thread scheduling module executes synchronization operation to synchronize the content in the file system server to the web server.
The monitoring event module comprises an event module and a filtering module;
the event module stores the monitored events in the monitoring mode;
the filtering module filters the generated repeated events;
the filtering module creates an exclusion file;
the filtering module sets a filtering priority.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a thread synchronization method as described in any one of the above.
The invention has the beneficial effects that:
1. according to the thread synchronization method, the message notification mechanism inotify of the Linux kernel is utilized, so that the synchronization work from the server to the client is completed, the frequent operation of the rsync and the starting of excessive rsync threads are avoided, and the resource consumption of the server is reduced;
2. the thread synchronization system of the invention introduces the monitoring time module to trigger synchronization when reaching a certain number of changing times, thereby avoiding frequent synchronization, adds the thread scheduling module, sets the threshold value of rsync, and avoids the problem that the excessive rsync is started due to excessive triggering synchronization times, and occupies excessive resources to influence the normal operation of the server;
3. the computer-readable storage medium can be used for counting events of the file system by monitoring the file system in real time, can synchronously transmit files, and has high safety and rapid backup.
Drawings
Fig. 1 is a flowchart of a thread synchronization method according to embodiment 1 of the present invention;
fig. 2 is a schematic diagram of a thread synchronization system according to embodiment 1 of the present invention.
Detailed Description
The following detailed description of the preferred embodiments of the present invention, taken in conjunction with the accompanying drawings, will make the advantages and features of the invention easier to understand by those skilled in the art, and thus will clearly and clearly define the scope of the invention.
In the description of the present invention, it should be noted that the term "first" is used for descriptive purposes only and is not to be construed as indicating or implying a relative importance. It should be noted that, in the description of the present invention, rsync is a data mirror backup tool under linux system; inotify is a mechanism for monitoring file system events in a Linux kernel, and Maxdelays refers to the maximum accessible event threshold in this embodiment; inotifyMode is a monitoring mode; excludwile is an excluded file, the web (world Wide web), a global Wide area network, also known as the world Wide web.
The embodiment of the invention comprises the following steps:
example 1:
referring to fig. 1, a method for thread synchronization includes:
s100, setting the maximum event values Maxdelays and thresholds of rsync;
s200, setting a file directory to be monitored in a file system, establishing a monitoring process for monitoring and reading an event in a monitoring mode inotifyMode, wherein the event in the monitoring mode inotifyMode comprises creation, deletion, modification, movement and file attribute change of a file, and 5 modes; writing the message into an event pool for accumulation every monitoring of the event in the monitoring mode, and judging whether the number of the events in the event pool exceeds Maxdelays;
and S300, if the number of the called rsyncs exceeds the rsync threshold value, executing synchronous operation, and synchronizing the main directory content in the file system server to each web server in real time.
Establishing a monitoring progress to monitor and read events in the monitoring mode inotifyMode, wherein the events comprise a storage mechanism and a filtering mechanism;
the storage mechanism stores monitored events, when the monitoring mode inotifyMode is read, when the events meeting the conditions are read, for example, the creation of a file is monitored, the monitored events are returned as the events meeting the conditions, the monitored events are written into an event pool, the monitored events are written into log files logfiles, when the number of the events in the event pool reaches Maxdelays, the events in the event pool are transmitted to a multithread calling process, the events in the event pool are emptied, and the next round of circulation is started.
The filtering mechanism is used for filtering events generated in a first time period, the range of the filtering events comprises temporary files, repeated queues and the like, the repeated queues are events in a monitoring mode inotifyMode which is repeatedly generated, the temporary files are files which are not synchronized, and 5 modes of the monitoring mode inotifyMode are continuously monitored, if a deleted folder is monitored, other events related to the folder are all filtered, and only the deleted events are reserved so as to reduce the synchronization burden; for example, if the creation, modification and deletion are performed first in the folder, the previous creation and modification actions are filtered out, and only the deletion action is retained, that is, the deletion has the highest priority, and all events performed before the deletion are not retained.
Defining to exclude the file excludwile from synchronization in a filtering mechanism, wherein the excludwile includes a temporary file generated when the file is created and a file which is not synchronized; an excluded file is a file created or a temporary file is needed to record the content, in which case it is placed in the excluded file without synchronization
The first time period is the time from the creation of the monitoring process to the time when the time quantity in the event pool reaches Maxdelays;
receiving an event transmitted by an event queue in a multithread calling process, and calling a synchronization module to synchronize a main directory of a file server to each web server in real time;
the multithread calling process comprises a main module process and a slave module process;
the main module process creates a sub-thread group, sets the upper limit threshold value with the most sub-threads, and can create a specified number of sub-threads in the sub-thread group; receiving events in the event pool, distributing tasks to the sub-threads, calling the rsync by the sub-threads to execute synchronous operation, returning an execution result after the execution is finished, logging a log file, and recovering threads.
The child thread is the rsync thread, invoking rsync.
Thread reclamation is when the rsync thread completes one synchronization operation and there is no task allocation for the next synchronization operation, then the rsync thread is released.
Before the synchronous operation is finished, if a new synchronous operation needs to be executed, the main module process continues to call the sub-threads to execute the synchronous action, if all the sub-threads in the sub-thread group are in the synchronous state, the new synchronous operation enters a waiting state, and when any one sub-thread is in the asynchronous state, the multi-thread calling process distributes the synchronous operation in the waiting state to the sub-threads in the asynchronous state.
The slave module process executes secondary synchronous operation: judging whether a record of execution failure exists in the log file logfile, if so, reading the record of execution failure in the log file logfile, calling the rsync tool to execute synchronous operation again, and generating a log file with the record of fail recording if the record of execution failure exists; and if the record of the execution failure does not exist, the slave module does not execute the secondary synchronization operation.
Example 2:
referring to fig. 2, based on the same inventive concept as the thread synchronization method in the foregoing embodiments, an embodiment of the present disclosure further provides a thread synchronization system, including: the system comprises a threshold setting module, an event monitoring module, a judging module and a thread scheduling module;
the threshold value setting module sets the threshold values of the maximum event values Maxdelays and rsync;
the monitoring event module monitors and reads an event in the monitoring mode inotifyMode; and events that are snooped into the monitoring mode are stored,
the judging module judges whether the number of the accumulated events exceeds Maxdelays;
if the current time exceeds the Maxdelays, triggering a thread scheduling module;
and the thread scheduling module calls the rsync processes, and when the number of the called rsync processes exceeds the threshold value of the rsync, the thread scheduling module executes synchronization operation to synchronize the content in the file system server to each web server.
The monitoring event module comprises an event module and a filtering module;
the event module stores the events monitored in the monitoring mode;
the filtering module filters the generated repeated events, continuously monitors the monitoring mode, sets priority, and defines that the exccludleFile is excluded from being synchronized, including temporary files generated when files are created and files which do not need to be synchronized.
Based on the same inventive concept as the method in the foregoing embodiments, the embodiments of the present specification further provide a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the thread synchronization method as disclosed in the foregoing.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, and a program that can be implemented by the hardware and can be instructed by the program to be executed by the relevant hardware may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic or optical disk, and the like.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A thread synchronization method is characterized by comprising the following steps:
setting an event threshold value and a rsync threshold value;
establishing a monitoring process, monitoring and reading events in a monitoring mode, writing the events in the monitoring mode into an event pool, and judging whether the number of the events in the event pool exceeds the event threshold value;
if the number of the called rsyncs exceeds the threshold value of the rsync, executing synchronous operation, and synchronizing the content in the file system server to the web server;
if the event threshold is not exceeded, the content in the file system server is not synchronized.
2. A thread synchronization method according to claim 1, characterized in that: a storage mechanism and a filtering mechanism are arranged in the monitoring process;
the monitoring process stores the events in the monitoring mode into an event pool through the storage mechanism;
and the monitoring process creates an exclusion file through the filtering mechanism, filters a filtering event generated in the first time period and sets a filtering priority.
3. A thread synchronization method according to claim 2, characterized in that: the step of storing the events in the monitoring mode into an event pool comprises the following steps:
reading an event in a monitoring mode;
writing the event into an event pool and a log file;
and if the number of the events in the event pool exceeds the event threshold value, sending the events in the event pool to a multithread calling process, and emptying the event pool.
4. A thread synchronization method according to claim 2 or 3, characterized in that: the events in the monitoring mode comprise a file creating event, a file deleting event, a file modifying event, a file moving event and a file attribute changing event.
5. The thread synchronization method of claim 4, wherein: the set filtering priority is used for filtering an event before a deletion event of the file; the excluded file is a file which is not synchronized;
the filtering events comprise temporary files and repeated events in the monitoring mode; the first time period is from the beginning of listening to the time when the number of events in the event pool reaches the event threshold.
6. A thread synchronization method according to claim 1, characterized in that: the multithread calling process comprises a main module process and a slave module process;
the main module process creates a sub-thread group, sets the number of sub-threads, receives the events in the event pool, distributes tasks to the sub-threads, the sub-threads call the rsync to execute synchronous operation, and writes an execution result into a log file after the execution is finished, so as to perform thread recovery;
before the synchronous operation is finished, if new synchronous operation needs to be executed, the main module process continues to call the sub-threads to execute the synchronous operation, if all the sub-threads in the sub-thread group are in a synchronous state, the new synchronous operation enters a waiting state, and when any one of the sub-threads is in an asynchronous state, the multi-thread calling process distributes the synchronous operation in the waiting state to the sub-threads in the asynchronous state.
And the slave module process judges whether a record of synchronization failure exists in the log file, if the record of synchronization failure exists, the record of synchronization failure in the log file is read, the rsync is called to execute the synchronization operation again, and if the synchronization failure exists, a record is generated and a failure log file is recorded.
7. The thread synchronization method of claim 6, wherein: the thread recovery is to release the child thread when the child thread completes one synchronization operation and there is no next synchronization operation allocation.
8. A thread synchronization system, comprising: the system comprises a threshold setting module, an event monitoring module, a judging module and a thread scheduling module;
the threshold setting module sets an event threshold and a rsync threshold;
the monitoring event module monitors and reads events in a monitoring mode and stores the monitored events in the monitoring mode;
the judging module judges whether the number of the stored events exceeds the event threshold value;
if the event threshold is exceeded, triggering a thread scheduling module;
and the thread scheduling module calls rsync processes, and when the number of the called rsync processes exceeds the threshold value of the rsync, the thread scheduling module executes synchronization operation to synchronize the content in the file system server to the web server.
9. A thread synchronization system as recited in claim 8, wherein: the monitoring event module comprises an event module and a filtering module;
the event module stores the monitored events in the monitoring mode;
the filtering module filters the generated repeated events;
the filtering module creates an exclusion file;
the filtering module sets a filtering priority.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of a thread synchronization method as claimed in any one of the preceding claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011521685.6ACN112667374B (en) | 2020-12-21 | 2020-12-21 | Thread synchronization method, system and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011521685.6ACN112667374B (en) | 2020-12-21 | 2020-12-21 | Thread synchronization method, system and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112667374Atrue CN112667374A (en) | 2021-04-16 |
| CN112667374B CN112667374B (en) | 2023-01-10 |
Family
ID=75407126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011521685.6AActiveCN112667374B (en) | 2020-12-21 | 2020-12-21 | Thread synchronization method, system and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112667374B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114679469A (en)* | 2022-05-26 | 2022-06-28 | 天津联想协同科技有限公司 | Data processing method and device and computer readable storage medium |
| CN114936182A (en)* | 2022-05-20 | 2022-08-23 | 天津电气科学研究院有限公司 | Trigger storage method of electric dynamometer test data based on C# |
| CN117009309A (en)* | 2023-05-31 | 2023-11-07 | 合芯科技(苏州)有限公司 | File real-time synchronization method and device based on rsync |
| CN118152418A (en)* | 2024-05-09 | 2024-06-07 | 麒麟软件有限公司 | Dynamic update method and system of index database based on inotify |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120983A (en)* | 2019-06-14 | 2019-08-13 | 浪潮软件集团有限公司 | The method and system of SVN real-time synchronization backup is realized based on inotify and rsync automation installation and deployment |
| CN111538710A (en)* | 2020-04-23 | 2020-08-14 | 苏州浪潮智能科技有限公司 | Method and system for real-time data synchronization based on rsync |
- 2020
- 2020-12-21CNCN202011521685.6Apatent/CN112667374B/enactiveActive
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120983A (en)* | 2019-06-14 | 2019-08-13 | 浪潮软件集团有限公司 | The method and system of SVN real-time synchronization backup is realized based on inotify and rsync automation installation and deployment |
| CN111538710A (en)* | 2020-04-23 | 2020-08-14 | 苏州浪潮智能科技有限公司 | Method and system for real-time data synchronization based on rsync |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114936182A (en)* | 2022-05-20 | 2022-08-23 | 天津电气科学研究院有限公司 | Trigger storage method of electric dynamometer test data based on C# |
| CN114936182B (en)* | 2022-05-20 | 2025-06-24 | 天津电气科学研究院有限公司 | Trigger storage method of electric dynamometer test data based on C# |
| CN114679469A (en)* | 2022-05-26 | 2022-06-28 | 天津联想协同科技有限公司 | Data processing method and device and computer readable storage medium |
| CN117009309A (en)* | 2023-05-31 | 2023-11-07 | 合芯科技(苏州)有限公司 | File real-time synchronization method and device based on rsync |
| CN117009309B (en)* | 2023-05-31 | 2024-05-07 | 合芯科技(苏州)有限公司 | File real-time synchronization method and device based on rsync |
| CN118152418A (en)* | 2024-05-09 | 2024-06-07 | 麒麟软件有限公司 | Dynamic update method and system of index database based on inotify |
| CN118152418B (en)* | 2024-05-09 | 2024-08-13 | 麒麟软件有限公司 | Index database dynamic updating method and system based on inotify |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112667374B (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112667374A (en) | Thread synchronization method, system and medium | |
| US6990606B2 (en) | Cascading failover of a data management application for shared disk file systems in loosely coupled node clusters | |
| CN109101341B (en) | Distribution method and equipment of distributed lock | |
| US8458422B1 (en) | Policy based creation of export sets and backup media | |
| JP5021929B2 (en) | Computer system, storage system, management computer, and backup management method | |
| US20080184063A1 (en) | System and Method of Error Recovery for Backup Applications | |
| EP2940596B1 (en) | Data acquisition method and device | |
| CN110895484A (en) | Task scheduling method and device | |
| JP2007140700A (en) | Computer system, management computer, storage system, and backup management method | |
| US20050004952A1 (en) | Transaction processing method, transaction control apparatus and program thereof | |
| CN111538710A (en) | Method and system for real-time data synchronization based on rsync | |
| JP3951835B2 (en) | Business management method and business processing system | |
| US20110282843A1 (en) | Method and system for data backup and replication | |
| CN109697112B (en) | Distributed intensive one-stop operating system and implementation method | |
| CN101471810B (en) | Method, device and system for implementing task in cluster circumstance | |
| CN110895486A (en) | Distributed Task Scheduling System | |
| CN110895487A (en) | Distributed task scheduling system | |
| US8001339B1 (en) | Storage management policies in a virtual data storage system | |
| CN111897626A (en) | Cloud computing scene-oriented virtual machine high-reliability system and implementation method | |
| CN114328715A (en) | Data synchronization method and device and computer readable storage medium | |
| US8087021B1 (en) | Automated activity processing | |
| CN116582453B (en) | Monitoring data migration method and system in multi-service cluster scene | |
| CN113901141B (en) | Distributed data synchronization method and system | |
| CN117850998A (en) | Distributed task scheduling framework based on Quartz and application thereof | |
| CN113672341B (en) | A method, system, device and storage medium for improving k8s cluster stability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | Address after:215124 Jiangsu Province, Suzhou City, Wuzhong Economic Development Zone, Guoqiang Street, Guanpu Road No. 1, Building 9 Patentee after:Suzhou Yuannao Intelligent Technology Co.,Ltd. Country or region after:China Address before:215124 Jiangsu Province, Suzhou City, Wuzhong Economic Development Zone, Guoqiang Street, Guanpu Road No. 1, Building 9 Patentee before:SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Country or region before:China |