CN115470015A - Inter-process large-scale data transmission method based on shared memory - Google Patents

Abstract

The invention provides a shared memory-based interprocess large-scale data transmission method, which belongs to the technical field of distributed databases and ensures high speed of data transmission through a shared memory and controllable data transmission through creating a message queue in the shared memory. The bottom layer of the message queue is realized through a ring array ringbuffer, and the concurrency efficiency is ensured through atomic operation.

Description

Translated fromChinese

一种基于共享内存的进程间大规模数据传输方法A large-scale data transfer method between processes based on shared memory

技术领域technical field

本发明涉及分布式数据库技术领域，尤其涉及一种基于共享内存的进程间大规模数据传输方法。The invention relates to the technical field of distributed databases, in particular to a large-scale data transmission method between processes based on shared memory.

背景技术Background technique

对于面向物联网的分布式数据库，数据流量非常庞大，需要承受的数据传输压力也非常大。对于在同一台机器上的不同进程，如果采用本地socket或者管道等方式进行数据传输，其效率都比不上通过共享内存传输。而单纯使用共享内存，又无法有序组织起大规模数据；同时两个进程的多个线程同时访问共享内存，也会带来严重的并发问题，使用常规的锁或者信号量处理并发也会大大降低并发效率。For distributed databases oriented to the Internet of Things, the data traffic is very large, and the pressure of data transmission that needs to be endured is also very high. For different processes on the same machine, if local sockets or pipelines are used for data transmission, the efficiency is not as efficient as that of shared memory transmission. However, simply using shared memory cannot organize large-scale data in an orderly manner; at the same time, multiple threads of two processes access shared memory at the same time, which will also bring serious concurrency problems, and using conventional locks or semaphores to process concurrency will greatly increase Reduce concurrency efficiency.

发明内容Contents of the invention

为了解决以上技术问题，本发明提供了一种基于共享内存的进程间大规模数据传输方法。这一方法扩展了常规的socket，管道等进程间通信方法，充分利用进程处于同一台机器上的共享内存的高效性能提高数据传输的效率；同时可以解决多进程以及多线程同时访问共享内存带来的并发问题，允许多进程和多线程的并发访问，通过lock-free的逻辑避免对进程间mutex的依赖，同时减少资源占用，提高并发效率。在保证并发安全的基础上提供数据有序高速传输的服务，实现大规模有序数据的高速、安全传输In order to solve the above technical problems, the present invention provides a large-scale data transmission method between processes based on shared memory. This method expands the conventional inter-process communication methods such as sockets and pipes, and makes full use of the high-efficiency performance of the shared memory of the process on the same machine to improve the efficiency of data transmission; at the same time, it can solve the problems caused by multiple processes and multiple threads accessing the shared memory at the same time. Concurrency issues, allowing concurrent access by multiple processes and threads, avoiding dependence on mutex between processes through lock-free logic, while reducing resource usage and improving concurrency efficiency. Provide orderly and high-speed data transmission services on the basis of ensuring concurrent security, and realize high-speed and safe transmission of large-scale orderly data

本发明专门用于优化分布式数据库中同一台机器上的数据传输效率，提供安全，高速，有序的数据传输。The invention is specially used to optimize the data transmission efficiency on the same machine in the distributed database, and provide safe, high-speed and orderly data transmission.

本发明的技术方案是：Technical scheme of the present invention is:

一种基于共享内存的进程间大规模数据传输方法，使用基于共享内存的ringbuffer消息队列进行进程间数据传输。A large-scale data transmission method between processes based on shared memory, using a ringbuffer message queue based on shared memory for inter-process data transmission.

进一步的，further,

首先，在机器上创建共享内存，提供给一个以上进程可以同时访问；First, create shared memory on the machine, which can be accessed by more than one process at the same time;

其次，在共享内存内创建用于通信的消息队列；消息队列底层用ringbuffer实现；Secondly, create a message queue for communication in the shared memory; the bottom layer of the message queue is implemented with ringbuffer;

然后，采用lock-free的方式处理并发问题；Then, use the lock-free method to deal with concurrency issues;

最后，消息队列负责传递数据在共享内存中的偏移，无需拷贝。把待传输的数据存放到共享内存中，即可通过消息队列进行传输。Finally, the message queue is responsible for passing the offset of the data in the shared memory without copying. Store the data to be transmitted in the shared memory, and then transmit it through the message queue.

其中，in,

工作逻辑采用先入先出的方式，队列一端用于发送数据，另一端用于接收数据。The working logic adopts the first-in first-out method, one end of the queue is used to send data, and the other end is used to receive data.

在消息队列中，使用原子变量实现的自旋锁锁住队列的发送端和接收端。In the message queue, the sender and receiver of the queue are locked using the spin lock implemented by the atomic variable.

消息队列采用原子变量实现多进程和多线程的并发安全。The message queue uses atomic variables to realize the concurrent safety of multi-process and multi-thread.

再进一步的，Going one step further,

数据发送的过程如下：The process of sending data is as follows:

1)发送进程将数据的指针传递给消息队列；1) The sending process passes the data pointer to the message queue;

2)消息队列检查写自旋锁是否可以获得，如果未获得则自旋获取；2) The message queue checks whether the write spin lock can be obtained, and if it is not obtained, it will be obtained by spin;

3)消息队列获得了写自旋锁，检查消息队列是否有空间，若有空间则将数据地址写入队列；3) The message queue obtains a write spin lock, checks whether there is space in the message queue, and writes the data address into the queue if there is space;

4)消息队列修改自身状态，释放写自旋锁；4) The message queue modifies its own state and releases the write spin lock;

5)返回操作结果。5) Return the operation result.

数据接收的过程如下：The process of data reception is as follows:

1)接收进程访问消息队列；1) The receiving process accesses the message queue;

2)消息队列检查读自旋锁是否可以获得，如果未获得则自旋获取；2) The message queue checks whether the read spin lock can be obtained, and if it is not obtained, it will be obtained by spin;

3)消息队列获得了读自旋锁，检查消息队列是否有数据，若有则获取最前面的数据；3) The message queue acquires a read spin lock, checks whether there is data in the message queue, and if so, obtains the frontmost data;

4)消息队列修改自身状态，释放读自旋锁；4) The message queue modifies its own state and releases the read spin lock;

5)返回数据。5) Return data.

本发明的有益效果是The beneficial effect of the present invention is

通过结合共享内存，ringbuffer消息队列和原子变量的优秀特性，提升了分布式数据库中一个节点上数据交换的效率，同时兼顾了并发安全和数据有序，可以大大提升数据的流量和数据系统的规模。By combining the excellent characteristics of shared memory, ringbuffer message queue and atomic variables, the efficiency of data exchange on a node in the distributed database is improved, and at the same time, concurrent security and data order are taken into account, which can greatly improve the data flow and the scale of the data system .

附图说明Description of drawings

图1是本发明的工作示意图。Fig. 1 is the working schematic diagram of the present invention.

具体实施方式detailed description

为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例，基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection of the present invention. scope.

本发明提供了一种基于共享内存的进程间大规模数据传输方法。使用基于共享内存的ringbuffer消息队列进行进程间数据传输。The invention provides a large-scale data transmission method between processes based on shared memory. Use the shared memory-based ringbuffer message queue for inter-process data transfer.

进一步的，further,

首先，在机器上创建共享内存，提供给多个进程可以同时访问，共享内存的高效保证了高速传输的基础。First, a shared memory is created on the machine, which can be accessed by multiple processes at the same time. The high efficiency of the shared memory guarantees the basis for high-speed transmission.

其次，在共享内存内创建用于通信的消息队列。消息队列底层用ringbuffer实现，占用有限的内存实现无限的数据的传递。工作逻辑采用先入先出的方式，队列一端用于发送数据，另一端用于接收数据，既能减少相互的干扰，也能保证数据传递的有序性。更早发送的数据也会更早被接收，这对于时序数据等对顺序要求较高的数据十分有利。Second, create a message queue for communication within the shared memory. The bottom layer of the message queue is implemented with ringbuffer, which occupies limited memory to achieve unlimited data transmission. The working logic adopts the first-in-first-out method. One end of the queue is used to send data, and the other end is used to receive data, which can not only reduce mutual interference, but also ensure the orderliness of data transmission. Data sent earlier will also be received earlier, which is very beneficial for data that requires a higher order, such as timing data.

然后，本方案采用lock-free的方式处理并发问题。在消息队列中，使用原子变量锁住队列的发送端和接收端。原子变量是硬件级别的，可以在进程之间保证并发的安全性，比一般的mutex具有更好的跨平台性，更安全。同时原子变量占用资源少，操作速度也更快。本方案通过原子变量实现自旋锁，比mutex更快速。Then, this solution uses a lock-free approach to deal with concurrency issues. In a message queue, an atomic variable is used to lock the sender and receiver of the queue. Atomic variables are at the hardware level, which can ensure the safety of concurrency between processes, and are better cross-platform and safer than ordinary mutexes. At the same time, atomic variables occupy less resources and operate faster. This solution implements spin locks through atomic variables, which is faster than mutex.

依赖消息队列和原子变量自旋锁，本方案可以实现数据的高速、有序、并发安全的传输。同时也可以实现数据的阻塞和非阻塞的传递，阻塞和非阻塞的接收，满足数据传输各种场景的需求。Relying on message queues and atomic variable spin locks, this solution can achieve high-speed, orderly, concurrent and safe data transmission. At the same time, it can also realize blocking and non-blocking data transmission, blocking and non-blocking reception, and meet the needs of various data transmission scenarios.

最后，消息队列只负责传递数据在共享内存中的偏移，无需拷贝。只需要把待传输的数据存放到共享内存中，就可以通过消息队列进行高效传输。Finally, the message queue is only responsible for transferring the offset of the data in the shared memory without copying. You only need to store the data to be transmitted in the shared memory, and then it can be efficiently transmitted through the message queue.

数据发送的过程如下：The process of sending data is as follows:

1)发送进程将数据的指针传递给消息队列；1) The sending process passes the data pointer to the message queue;

2)消息队列检查写自旋锁是否可以获得，如果未获得则自旋获取；2) The message queue checks whether the write spin lock can be obtained, and if it is not obtained, it will be obtained by spin;

3)消息队列获得了写自旋锁，检查消息队列是否有空间，若有空间则将数据地址写入队列；3) The message queue obtains a write spin lock, checks whether there is space in the message queue, and writes the data address into the queue if there is space;

4)消息队列修改自身状态，释放写自旋锁；4) The message queue modifies its own state and releases the write spin lock;

5)返回操作结果。5) Return the operation result.

数据接收的过程如下：The process of data reception is as follows:

1)接收进程访问消息队列；1) The receiving process accesses the message queue;

2)消息队列检查读自旋锁是否可以获得，如果未获得则自旋获取；2) The message queue checks whether the read spin lock can be obtained, and if it is not obtained, it will be obtained by spin;

3)消息队列获得了读自旋锁，检查消息队列是否有数据，若有则获取最前面的数据；3) The message queue acquires a read spin lock, checks whether there is data in the message queue, and if so, obtains the frontmost data;

4)消息队列修改自身状态，释放读自旋锁；4) The message queue modifies its own state and releases the read spin lock;

5)返回数据。5) Return data.

本发明的有益效果是The beneficial effect of the present invention is

通过结合共享内存，ringbuffer消息队列和原子变量的优秀特性，提升了分布式数据库中一个节点上数据交换的效率，同时兼顾了并发安全和数据有序，可以大大提升数据的流量和数据系统的规模。By combining the excellent characteristics of shared memory, ringbuffer message queue and atomic variables, the efficiency of data exchange on a node in the distributed database is improved, and at the same time, concurrent security and data order are taken into account, which can greatly improve the data flow and the scale of the data system .

以上所述仅为本发明的较佳实施例，仅用于说明本发明的技术方案，并非用于限定本发明的保护范围。凡在本发明的精神和原则之内所做的任何修改、等同替换、改进等，均包含在本发明的保护范围内。The above descriptions are only preferred embodiments of the present invention, and are only used to illustrate the technical solution of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (8)

1. A method for transmitting large-scale data among processes based on shared memory is characterized in that,

and carrying out interprocess data transmission by using a shared memory-based ringbuffer message queue.

2. The method of claim 1,

firstly, establishing a shared memory on a machine, and providing the shared memory for more than one process to access simultaneously;

secondly, creating a message queue for communication in the shared memory; the bottom layer of the message queue is realized by using a ringbuffer;

then, a lock-free mode is adopted to process concurrency problems;

finally, the message queue is responsible for transferring the offset of the data in the shared memory without copying.

3. The method of claim 2,

the working logic adopts a first-in first-out mode, one end of the queue is used for sending data, and the other end of the queue is used for receiving data.

4. The method of claim 3,

in a message queue, a spin lock implemented using atomic variables locks the sending and receiving ends of the queue.

5. The method of claim 4,

the message queue adopts atomic variables to realize the concurrency safety of multiple processes and multiple threads.

6. The method of claim 5,

and storing the data to be transmitted into the shared memory, and transmitting the data through the message queue.

7. The method of claim 6,

the data transmission process is as follows:

1) The sending process transmits the pointer of the data to a message queue;

2) The message queue checks whether the write spin lock can be obtained, and if not, the spin lock is obtained;

3) The message queue obtains a write spin lock, whether the message queue has a space is checked, and if the message queue has the space, the data address is written into the queue;

4) The message queue modifies the state of the message queue and releases the write spin lock;

5) And returning an operation result.

8. The method of claim 7,

the data reception process is as follows:

1) Receiving a process access message queue;

2) The message queue checks whether the read spin lock can be obtained, and if not, the spin lock is obtained;

3) The message queue obtains a read spin lock, whether the message queue has data or not is checked, and if the message queue has the data, the foremost data is obtained;

4) The message queue modifies the state of the message queue and releases the read spin lock;

5) And returning the data.

Images