技术领域Technical Field
本发明涉及计算机技术领域，尤其涉及一种自动导引运输车数据采集的方法和装置。The present invention relates to the field of computer technology, and in particular to a method and device for collecting data of an automatic guided transport vehicle.
背景技术Background Art
物流仓储系统部署了大量自动导引运输车(AGV)的仓储系统，相较于传统的WMS、WCS软件系统在业务场景上存在巨大的差异，这些差异集中体现在：高频无线数据传输、海量的传输数据、中心化数据存储要求。这三点主要差异直接导致AGV设备数据实时采集面临较大挑战；如何在不影响生产业务的情况下保证数据传输的可靠性、采集数据的高可用、海量数据中心后存储等要求。The logistics warehousing system deploys a large number of Automated Guided Vehicles (AGV) warehousing systems. Compared with traditional WMS and WCS software systems, there are huge differences in business scenarios. These differences are mainly reflected in: high-frequency wireless data transmission, massive transmission data, and centralized data storage requirements. These three major differences directly lead to greater challenges in the real-time data collection of AGV equipment; how to ensure the reliability of data transmission, high availability of collected data, and massive data center back-end storage without affecting production business.
对于上述问题，现有技术是通过设备心跳数据与控制台实现本地数据传输，然后由控制台将数据写入MYSQL数据表或者日志的形式进行存储。最后，由统一的抽数工具定时批量的将数据抽取到数据中心进行整合。For the above problem, the existing technology is to realize local data transmission through the device heartbeat data and the console, and then the console writes the data into the MYSQL data table or log for storage. Finally, a unified extraction tool extracts the data in batches to the data center for integration.
在上述技术实现过程中，发现存在以下问题：频繁的心跳交互会显著增加AGV设备与控制台通讯并发量，加剧网络负载的压力和占用网络带宽。网络负载和处理器负载的增加会增加业务系统运行风险，降低系统稳定性，并且网络异常可能导致传感器数据丢失和数据重复的问题，影响数据可用性。传统的MYSQL业务表需要预先设计好业务表结构和索引，海量的(万条/秒)传感器数据连续大量的写入MYSQL会对MYSQL带来大量的存储和查询压力产生性能问题；这可能导致数据延迟，数据实时性难以得到保证。写日志的方式会增加本地磁盘存储压力，需要及时关注磁盘空间大小防止磁盘写满造成程序崩溃，日志数据解析难度大，导致数据可用性降低。于企业级大数据平台往往部署在IDC(数据中心)机房，而仓储服务器零散分布在全国各处。数据中心化存储需要将各仓存储的MYSQL表或日志同步到IDC。目前有成熟的抽数工具以定时批量抽数的方式同步处理，经过此方法同步的中心化数据已经丧失了时效性。In the process of implementing the above technology, the following problems were found: Frequent heartbeat interactions will significantly increase the number of concurrent communications between AGV equipment and the console, increase the pressure of network load and occupy network bandwidth. The increase in network load and processor load will increase the risk of business system operation and reduce system stability. In addition, network anomalies may cause sensor data loss and data duplication, affecting data availability. Traditional MYSQL business tables require pre-designed business table structures and indexes. The continuous and large-scale writing of massive (10,000/second) sensor data to MYSQL will bring a lot of storage and query pressure to MYSQL and cause performance problems; this may cause data delays and data real-time performance is difficult to ensure. The way of writing logs will increase the storage pressure of local disks. It is necessary to pay attention to the size of disk space in time to prevent the disk from being full and causing program crashes. The difficulty of parsing log data is high, resulting in reduced data availability. Enterprise-level big data platforms are often deployed in IDC (data center) computer rooms, while storage servers are scattered throughout the country. Data centralized storage requires synchronizing MYSQL tables or logs stored in each warehouse to IDC. At present, there are mature extraction tools that synchronize data in a timed batch extraction manner. The centralized data synchronized by this method has lost its timeliness.
发明内容Summary of the invention
有鉴于此，本发明实施例提供一种自动导引运输车数据采集的方法和装置，能够批量传输自动导引运输车数据，进而降低了网络负载。并且，提高了数据的高可用性。分布式消息队列具有很高的吞吐量，在流式传输环境下，报文数据可以不落地地直接发送回IDC，节省了处理器资源。并且，可以以数据流的方式实时的将数据采集到IDC，由分布式消息队列集群进行管理，使得数据的实时性得到了巨大的提升。In view of this, an embodiment of the present invention provides a method and device for collecting data of an automated guided vehicle, which can transmit data of an automated guided vehicle in batches, thereby reducing the network load. In addition, the high availability of data is improved. The distributed message queue has a very high throughput. In a streaming transmission environment, the message data can be sent directly back to the IDC without landing, saving processor resources. In addition, data can be collected to the IDC in real time in the form of a data stream and managed by a distributed message queue cluster, which greatly improves the real-time performance of the data.
为实现上述目的，根据本发明实施例的一个方面，提供了一种自动导引运输车数据采集的方法。To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for collecting data of an automated guided vehicle is provided.
本发明实施例的自动导引运输车数据采集的方法包括：基于用户数据报协议，接收自动导引运输车批量发送的数据；根据所述数据的防重码，对所述数据进行去重处理；将去重处理后的数据发送至分布式消息队列。The method for collecting data of an automated guided transport vehicle in an embodiment of the present invention includes: receiving data sent in batches by an automated guided transport vehicle based on a user datagram protocol; deduplicating the data according to an anti-duplication code of the data; and sending the deduplicated data to a distributed message queue.
可选地，将去重处理后的数据发送至分布式消息队列的步骤包括：根据传输需求建立的网关服务；基于所述网关服务，将去重处理后的数据发送至分布式消息队列。Optionally, the step of sending the deduplicated data to the distributed message queue includes: establishing a gateway service according to transmission requirements; and sending the deduplicated data to the distributed message queue based on the gateway service.
可选地，将去重处理后的数据发送至分布式消息队列的步骤包括：对去重处理后的数据补充静态维度数据；将补充后的数据发送至分布式消息队列。Optionally, the step of sending the deduplicated data to the distributed message queue includes: supplementing the deduplicated data with static dimension data; and sending the supplemented data to the distributed message queue.
可选地，在将去重处理后的数据发送至分布式消息队列之前，还包括：根据预设的分类规则对所述数据进行分类；根据所述分类的结果，对所述数据添加标识信息。Optionally, before sending the deduplicated data to the distributed message queue, the method further includes: classifying the data according to a preset classification rule; and adding identification information to the data according to a result of the classification.
可选地，在将去重处理后的数据发送至分布式消息队列之后，还包括：所述分布式消息队列以单线程的方式接收数据；所述分布式消息队列根据接收到的数据的标识信息进行分类；所述分布式消息队列以日志的格式将分类后的数据顺序写入磁盘，并且记录所述数据的偏移量信息。Optionally, after sending the deduplicated data to the distributed message queue, it also includes: the distributed message queue receives data in a single-threaded manner; the distributed message queue classifies the received data according to identification information of the received data; the distributed message queue writes the classified data sequentially to the disk in a log format, and records the offset information of the data.
为实现上述目的，根据本发明实施例的另一个方面，提供了一种自动导引运输车数据采集的装置。To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for collecting data of an automated guided vehicle is provided.
本发明实施例的自动导引运输车数据采集的装置包括：接收模块，用于基于用户数据报协议，接收自动导引运输车批量发送的数据；去重模块，用于根据所述数据的防重码，对所述数据进行去重处理；发送模块，用于将去重处理后的数据发送至分布式消息队列。The device for collecting data of an automatic guided transport vehicle in an embodiment of the present invention includes: a receiving module, which is used to receive data sent in batches by the automatic guided transport vehicle based on the user datagram protocol; a deduplication module, which is used to deduplicate the data according to the anti-duplicate code of the data; and a sending module, which is used to send the deduplicated data to a distributed message queue.
可选地，所述发送模块，还用于根据传输需求建立的网关服务；基于所述网关服务，将去重处理后的数据发送至分布式消息队列。Optionally, the sending module is also used to establish a gateway service according to transmission requirements; based on the gateway service, the deduplicated data is sent to a distributed message queue.
可选地，所述发送模块，还用于对去重处理后的数据补充静态维度数据；并且，将补充后的数据发送至分布式消息队列。Optionally, the sending module is further used to supplement the deduplicated data with static dimension data; and send the supplemented data to the distributed message queue.
可选地，本发明实施例的自动导引运输车数据采集的装置还包括标识信息添加模块，用于根据预设的分类规则对所述数据进行分类；并且，根据所述分类的结果，对所述数据添加标识信息。Optionally, the device for collecting data of an automated guided vehicle of an embodiment of the present invention further includes an identification information adding module, which is used to classify the data according to preset classification rules; and, based on the classification result, add identification information to the data.
可选地，所述分布式消息队列以单线程的方式接收数据；所述分布式消息队列根据接收到的数据的标识信息进行分类；所述分布式消息队列以日志的格式将分类后的数据顺序写入磁盘，并且记录所述数据的偏移量信息。Optionally, the distributed message queue receives data in a single-threaded manner; the distributed message queue classifies the received data according to identification information of the data; the distributed message queue writes the classified data sequentially to the disk in a log format, and records the offset information of the data.
为实现上述目的，根据本发明实施例的再一个方面，提供了一种电子设备。To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.
本发明实施例的电子设备包括：一个或多个处理器；存储装置，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现上述任一项的自动导引运输车数据采集的方法。The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned methods for collecting data of an automated guided vehicle.
为实现上述目的，根据本发明实施例的再一个方面，提供了一种计算机可读介质，其上存储有计算机程序，其特征在于，所述程序被处理器执行时实现上述任一项的自动导引运输车数据采集的方法。To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any of the above-mentioned methods for collecting data of an automated guided vehicle is implemented.
上述发明中的一个实施例具有如下优点或有益效果：基于用户数据报协议，接收自动导引运输车批量发送的数据，则可通过数据上报的频率和模式，根据设备通讯的压力灵活调节采集的模式，达到降低网络负载的目的；去重机制可提高数据的高可用性；分布式消息队列具有很高的吞吐量，在流式传输环境下，报文数据可以不落地地直接发送回IDC，因此报文数据在上报过程中可以不对数据做序列化操作，节省了处理器资源。One embodiment of the above invention has the following advantages or beneficial effects: based on the user datagram protocol, data sent in batches by automatic guided vehicles are received, and the collection mode can be flexibly adjusted according to the pressure of device communication through the frequency and mode of data reporting, so as to achieve the purpose of reducing network load; the deduplication mechanism can improve the high availability of data; the distributed message queue has a very high throughput, and in a streaming transmission environment, the message data can be sent directly back to the IDC without landing, so the message data does not need to be serialized during the reporting process, saving processor resources.
上述的非惯用的可选方式所具有的进一步效果将在下文中结合具体实施方式加以说明。The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
附图用于更好地理解本发明，不构成对本发明的不当限定。其中：The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
图1是根据本发明实施例的自动导引运输车数据采集的方法的主要流程的示意图；FIG1 is a schematic diagram of the main process of a method for collecting data of an automated guided transport vehicle according to an embodiment of the present invention;
图2是根据本发明实施例的自动导引运输车数据采集的方法的系统结构的示意图；2 is a schematic diagram of a system structure of a method for collecting data of an automated guided transport vehicle according to an embodiment of the present invention;
图3是根据本发明实施例的自动导引运输车数据采集的装置的主要模块的示意图；3 is a schematic diagram of main modules of an apparatus for collecting data of an automated guided transport vehicle according to an embodiment of the present invention;
图4是本发明实施例可以应用于其中的示例性系统架构图；FIG4 is a diagram of an exemplary system architecture in which an embodiment of the present invention may be applied;
图5是适于用来实现本发明实施例的终端设备或服务器的计算机系统的结构示意图。FIG. 5 is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention.
具体实施方式DETAILED DESCRIPTION
以下结合附图对本发明的示范性实施例做出说明，其中包括本发明实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本发明的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
图1是根据本发明实施例的自动导引运输车数据采集的方法的主要流程的示意图，如图1所示，本发明实施例的自动导引运输车数据采集的方法主要包括：FIG. 1 is a schematic diagram of the main process of the method for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention. As shown in FIG. 1 , the method for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention mainly includes:
步骤S101：基于用户数据报协议，接收自动导引运输车批量发送的数据。自动化设备库房中的自动导引运输车AGV设备通过无线网络以数据报文的方式上报给分布式采集系统，其中为了降低网络负载和交互频率，采用批量上传的方式将数据上报。为了保证在网络中断的情况下防止数据丢失和数据重复的情况，要采用数据重发和数据防重码技术。从数据产生端尽可能保证数据的Exactly-Once特性。Step S101: Based on the user datagram protocol, receive the data sent in batches by the automated guided vehicles. The AGV equipment in the automated equipment warehouse reports to the distributed acquisition system in the form of data packets through the wireless network. In order to reduce the network load and interaction frequency, the data is reported in batch upload. In order to prevent data loss and data duplication in the event of network interruption, data retransmission and data anti-duplicate coding technology should be used. The Exactly-Once feature of the data should be guaranteed as much as possible from the data generation end.
对于上述过程的实现，可在AGV的嵌入式平台中部署一个数据上报系统，该数据上报系统负责实时搜集某一个或者几个库房内AGV传感器数据。搜集后的数据通过UDP(UserDatagram Protocol)协议定时或触发式地将搜集到的数据进行上报。UDP(User DatagramProtocol)协议全称是用户数据报协议，是一种无连接的协议，具有资源消耗小、处理速度快的优点。除了传统的报文通讯机制外，还采用了三种机制保证数据的高可用性，其分别是报文约束、数据重传和数据防重机制。For the realization of the above process, a data reporting system can be deployed in the embedded platform of AGV, which is responsible for collecting the sensor data of AGV in one or several warehouses in real time. The collected data is reported regularly or triggered through the UDP (User Datagram Protocol) protocol. The full name of the UDP (User Datagram Protocol) protocol is the User Datagram Protocol, which is a connectionless protocol with the advantages of low resource consumption and fast processing speed. In addition to the traditional message communication mechanism, three mechanisms are also used to ensure the high availability of data, namely message constraints, data retransmission and data anti-duplicate mechanism.
具体的，报文内容要求采集需要实时采集的关键动态数据，如实时速度、温度等，同时为了降低服务器和嵌入式设备的处理器负载，同时利用分布式消息队列的高吞吐性能，采用字符串的方式进行传输。UDP传输可能由于网络不稳定导致数据丢失，为了避免这一问题，上报系统需要设置重传机制，在检测到网络质量差可能存在数据丢失时，将最近的数据缓存，等到网络质量改善后重复发送。在报文中定义业务主键(防重码)，用于在接收端根据主键识别重复数据用于去重。Specifically, the message content requires the collection of key dynamic data that needs to be collected in real time, such as real-time speed, temperature, etc. At the same time, in order to reduce the processor load of the server and embedded devices, and to utilize the high throughput performance of the distributed message queue, the transmission is carried out in the form of strings. UDP transmission may cause data loss due to network instability. To avoid this problem, the reporting system needs to set up a retransmission mechanism. When it is detected that the network quality is poor and there may be data loss, the most recent data will be cached and sent again after the network quality improves. Define the business primary key (anti-duplicate code) in the message, which is used to identify duplicate data based on the primary key at the receiving end for deduplication.
步骤S102：根据数据的防重码，对数据进行去重处理。接收到数据上报系统批量传输的数据之后，对该数据进行去重处理。以及，在数据上报系统采集的数据为AGV的动态维度数据的前提下，可对去重处理后的数据补充静态维度数据，将补充后的数据发送至分布式消息队列。其中，动态维度数据为可实时变化的数据，如实时速度、温度等；静态维度数据为属性信息，一般不会变更，如库房号、IP地址等。Step S102: De-duplicate the data according to the anti-duplicate code of the data. After receiving the data transmitted in batches by the data reporting system, deduplicate the data. Also, under the premise that the data collected by the data reporting system is the dynamic dimension data of the AGV, the de-duplicated data can be supplemented with static dimension data, and the supplemented data can be sent to the distributed message queue. Among them, dynamic dimension data is data that can change in real time, such as real-time speed, temperature, etc.; static dimension data is attribute information, which generally does not change, such as warehouse number, IP address, etc.
步骤S103：将去重处理后的数据发送至分布式消息队列。具体的，根据传输需求建立的网关服务；基于网关服务，将去重处理后的数据发送至分布式消息队列。由于仓储库房地理分散的物理特性，各地不同的库房直接传输数据存在跨地区、网络权限限制、网络质量不稳定等问题。需要建立专门的网关服务，建立网络专线可保证传输质量。因此在将数据发送至分布式消息队列时，调用专门的网关服务将数据传输给数据中心IDC的分布式消息队列集群。分布式消息队列集群由多台服务器组成，每台服务器作为Broker(集群中服务器节点)接收发送的数据。在发送的过程中，会根据传输数据中的时间作为Key通过哈希算法指定发送到那些Broker上，Broker接收到数据后使用磁盘分页的机制快速的写入到本地磁盘并对数据的偏移量(offset)进行记录，从而实现海量数据的快速缓存。经过该方式存储的数据实时性可以达到毫秒级，且数据可以达到O(1)的复杂度写，和常数时间复杂的快速访问。其中，对于磁盘分页的机制，和传统的磁盘写入的方式有所区别，做了一定优化。其采用磁盘的顺序写入方式，将数据顺序的添加至磁盘文件的末尾，不仅降低了磁盘随机写入产生的寻址的开销，还提升了写入效率。数据被顺序写入到磁盘文件中，每条数据在写入是会记录写入的顺序，这个顺序的ID就是数据偏移量(offset)。而且，操作系统的分页存储及操作系统写入磁盘的操作并不是实时进行的，而是将文件和内存进行了映射管理，系统对文件的操作首先都会在内存中执行。在系统空闲的时间才会把操作同步在磁盘中执行性，则可让文件的读写接近内存的读写速度。Step S103: Send the deduplicated data to the distributed message queue. Specifically, a gateway service is established according to the transmission requirements; based on the gateway service, the deduplicated data is sent to the distributed message queue. Due to the geographically dispersed physical characteristics of storage warehouses, direct data transmission between different warehouses in different places has problems such as cross-regional, network authority restrictions, and unstable network quality. It is necessary to establish a special gateway service, and the establishment of a dedicated network line can ensure the transmission quality. Therefore, when sending data to the distributed message queue, a special gateway service is called to transmit the data to the distributed message queue cluster of the data center IDC. The distributed message queue cluster consists of multiple servers, and each server acts as a Broker (a server node in the cluster) to receive the sent data. During the sending process, the time in the transmitted data is used as the key to specify which Brokers are sent through a hash algorithm. After receiving the data, the Broker uses the disk paging mechanism to quickly write it to the local disk and record the offset of the data, thereby realizing the rapid caching of massive data. The real-time performance of the data stored in this way can reach the millisecond level, and the data can be written with a complexity of O(1), and accessed quickly with constant time complexity. Among them, the disk paging mechanism is different from the traditional disk writing method and has been optimized to a certain extent. It adopts the disk sequential writing method to add data sequentially to the end of the disk file, which not only reduces the addressing overhead caused by random disk writing, but also improves the writing efficiency. The data is written sequentially to the disk file, and the writing order of each data is recorded when it is written. The ID of this order is the data offset. Moreover, the paging storage of the operating system and the operation of the operating system writing to the disk are not performed in real time, but the file and memory are mapped and managed, and the system's operations on the file will first be executed in the memory. The operation will be synchronized on the disk when the system is idle, which can make the reading and writing of files close to the reading and writing speed of the memory.
采集到的数据可能拥有不同的用途，每种特殊用途的数据会拥有该用途特定的标识，即topic标识。由于采集到的数据是没有对数据的用途等进行分类的，所以在将去重处理后的数据发送至分布式消息队列之前，根据预设的分类规则对数据进行分类，并且根据分类的结果，对数据添加标识信息。则分布式消息队列以单线程的方式接收数据后，可根据接收到的数据的标识信息进行分类。并且，分布式消息队列以日志的格式将分类后的数据顺序写入磁盘，并且记录数据的偏移量信息。The collected data may have different uses, and each special-purpose data will have a specific identifier for that use, namely, a topic identifier. Since the collected data is not classified according to the purpose of the data, before sending the deduplicated data to the distributed message queue, the data is classified according to the preset classification rules, and identification information is added to the data according to the classification results. After receiving the data in a single-threaded manner, the distributed message queue can classify the data according to the identification information of the received data. In addition, the distributed message queue writes the classified data sequentially to the disk in the format of a log, and records the offset information of the data.
现有技术中，仓储设备控制系统一般通过心跳数据和业务指令与自动化设备进行交互，传统方案是在自动化设备的心跳报文中加入要上报的数据如设备导航数据、电池数据等。将新的报文格式序列化后传给控制台(仓储系统C/S架构中用于和AGV通讯以及指令交互的服务器端控制系统)，由控制台进行解析，获得采集的数据并将数据写入MYSQL数据表或者日志的形式进行存储。传统的MYSQL业务表需要预先设计好业务表结构和索引，海量的(万条/秒)传感器数据在控制台解析后使用worker连续大量的写入MYSQL表进行存储；或者是由控制台以写日志的方式写入本地的服务器磁盘进行缓存。并且，使用企业级的数据同步工具，每日定时批量从各个仓储服务器上将数据或日志同步到数据中心(IDC)进行加工，以实现数据的中心化存储。其存在的问题，主要包括：心跳方式采集导致的网络负载高的问题；报文数据序列化反序列化导致的处理器负载高的问题；网络异常导致的信息丢失、信息重复问题；库房服务器频繁I/O导致的磁盘寿命降低问题；日志数据可用性低的问题；中心化存储延时高的问题，将T+1的时效性提升到实时(毫秒级)。In the prior art, storage equipment control systems generally interact with automation equipment through heartbeat data and business instructions. The traditional solution is to add the data to be reported, such as equipment navigation data, battery data, etc., to the heartbeat message of the automation equipment. The new message format is serialized and transmitted to the console (the server-side control system used for communication with AGV and instruction interaction in the C/S architecture of the storage system), which is parsed by the console to obtain the collected data and write the data into a MYSQL data table or log for storage. The traditional MYSQL business table requires the business table structure and index to be designed in advance. After the console parses the massive (10,000/second) sensor data, it is continuously written into the MYSQL table by the worker for storage; or the console writes the data to the local server disk for caching in the form of log writing. In addition, using enterprise-level data synchronization tools, data or logs are synchronized from each storage server to the data center (IDC) in batches on a daily basis for processing to achieve centralized data storage. The existing problems mainly include: high network load caused by heartbeat collection; high processor load caused by message data serialization and deserialization; information loss and duplication caused by network anomalies; reduced disk life caused by frequent I/O of warehouse servers; low availability of log data; high latency of centralized storage, which improves the timeliness of T+1 to real-time (millisecond level).
针对上述问题，本发明实施例将数据采集和设备控制分离，通过数据上报系统可以控制数据上报的频率和模式，可以根据设备通讯的压力灵活调节采集的模式，达到降低网络负载的目的。并且，由于分布式消息队列具有很高的吞吐量，在流式传输环境下，报文数据可以不落地地直接发送回IDC，因此报文数据在上报过程中可以不对数据做序列化操作，节省了处理器资源。通过逻辑简单的数据重传、数据去重机制，通过一定的数据冗余换取数据的高可用性。可维护一条虚拟的数据管道，而高频的读写操作由分布式消息队列集群负责。库房服务器I/O量大大降低，同时通过分布式消息队列的偏移量服务，可以快速定位到需要的数据，保证数据的实时性。国各地的库房，可以以数据流的方式实时的将数据采集到IDC，由分布式消息队列集群进行管理。数据的实时性得到了巨大的提升。In view of the above problems, the embodiment of the present invention separates data collection and device control. The frequency and mode of data reporting can be controlled by the data reporting system, and the collection mode can be flexibly adjusted according to the pressure of device communication, so as to achieve the purpose of reducing network load. In addition, since the distributed message queue has a very high throughput, in a streaming transmission environment, the message data can be directly sent back to the IDC without landing, so the message data can be reported without serialization, saving processor resources. Through a logically simple data retransmission and data deduplication mechanism, a certain data redundancy is exchanged for high data availability. A virtual data pipeline can be maintained, and high-frequency read and write operations are handled by the distributed message queue cluster. The I/O volume of the warehouse server is greatly reduced. At the same time, through the offset service of the distributed message queue, the required data can be quickly located to ensure the real-time performance of the data. Warehouses across the country can collect data to the IDC in real time in the form of data streams, which are managed by the distributed message queue cluster. The real-time performance of the data has been greatly improved.
图2是根据本发明实施例的自动导引运输车数据采集的方法的系统结构的示意图；如图2所示，本发明实施例的自动导引运输车数据采集的方法的系统结构包括数据上报系统、分布式数据采集系统、网关模块、分布式消息队列集群这四大部分。Figure 2 is a schematic diagram of the system structure of the method for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention; as shown in Figure 2, the system structure of the method for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention includes four major parts: a data reporting system, a distributed data acquisition system, a gateway module, and a distributed message queue cluster.
对于数据上报系统，其部署在AGV的嵌入式平台中，负责实现AGV设备运行过程中传感器数据的实时搜集，搜集后的数据通过UDP协议定时或触发式地将搜集到的数据上报到分布式数据采集系统的数据接收模块。除了传统的报文通讯机制外，还采用了三种机制(报文约束、数据重传和数据防重)保证数据的高可用性。The data reporting system is deployed in the embedded platform of AGV and is responsible for the real-time collection of sensor data during the operation of AGV equipment. The collected data is reported to the data receiving module of the distributed data acquisition system in a timed or triggered manner through the UDP protocol. In addition to the traditional message communication mechanism, three mechanisms (message constraint, data retransmission and data anti-duplicate) are also adopted to ensure the high availability of data.
分布式数据采集系统负责接收数据上报系统上报的数据，对上报数据进行去重处理，并且在上报数据的基础上补充用于业务区分的静态维度数据(如库房号、IP地址等)。以及，按照分布式消息队列支持的数据格式，补充数据分区的Key和序列化后的Value值。为了保证数据的一致(按照数据用途对数据进行分类)，要对数据添加topic标识，最后调用网关模块异步按照分区规则将数据发送到分布式消息队列集群对应的Broker服务器上。其在，分区规则是指发送端根据消息的Key通过哈希算法指定发送到那些Broker上。一类特定用途的数据，拥有一个topic标识，在发送的时候可以发送到几个Broker，就有几个分区。The distributed data collection system is responsible for receiving the data reported by the data reporting system, deduplicating the reported data, and supplementing the static dimension data (such as warehouse number, IP address, etc.) used to distinguish the business based on the reported data. In addition, according to the data format supported by the distributed message queue, the Key and serialized Value of the data partition are supplemented. In order to ensure the consistency of the data (classify the data according to its purpose), it is necessary to add a topic identifier to the data, and finally call the gateway module to asynchronously send the data to the Broker server corresponding to the distributed message queue cluster according to the partitioning rules. In this case, the partitioning rule refers to the sender specifying which Brokers to send the message to based on the key of the message through a hash algorithm. A type of data for a specific purpose has a topic identifier and can be sent to several Brokers when it is sent, and there are several partitions.
分布式数据采集系统的设备数据接收模块，提供设备上报数据的接收服务，并对数据进行去重处理。由库房现场AGV设备数量庞大，该模块维护一个线程池，将并发的接收设备传输的数据缓存到内存中，以固定时间间隔或触发式的调用数据采集模块，将数据发送出去并清空内存。分布式数据采集系统的数据采集模块，供设备数据接收模块调用，每当设备数据接收模块调用时，该模块会将接收模块传来的暂存数据使用Avro(数据序列化的系统)进行序列化处理成字节数组写入Value，并根据当前系统时间生成Key，将封装好的Key-value键值对按照分区规则发送到分布式消息队列集群对应的Broker服务器上。其中，Key为数据发送到分布式消息队列集群的时间戳，其作用主要是做数据散列和后期数据核验用。The device data receiving module of the distributed data acquisition system provides a service for receiving data reported by the device and performs deduplication processing on the data. Due to the large number of AGV devices on site in the warehouse, this module maintains a thread pool to cache the data transmitted by concurrent receiving devices into the memory, and calls the data acquisition module at fixed time intervals or in a triggered manner to send the data out and clear the memory. The data acquisition module of the distributed data acquisition system is called by the device data receiving module. Whenever the device data receiving module is called, the module will use Avro (a data serialization system) to serialize the temporary data sent by the receiving module into a byte array and write it into the Value, and generate a Key based on the current system time, and send the encapsulated Key-value pair to the Broker server corresponding to the distributed message queue cluster according to the partitioning rules. Among them, the Key is the timestamp when the data is sent to the distributed message queue cluster, and its main function is to do data hashing and later data verification.
网关模块负责业务网络中数据转发服务，数据质量保证服务。实际的仓储业务场景中，自动化仓储库可能地理上十分分散，如在东北区域、华北区域、西北区域的可能同时存在仓储业务。不同的库房所处不同的局域网中，且离数据中心的距离不同，网络传输的质量差异较大。在这种情况下，需要网关技术构建虚拟的专线网络，打通不同网段的仓储网络，同时为网络传输提供高质量服务。The gateway module is responsible for data forwarding services and data quality assurance services in the business network. In actual warehousing business scenarios, automated warehouses may be geographically dispersed. For example, warehousing businesses may exist simultaneously in the Northeast, North China, and Northwest regions. Different warehouses are located in different local area networks and at different distances from the data center, resulting in large differences in the quality of network transmission. In this case, gateway technology is needed to build a virtual dedicated line network, connect warehousing networks in different network segments, and provide high-quality services for network transmission.
分布式消息队列集群，负责承担数据接收、数据缓存、数据偏移量管理等服务。分布式消息队列集群由若干台服务器组成，每台服务器负责接收分布式数据采集系统发送过来的数据，并以日志的方式写入到磁盘。这样的服务器称为一个Broker，如图2所示，共有N个Broker。每个Broker针对每一个发送端发送过来的数据以单线程的方式接收，从而保证接收到的数据是顺序一致的。不同数据源的数据通过topic进行区分。接收到的文件使用page cache机制log文件的大小固定，且同时生成对应的index文件记录数据的偏移量信息，便于数据的读取。Metadata Server(元数据管理服务节点)负责管理分布式消息队列集群中的Broker，主要包括监控Broker的存活情况，和维护Broker中的元数据信息。Broker通过RPC心跳定时向Metadata Server报告自己的状态信息，Broker通过心跳信息维护集群中Broker的状态列表，一旦发现集群新增Broker或者有Broker退出集群，会对Broker发出指令更新数据分区的信息，尽可能保证数据接收负载的均衡。另外如果发现Broker是因为异常终止，也要发出重启的指令，重启Broker服务。达到增强服务稳定和容错的目的。The distributed message queue cluster is responsible for services such as data reception, data caching, and data offset management. The distributed message queue cluster consists of several servers, each of which is responsible for receiving data sent by the distributed data acquisition system and writing it to the disk in the form of logs. Such a server is called a Broker. As shown in Figure 2, there are N Brokers in total. Each Broker receives the data sent by each sender in a single-threaded manner to ensure that the received data is in a consistent order. Data from different data sources are distinguished by topic. The size of the received file is fixed using the page cache mechanism log file, and the corresponding index file is generated at the same time to record the offset information of the data for easy data reading. The Metadata Server (metadata management service node) is responsible for managing the Broker in the distributed message queue cluster, mainly including monitoring the survival of the Broker and maintaining the metadata information in the Broker. The Broker reports its status information to the Metadata Server regularly through the RPC heartbeat. The Broker maintains the status list of the Broker in the cluster through the heartbeat information. Once a new Broker is found in the cluster or a Broker exits the cluster, it will issue an instruction to the Broker to update the data partition information to ensure the balance of the data receiving load as much as possible. In addition, if it is found that the Broker is terminated due to an abnormality, a restart instruction should also be issued to restart the Broker service, so as to enhance service stability and fault tolerance.
自动化设备库房中的AGV设备通过无线网络以数据报文的方式，将搜集到的数据上报给分布式采集系统，其中，为了降低网络负载和交互频率，采用批量上传的方式将数据上报给数据接收模块。为了保证在网络中断的情况下防止数据丢失和数据重复的情况，要采用数据重发和数据防重码技术，从数据产生端尽可能保证数据的Exactly-Once特性。在数据接收模块接收到上报的AGV设备数据后，在内存中进行临时缓存，以固定的时间间隔或是达到触发规则时异步调用数据采集模块的发送接口将数据发送给分布式消息队列集群。其中，触发规则主要有两种，当AGV出现异常时和数据量达到缓存区的存储上限时，则不在采用定时的逻辑，立即将所有缓存的数据发送给。分布式消息队列集群。由于仓储库房地理分散的物理特性，各地不同的库房接收模块直接传输数据存在跨地区、网络权限限制、网络质量不稳定等问题。需要建立专门的网关服务，建立网络专线保证传输质量。因此数据采集系统的采集模块调用网关模块的传输接口将数据传输给数据中心的分布式消息队列集群，分布式消息队列集群由多台服务器组成，每台服务器作为Broker接收采集模块发送的数据。采集模块会根据传输数据中的时间作为Key通过哈希算法指定发送到那些Broker上，Broker接收到数据后使用磁盘分页的机制快速的写入到本地磁盘并对数据的偏移量(offset)进行记录，从而实现海量数据的快速缓存。经过该方式存储的数据实时性可以达到毫秒级，且数据可以达到O(1)的复杂度写，和常数时间复杂的快速访问。最后，数据中心IDC可以根据业务需求从分布式消息队列集群中拉取数据消费。The AGV equipment in the automated equipment warehouse reports the collected data to the distributed collection system in the form of data packets through the wireless network. In order to reduce the network load and interaction frequency, the data is reported to the data receiving module in batch upload. In order to prevent data loss and data duplication in the event of network interruption, data retransmission and data anti-duplicate coding technology should be used to ensure the Exactly-Once characteristics of the data as much as possible from the data generation end. After the data receiving module receives the reported AGV equipment data, it is temporarily cached in the memory, and the data collection module's sending interface is asynchronously called at a fixed time interval or when the trigger rule is reached to send the data to the distributed message queue cluster. Among them, there are two main triggering rules. When the AGV is abnormal and the data volume reaches the storage limit of the cache area, the timing logic is no longer used, and all cached data is immediately sent to the distributed message queue cluster. Due to the geographically dispersed physical characteristics of storage warehouses, the direct data transmission of different warehouse receiving modules in different places has problems such as cross-regional, network permission restrictions, and unstable network quality. It is necessary to establish a dedicated gateway service and a dedicated network line to ensure transmission quality. Therefore, the acquisition module of the data acquisition system calls the transmission interface of the gateway module to transmit data to the distributed message queue cluster of the data center. The distributed message queue cluster consists of multiple servers, and each server acts as a Broker to receive the data sent by the acquisition module. The acquisition module uses the time in the transmission data as the key and sends it to those Brokers through a hash algorithm. After receiving the data, the Broker uses the disk paging mechanism to quickly write it to the local disk and record the offset of the data, thereby realizing the rapid caching of massive data. The real-time performance of data stored in this way can reach the millisecond level, and the data can be written with O(1) complexity and quickly accessed with constant time complexity. Finally, the data center IDC can pull data from the distributed message queue cluster for consumption according to business needs.
图3是根据本发明实施例自动导引运输车数据采集的装置的主要模块的示意图，如图3所示，本发明实施例的自动导引运输车数据采集的装置300包括接收模块301、去重模块302和发送模块303。Figure 3 is a schematic diagram of the main modules of the device for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention. As shown in Figure 3, the device 300 for collecting data of an automatic guided transport vehicle according to an embodiment of the present invention includes a receiving module 301, a deduplication module 302 and a sending module 303.
接收模块301用于，基于用户数据报协议，接收自动导引运输车批量发送的数据。The receiving module 301 is used to receive data sent in batches by the automatic guided transport vehicle based on the user datagram protocol.
去重模块302用于，根据数据的防重码，对数据进行去重处理。The deduplication module 302 is used to perform deduplication processing on the data according to the anti-duplication code of the data.
发送模块303用于，将去重处理后的数据发送至分布式消息队列。发送模块，还用于根据传输需求建立的网关服务；基于网关服务，将去重处理后的数据发送至分布式消息队列。发送模块，还用于对去重处理后的数据补充静态维度数据；并且，将补充后的数据发送至分布式消息队列。The sending module 303 is used to send the deduplicated data to the distributed message queue. The sending module is also used to establish a gateway service according to the transmission requirements; based on the gateway service, the deduplicated data is sent to the distributed message queue. The sending module is also used to supplement the deduplicated data with static dimension data; and send the supplemented data to the distributed message queue.
本发明实施例的自动导引运输车数据采集的装置还包括标识信息添加模块，用于根据预设的分类规则对数据进行分类；并且，根据分类的结果，对数据添加标识信息。分布式消息队列以单线程的方式接收数据；分布式消息队列根据接收到的数据的标识信息进行分类；分布式消息队列以日志的格式将分类后的数据顺序写入磁盘，并且记录数据的偏移量信息。The device for collecting data of an automated guided vehicle in an embodiment of the present invention further includes an identification information adding module, which is used to classify data according to a preset classification rule; and, according to the classification result, to add identification information to the data. The distributed message queue receives data in a single-threaded manner; the distributed message queue classifies the received data according to the identification information; the distributed message queue writes the classified data sequentially to the disk in a log format, and records the offset information of the data.
针对上述问题，本发明实施例将数据采集和设备控制分离，通过数据上报系统可以控制数据上报的频率和模式，可以根据设备通讯的压力灵活调节采集的模式，达到降低网络负载的目的。并且，由于分布式消息队列具有很高的吞吐量，在流式传输环境下，报文数据可以不落地地直接发送回IDC，因此报文数据在上报过程中可以不对数据做序列化操作，节省了处理器资源。通过逻辑简单的数据重传、数据去重机制，通过一定的数据冗余换取数据的高可用性。可维护一条虚拟的数据管道，而高频的读写操作由分布式消息队列集群负责。库房服务器I/O量大大降低，同时通过分布式消息队列的偏移量服务，可以快速定位到需要的数据，保证数据的实时性。国各地的库房，可以以数据流的方式实时的将数据采集到IDC，由分布式消息队列集群进行管理。数据的实时性得到了巨大的提升。In view of the above problems, the embodiment of the present invention separates data collection and device control. The frequency and mode of data reporting can be controlled through the data reporting system, and the collection mode can be flexibly adjusted according to the pressure of device communication, so as to achieve the purpose of reducing network load. In addition, since the distributed message queue has a very high throughput, in a streaming transmission environment, the message data can be directly sent back to the IDC without landing, so the message data can be reported without serialization, saving processor resources. Through a logically simple data retransmission and data deduplication mechanism, a certain data redundancy is exchanged for high data availability. A virtual data pipeline can be maintained, and high-frequency read and write operations are handled by the distributed message queue cluster. The I/O volume of the warehouse server is greatly reduced, and at the same time, the offset service of the distributed message queue can quickly locate the required data to ensure the real-time performance of the data. Warehouses across the country can collect data to the IDC in real time in the form of data streams, which are managed by the distributed message queue cluster. The real-time performance of the data has been greatly improved.
图4示出了可以应用本发明实施例的自动导引运输车数据采集的方法或自动导引运输车数据采集的装置的示例性系统架构400。FIG. 4 shows an exemplary system architecture 400 to which the method for collecting data from an automated guided transport vehicle or the apparatus for collecting data from an automated guided transport vehicle according to an embodiment of the present invention may be applied.
如图4所示，系统架构400可以包括终端设备401、402、403，网络404和服务器405。网络404用以在终端设备401、402、403和服务器405之间提供通信链路的介质。网络404可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。As shown in Fig. 4, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
用户可以使用终端设备401、402、403通过网络404与服务器405交互，以接收或发送消息等。终端设备401、402、403上可以安装有各种通讯客户端应用，例如购物类应用、网页浏览器应用、搜索类应用、即时通信工具、邮箱客户端、社交平台软件等(仅为示例)。Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
终端设备401、402、403可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
服务器405可以是提供各种服务的服务器，例如对用户利用终端设备401、402、403所浏览的购物类网站提供支持的后台管理服务器(仅为示例)。后台管理服务器可以对接收到的产品信息查询请求等数据进行分析等处理，并将处理结果反馈给终端设备。Server 405 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using terminal devices 401, 402, and 403. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results to the terminal device.
需要说明的是，本发明实施例所提供的自动导引运输车数据采集的方法一般由服务器405执行，相应地，自动导引运输车数据采集的装置一般设置于服务器405中。It should be noted that the method for collecting data of an automated guided vehicle provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the device for collecting data of an automated guided vehicle is generally disposed in the server 405 .
应该理解，图4中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。It should be understood that the number of terminal devices, networks and servers in Figure 4 is only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
下面参考图5，其示出了适于用来实现本发明实施例的终端设备的计算机系统500的结构示意图。图5示出的终端设备仅仅是一个示例，不应对本发明实施例的功能和使用范围带来任何限制。Referring to Figure 5, a schematic diagram of a computer system 500 suitable for implementing a terminal device of an embodiment of the present invention is shown. The terminal device shown in Figure 5 is only an example and should not limit the functions and scope of use of the embodiment of the present invention.
如图5所示，计算机系统500包括中央处理单元(CPU)501，其可以根据存储在只读存储器(ROM)502中的程序或者从存储部分508加载到随机访问存储器(RAM)503中的程序而执行各种适当的动作和处理。在RAM 503中，还存储有系统500操作所需的各种程序和数据。CPU 501、ROM 502以及RAM 503通过总线504彼此相连。输入/输出(I/O)接口505也连接至总线504。As shown in FIG5 , the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the system 500 are also stored in the RAM 503. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to the bus 504.
以下部件连接至I/O接口505：包括键盘、鼠标等的输入部分506；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分507；包括硬盘等的存储部分508；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分509。通信部分509经由诸如因特网的网络执行通信处理。驱动器510也根据需要连接至I/O接口505。可拆卸介质511，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器510上，以便于从其上读出的计算机程序根据需要被安装入存储部分508。The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
特别地，根据本发明公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本发明公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分509从网络上被下载和安装，和/或从可拆卸介质511被安装。在该计算机程序被中央处理单元(CPU)501执行时，执行本发明的系统中限定的上述功能。In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and/or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.
需要说明的是，本发明所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本发明中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本发明中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
附图中的流程图和框图，图示了按照本发明各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
描述于本发明实施例中所涉及到的模块可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的模块也可以设置在处理器中，例如，可以描述为：一种处理器包括获取接收模块、去重模块和发送模块。其中，这些模块的名称在某种情况下并不构成对该模块本身的限定，例如，自动导引运输车数据采集还可以被描述为“基于用户数据报协议，接收自动导引运输车批量发送的数据的模块”。The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including an acquisition receiving module, a deduplication module and a sending module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the data acquisition of the automated guided vehicle may also be described as "a module for receiving data sent in batches by the automated guided vehicle based on the user datagram protocol".
作为另一方面，本发明还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的设备中所包含的；也可以是单独存在，而未装配入该设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被一个该设备执行时，使得该设备包括：基于用户数据报协议，接收自动导引运输车批量发送的数据；根据所述数据的防重码，对所述数据进行去重处理；将去重处理后的数据发送至分布式消息队列。As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving data sent in batches by an automated guided transport vehicle based on a user datagram protocol; performing deduplication processing on the data according to an anti-duplicate code of the data; and sending the deduplicated data to a distributed message queue.
针对上述问题，本发明实施例将数据采集和设备控制分离，通过数据上报系统可以控制数据上报的频率和模式，可以根据设备通讯的压力灵活调节采集的模式，达到降低网络负载的目的。并且，由于分布式消息队列具有很高的吞吐量，在流式传输环境下，报文数据可以不落地地直接发送回IDC，因此报文数据在上报过程中可以不对数据做序列化操作，节省了处理器资源。通过逻辑简单的数据重传、数据去重机制，通过一定的数据冗余换取数据的高可用性。可维护一条虚拟的数据管道，而高频的读写操作由分布式消息队列集群负责。库房服务器I/O量大大降低，同时通过分布式消息队列的偏移量服务，可以快速定位到需要的数据，保证数据的实时性。国各地的库房，可以以数据流的方式实时的将数据采集到IDC，由分布式消息队列集群进行管理。数据的实时性得到了巨大的提升。In view of the above problems, the embodiment of the present invention separates data collection and device control. The frequency and mode of data reporting can be controlled by the data reporting system, and the collection mode can be flexibly adjusted according to the pressure of device communication, so as to achieve the purpose of reducing network load. In addition, since the distributed message queue has a very high throughput, in a streaming transmission environment, the message data can be directly sent back to the IDC without landing, so the message data can be reported without serialization, saving processor resources. Through a logically simple data retransmission and data deduplication mechanism, a certain data redundancy is exchanged for high data availability. A virtual data pipeline can be maintained, and high-frequency read and write operations are handled by the distributed message queue cluster. The I/O volume of the warehouse server is greatly reduced. At the same time, through the offset service of the distributed message queue, the required data can be quickly located to ensure the real-time performance of the data. Warehouses across the country can collect data to the IDC in real time in the form of data streams, which are managed by the distributed message queue cluster. The real-time performance of the data has been greatly improved.
上述具体实施方式，并不构成对本发明保护范围的限制。本领域技术人员应该明白的是，取决于设计要求和其他因素，可以发生各种各样的修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等，均应包含在本发明保护范围之内。The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810941820.9ACN110837423B (en) | 2018-08-17 | 2018-08-17 | A method and device for collecting data of an automatic guided transport vehicle |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810941820.9ACN110837423B (en) | 2018-08-17 | 2018-08-17 | A method and device for collecting data of an automatic guided transport vehicle |
| Publication Number | Publication Date |
|---|---|
| CN110837423A CN110837423A (en) | 2020-02-25 |
| CN110837423Btrue CN110837423B (en) | 2024-09-20 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810941820.9AActiveCN110837423B (en) | 2018-08-17 | 2018-08-17 | A method and device for collecting data of an automatic guided transport vehicle |
| Country | Link |
|---|---|
| CN (1) | CN110837423B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597098B (en)* | 2020-05-14 | 2024-07-19 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN113810337B (en)* | 2020-06-12 | 2023-08-08 | 天翼云科技有限公司 | Method, device, and storage medium for deduplication of network packets |
| CN112416626B (en)* | 2020-12-02 | 2023-06-06 | 中国联合网络通信集团有限公司 | A data processing method and device |
| CN112783899B (en)* | 2021-02-08 | 2023-03-24 | 青岛港国际股份有限公司 | Method for interaction between air rail single machine control system and wharf equipment control system |
| CN113341833A (en)* | 2021-06-22 | 2021-09-03 | 天津航通科技有限公司 | Acquisition and transmission method for operating state data of ground equipment of airport apron |
| CN113641520B (en)* | 2021-08-20 | 2024-04-05 | 北京百度网讯科技有限公司 | Data processing method, system, device and storage medium |
| CN114253745B (en)* | 2021-12-16 | 2023-06-20 | 北京金堤科技有限公司 | Message deduplication processing method and device, storage medium and electronic equipment |
| CN115878702A (en)* | 2022-11-30 | 2023-03-31 | 浪潮通用软件有限公司 | A method, device and equipment for realizing operation and maintenance data collection |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635279A (en)* | 2015-12-29 | 2016-06-01 | 长城信息产业股份有限公司 | Distributed monitor system and data acquisition method thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9521197B2 (en)* | 2012-12-05 | 2016-12-13 | International Business Machines Corporation | Utilizing data object storage tracking in a dispersed storage network |
| CN105933440A (en)* | 2016-06-20 | 2016-09-07 | 福建省福工动力技术有限公司 | Remote monitoring system and remote monitoring method of new energy automobile |
| CN107579929B (en)* | 2017-09-18 | 2021-12-03 | 郑州云海信息技术有限公司 | Method, system and related device for setting reliable connection communication queue pair |
| CN107863824A (en)* | 2017-12-10 | 2018-03-30 | 国家电网公司 | Distribute-electricity transformer district intelligence managing and control system based on the optimization of load Intelligent cluster |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635279A (en)* | 2015-12-29 | 2016-06-01 | 长城信息产业股份有限公司 | Distributed monitor system and data acquisition method thereof |
| Publication number | Publication date |
|---|---|
| CN110837423A (en) | 2020-02-25 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110837423B (en) | A method and device for collecting data of an automatic guided transport vehicle | |
| CN109947668B (en) | Method and device for storing data | |
| US11676066B2 (en) | Parallel model deployment for artificial intelligence using a primary storage system | |
| CN108536778B (en) | Data application sharing platform and method | |
| CN109063196B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN107861686A (en) | File memory method, service end and computer-readable recording medium | |
| CN108306980A (en) | A kind of engineering flight support big data Log Analysis System | |
| CN111339192A (en) | Distributed edge computing data storage system | |
| US20250265247A1 (en) | Runtime systems query coordinator | |
| CN110334145A (en) | Method and device for data processing | |
| CN118540319B (en) | Method and device for downloading file through browser and electronic equipment | |
| US20140236987A1 (en) | System and method for audio signal collection and processing | |
| CN113656469A (en) | Big data processing method and device | |
| CN116467275A (en) | Shared remote storage method, device, system, electronic equipment and storage medium | |
| CN113760966B (en) | Data processing method and device based on heterogeneous database system | |
| CN115695426A (en) | Data processing method, device, equipment and storage medium in multi-cluster scenario | |
| US11210212B2 (en) | Conflict resolution and garbage collection in distributed databases | |
| CN113051244B (en) | Data access method and device, data acquisition method and device | |
| CN114925044A (en) | Data synchronization method, device, device and storage medium based on cloud storage | |
| CN114398372A (en) | Data caching method and device | |
| WO2023019712A1 (en) | Zlib compression algorithm-based cloud computing resource manager communication delay optimization method | |
| CN110866001A (en) | Method and device for determining order to be processed | |
| CN111597157A (en) | Method for improving log processing system architecture | |
| CN117194562A (en) | Data synchronization method and device, electronic equipment and computer readable medium | |
| CN115374078A (en) | Data storage and reading system and working method thereof |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right | Effective date of registration:20200601 Address after:Room A1905, 19th floor, No. 2 Building, 18 Kechuang 11th Street, Beijing Daxing District, Beijing Applicant after:Beijing Jingdong Qianshi Technology Co.,Ltd. Address before:300 457 days Tianjin Binhai New Area, Tianjin Economic and Technological Development Zone, No. 10 Fourth Avenue, Phase 2, First Floor Applicant before:Tianjin Jingdong Shentuo Robot Technology Co.,Ltd. | |
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |