技术领域Technical Field
本发明涉及可视化确认交互领域技术领域，特别涉及一种石化行业移动信息防止误操作确认交互方法。The present invention relates to the technical field of visual confirmation interaction, and in particular to a mobile information confirmation interaction method for preventing misoperation in the petrochemical industry.
背景技术Background technique
石油化工行业具有高温高压、易燃易爆、有毒有害等的特点，现场施工作业量大，一旦管控不力，极易发生中毒、窒息、着火、爆炸等安全事故，不仅对企业利益造成严重影响，也严重威胁人民的生命安全。The petrochemical industry is characterized by high temperature, high pressure, flammability, explosiveness, toxicity and harm. The amount of on-site construction work is large. If it is not properly managed and controlled, safety accidents such as poisoning, suffocation, fire and explosion are very likely to occur, which will not only have a serious impact on corporate interests, but also seriously threaten people's lives and safety.
目前在石油化工行业固定场景下安防监控已经做的比较完善，但是在移动作业过程中作业过程监控亟待解决，根据相关要求，手指口述其核心就是解决移动作业监控的一种技术解决方案。At present, security monitoring in fixed scenarios in the petrochemical industry has been relatively complete, but the monitoring of operation processes in mobile operations needs to be urgently addressed. According to relevant requirements, the core of finger dictation is to provide a technical solution to solve the problem of mobile operation monitoring.
手指口述是采用音视频技术指导石化行业现场作业，其主要目的是确保安全作业，该技术核心解决方案为将现场情况高清传送到远程专家面前，由专家进行会诊，专家无论在PC上还是智能手机上均可与现场人员实时多方高清视频指导，利用冻屏、白板、文件传输，共享屏幕等工具，共同解决现场遇到的难题，该发明技术在长岭石化已经全场推广应用。Finger dictation uses audio and video technology to guide on-site operations in the petrochemical industry. Its main purpose is to ensure safe operations. The core solution of this technology is to transmit the on-site situation in high definition to remote experts for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on both PCs and smartphones, and use tools such as frozen screen, whiteboard, file transfer, and shared screen to jointly solve problems encountered on-site. This invention has been promoted and applied throughout Changling Petrochemical.
现有的技术问题是：移动4/5G布控球：视野范围有限，缺乏灵活性，难以追踪移动目标，视角受限，易被遮挡，恶劣天气条件下效果无法保证，现场第一时间获取视频信息不易，无交互能力。The existing technical problems are: Mobile 4/5G surveillance ball: limited field of view, lack of flexibility, difficulty in tracking moving targets, limited viewing angle, easy to be blocked, the effect cannot be guaranteed under bad weather conditions, it is not easy to obtain video information on site in the first time, and there is no interactive capability.
法记录仪：抖动明显，难以获得长期稳定图像，镜头与视线不同步，录制内容有可能非所需，镜头极易被衣物，肢体或设备等遮挡，对佩戴者稳定性要求高，在移动状态下容易失真，现场第一时间获取视频信息不易，无交互能力为此，我们提出一种移动信息防止误操作确认交互方法。Legal recorder: obvious jitter, difficult to obtain long-term stable images, the lens is not synchronized with the line of sight, the recorded content may not be what is needed, the lens is easily blocked by clothing, limbs or equipment, etc., high stability requirements for the wearer, easy to distort when moving, difficult to obtain video information on the scene in the first time, no interactive capability. Therefore, we propose a mobile information to prevent misoperation confirmation interaction method.
发明内容Summary of the invention
有鉴于此，本发明实施例希望提供一种移动信息防止误操作确认交互方法，其目的在于移动作业过程中信息交互，远程专家对现场作业人员可进行手指口述操作，并对此至少提供一种有益的选择；In view of this, the embodiment of the present invention hopes to provide a mobile information confirmation interaction method to prevent misoperation, the purpose of which is to interact with information during mobile operations, and remote experts can perform finger-dictated operations on on-site operators, and at least provide a beneficial option for this;
本发明的技术方案是这样实现的：The technical solution of the present invention is achieved in this way:
一种移动信息防止误操作确认交互方法，移动作业过程中信息交互，远程专家对现场作业人员可进行手指口述操作。A mobile information confirmation interaction method to prevent misoperation. During the mobile operation process, information is exchanged and remote experts can perform finger-dictated operations on on-site workers.
（一）概述(I. Overview
本申请旨在进一步移动作业过程中信息交互技术解决方案，通过头戴计算机等智能设备采集音视频数据，将音视频数据加工转换成符合国标GB/T28181协议，形成视频流，可推送到手指口述可视化管理平台，同时手指口述平台可将音视频流实时推送至头戴计算机等智能设备，实现音频互动交流。This application aims to further develop technical solutions for information interaction during mobile operations. By collecting audio and video data through smart devices such as head-mounted computers, the audio and video data are processed and converted into video streams that comply with the national standard GB/T28181 protocol, which can be pushed to the finger-dictation visualization management platform. At the same time, the finger-dictation platform can push the audio and video streams to smart devices such as head-mounted computers in real time to achieve audio interactive communication.
“手指口述”是一种安全确认的方法，主要应用于危险性较大的工种、行业，主要用于矿山等行业。"Finger-to-mouth" is a method of safety confirmation, which is mainly used in jobs and industries with higher risks, mainly in mining and other industries.
（二）技术内容(II) Technical content
第一方面：一种移动信息防止误操作确认交互方法，包括如下步骤T-1、T-2、T-3和T-4：A first aspect: A mobile information confirmation interaction method for preventing misoperation includes the following steps T-1, T-2, T-3 and T-4:
所述T-1、信令注册：前端设备首次接入平台时，会持续发送REGISTER消息直到服务器回应“200 OK”表示成功注册，若失败则会随机延迟后重新尝试注册，同时，所述前端设备会定期刷新注册；T-1, signaling registration: When the front-end device accesses the platform for the first time, it will continue to send REGISTER messages until the server responds with "200 OK" to indicate successful registration. If it fails, it will try to register again after a random delay. At the same time, the front-end device will refresh the registration periodically;
所述T-2、数据传输交互：信令注册成功后，服务器会发送目录信息给客户端，客户端构建内容并将其发送给服务器，服务器进行校验并返回INVITE响应；T-2, data transmission interaction: After the signaling registration is successful, the server will send the directory information to the client, the client will build the content and send it to the server, the server will verify it and return the INVITE response;
所述T-3、获取前端设备音视频编解码数据；其中，提取所述音视频编解码数据的关联信息，所述关联信息包括：T-3, obtaining audio and video codec data of the front-end device; wherein, extracting associated information of the audio and video codec data, the associated information includes:
S3.1、视频编码：把相机采集到的数据通过MediaCodec编码h264，再使用RTP协议打包PS进行传输；S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;
S3.2、音频编码：通过AudioRecord来采集声音进行G711编码RTP打包传输发送；S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711, and then transmit it in RTP packaging;
S3.3、混流编码：通过MediaMuxer把音视频混流，初始化MediaExtractor用于从视频文件和音频文件中提取视频和音频轨道信息并获取其格式；S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, initialize MediaExtractor to extract video and audio track information from video files and audio files and obtain their formats;
所述T-4，获取云平台的音频媒体流交互：前端设备与云平台通过信令来建立广播连接，最后实现音视频交互。The T-4 obtains the audio media stream interaction of the cloud platform: the front-end device and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction.
优选的：在所述T-1中，信令注册具体包括：Preferably: in the T-1, the signaling registration specifically includes:
1）客户端向SIP服务器发起注册请求，服务器收到请求后对注册信息进行校验，校验通过后向客户端发送响应401，并在响应的header WWW-Authenticate字段中给出客户端的认证体制和参数；1) The client initiates a registration request to the SIP server. After receiving the request, the server verifies the registration information. After the verification is passed, it sends a 401 response to the client and gives the client's authentication system and parameters in the header WWW-Authenticate field of the response.
2）客户端收到认证体制和参数数据后重新向 SIP 服务器发送注册请求，在请求的 Authorization 字段给出信任书并携带认证信息；2) After receiving the authentication system and parameter data, the client resends a registration request to the SIP server, providing a letter of trust and carrying authentication information in the Authorization field of the request;
3）SIP 服务器收到请求对请求再次进行校验，校验之后，若客户端身份合法，则向客户端发送成功响应“200 OK”，若客户端身份不合法则会发送本次服务拒绝服务应答消息。3) The SIP server receives the request and verifies it again. After verification, if the client identity is legal, it sends a successful response "200 OK" to the client. If the client identity is illegal, it sends a service denial response message.
优选的：在所述T-2中，所述数据传输交互：具体步骤包括：Preferably, in said T-2, said data transmission interaction: the specific steps include:
S2.1、信令注册成功后服务端会发送CATALOG目录信息给客户端；S2.1. After the signaling registration is successful, the server will send the CATALOG directory information to the client;
S2.2、客户端收到CATALOG消息并对内容进行构建，完成后发送给服务端；S2.2, the client receives the CATALOG message and constructs the content, and sends it to the server after completion;
S2.3、服务端收到CATALOG信息后对内容进行校验，校验通过后服务端发送INVITE响应给客户端；S2.3, after receiving the CATALOG information, the server verifies the content. After the verification passes, the server sends an INVITE response to the client;
S2.4、客户端收到INVITE响应后对INVITE信息与SDP信息进行封装并对服务端做出回复；客户端根据服务端发过来的INVITE信息与SDP信息获取到RTP传输的端口，其中所述SDP信息包括TCP、UDP模式和SSRC信息；S2.4. After receiving the INVITE response, the client encapsulates the INVITE information and SDP information and responds to the server. The client obtains the RTP transmission port according to the INVITE information and SDP information sent by the server, wherein the SDP information includes TCP, UDP mode and SSRC information.
S2.5、所有连接建立之后客户端把采集到的音视频数据，打包成符合国标GB/T28181协议的PS流，基于RTP封包的形式发送到服务端；S2.5. After all connections are established, the client packages the collected audio and video data into a PS stream that complies with the national standard GB/T28181 protocol and sends it to the server in the form of RTP packets;
对于所述国标GB/T28181协议中常用的数据流打包格式PS流方式，将SDP文件中的视频属性字段集合中描述流结构的字段写成：a=rtpmap：96MP2P/90000，a=rtpmap为固定格式，96为负载类型，MP2P为编码类型，90000为时钟频率；接着上一行a=rtpmap：96MP2P/90000，设计a=fmtp：96，其中a=fmtp为固定格式，96为负载类型；For the PS stream mode, a data stream packaging format commonly used in the national standard GB/T28181 protocol, the field describing the stream structure in the video attribute field set in the SDP file is written as: a=rtpmap:96MP2P/90000, a=rtpmap is a fixed format, 96 is the payload type, MP2P is the encoding type, and 90000 is the clock frequency; following the previous line a=rtpmap:96MP2P/90000, a=fmtp:96 is designed, where a=fmtp is a fixed format and 96 is the payload type;
S2.6、若客户端需要断开Invite会话连接，则会需要发送BYE请求给服务端，服务端返回“200 OK”之后断开连接；S2.6. If the client needs to disconnect the Invite session, it will need to send a BYE request to the server. The server will disconnect after returning "200 OK";
S2.7、若客户端发起注销：发送 Register命令消息给服务端，消息中的 Expire字段设置为0即代表注销，此时中止会话。S2.7. If the client initiates a logout: it sends a Register command message to the server, and the Expire field in the message is set to 0, which means logout, and the session is terminated at this time.
优选的：在所述S3.1中，所述视频编码：把相机采集到的数据通过MediaCodec编码h264，再使用RTP协议打包PS进行传输，具体包括：Preferably, in S3.1, the video encoding: encoding the data collected by the camera into h264 through MediaCodec, and then using the RTP protocol to package PS for transmission, specifically includes:
S3.11、初始化获取设备Camera2通过PreviewFrame实时监听检测相机采集返回的原始数据；S3.11, initialize and obtain the original data returned by the device Camera2 through PreviewFrame real-time monitoring detection camera collection;
S3.12、把采集到的NV21数据转成h264编码为YUV420P数据；S3.12, convert the collected NV21 data into h264 encoding into YUV420P data;
S3.13、创建MediaCodec实例设置编码格式参数，对YUV数据进行h264硬编码处理；S3.13. Create a MediaCodec instance to set encoding format parameters and perform h264 hard encoding on YUV data;
S3.14、获取编码后的h264数据，对所述h264数据进行国标GB/T28181协议的PS格式封装；S3.14, obtaining the encoded h264 data, and encapsulating the h264 data in PS format of the national standard GB/T28181 protocol;
S3.15、基于RTP/UDP把封装后的数据进行发送到服务端。S3.15. Send the encapsulated data to the server based on RTP/UDP.
优选的：在所述S3.2中，所述音频编码：通过AudioRecord来采集声音进行G711编码RTP打包传输发送，具体包括：Preferably, in S3.2, the audio encoding: collecting sound through AudioRecord and performing G711 encoding RTP packaging transmission and sending, specifically includes:
S3.21、初始化AudioRecord设置编码格式参数，采集音频数据；S3.21, initialize AudioRecord to set encoding format parameters and collect audio data;
S3.22、开启录音后通过audioRecord.read将获取到的音频数据写入创建好的字节缓存区；S3.22, after starting recording, write the acquired audio data into the created byte buffer through audioRecord.read;
S3.23、读取缓存区内的数据并进行short类型转换，把转换后的short数据进行G711编码；S3.23, read the data in the buffer area and perform short type conversion, and encode the converted short data into G711;
S3.24、编码后的数据通过RTP协议打包发送服务端。S3.24. The encoded data is packaged and sent to the server via the RTP protocol.
优选的：在所述S3.3中，所述混流编码：通过MediaMuxer把音视频混流，初始化MediaExtractor用于从视频文件和音频文件中提取视频和音频轨道信息并获取其格式，具体包括：Preferably, in S3.3, the mixed stream encoding: mixing the audio and video streams through MediaMuxer, initializing MediaExtractor to extract video and audio track information from the video file and the audio file and obtain their formats, specifically includes:
S3.31、初始化MediaExtractor从视频文件和音频文件中提取视频和音频轨道信息；S3.31, initialize MediaExtractor to extract video and audio track information from video files and audio files;
S3.32、用MediaExtractor.readSampleData从视频文件和音频文件各自读取一帧数据到缓存区；S3.32, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file respectively into the buffer area;
S3.33、实例化MediaMuxer对象将缓存区内的音频和视频数据合并输出；S3.33, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output them;
S3.34、设置缓存区信息，包含当前帧的时间戳和标志位，判定是否为关键帧；S3.34, set the buffer information, including the timestamp and flag of the current frame, to determine whether it is a key frame;
S3.35、基于RTP/UDP使用DatagramSocket进行发送。S3.35. Send using DatagramSocket based on RTP/UDP.
优选的：在所述T-4中，所述获取云平台的音频媒体流交互：Preferably, in said T-4, the audio media stream interaction of the cloud platform is obtained:
S4.1、云平台发MESSAGE语音广播命令到前端设备；S4.1. The cloud platform sends a MESSAGE voice broadcast command to the front-end device;
S4.2、前端设备发MESSAGE消息应答语音广播命令；S4.2, the front-end device sends a MESSAGE message to respond to the voice broadcast command;
S4.3、前端设备收到MESSAGE响应“200 OK”后，发送INVITE消息：S4.3. After receiving the MESSAGE response "200 OK", the front-end device sends an INVITE message:
S4.4、前端设备收到INVITE和“200 OK”响应；S4.4, the front-end device receives the INVITE and the "200 OK" response;
S4.5、前端设备回复ACK消息后,开始读取RTP数据包，解析RTP数据包，运行编码逻辑对音频数据解码，输出到前端设备播放，完成交互。S4.5. After the front-end device replies with an ACK message, it starts to read the RTP data packet, parse the RTP data packet, run the encoding logic to decode the audio data, and output it to the front-end device for playback to complete the interaction.
云平台包括：宇视平台或海康平台。Cloud platforms include: Uniview platform or Hikvision platform.
优选的：所述前端设备包括智能头戴计算机、麦克风、布控摄像头、显示光机和智能眼镜处理器。Preferably: the front-end device includes an intelligent head-mounted computer, a microphone, a control camera, a display optical machine and an intelligent glasses processor.
优选的：在所述T-2中，所述数据传输交互，还包括：数据源、设备接入端和国际平台端；Preferably: in said T-2, said data transmission interaction further includes: a data source, a device access terminal and an international platform terminal;
所述数据源包括外部编码前数据、Android摄像头（含Android Camera2）、屏幕数据、编码后数据H264、HEVC、RTSP或RTMP流数据，所述外部编码前数据类型为YV12、NV21、NV12、1420、RGB24、RGBA32、GB565中的任意一种；The data source includes data before external encoding, Android camera (including Android Camera2), screen data, encoded data H264, HEVC, RTSP or RTMP stream data, and the data type before external encoding is any one of YV12, NV21, NV12, 1420, RGB24, RGBA32, GB565;
所述设备接入端，用于国标GB/T 28181设备接入信令注册、流媒体传输、语音广播接收；The device access terminal is used for national standard GB/T 28181 device access signaling registration, streaming media transmission, and voice broadcast reception;
所述国际平台端为国标GB/T 28181平台；The international platform is the national standard GB/T 28181 platform;
所述数据源与设备接入端连接，所述设备接入端与国际平台端交互连接。The data source is connected to the device access terminal, and the device access terminal is interactively connected to the international platform terminal.
第二方面：一种移动信息防止误操作确认交互系统；A second aspect: a mobile information confirmation interaction system to prevent misoperation;
所述系统包括处理器、与所述处理器耦接的存储器，所述存储器中存储有程序指令，所述程序指令被所述处理器执行时，使所述处理器执行如上述所述的移动信息防止误操作确认交互方法。The system includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the mobile information misoperation prevention confirmation interaction method as described above.
（1）处理器：处理器是该交互系统的核心组件，负责执行移动信息防止误操作确认交互方法中的各个步骤。它能够处理输入数据并根据程序指令执行相应的操作，从而实现整个交互方法的运行。(1) Processor: The processor is the core component of the interactive system and is responsible for executing each step in the mobile information error prevention confirmation interactive method. It can process input data and perform corresponding operations according to program instructions, thereby realizing the operation of the entire interactive method.
（2）存储器：存储器与处理器耦接，用于存储程序指令。这些程序指令是实现移动信息防止误操作确认交互方法的关键，能够指导处理器按照事先设计好的步骤进行运算和处理。(2) Memory: The memory is coupled to the processor and is used to store program instructions. These program instructions are the key to implementing the mobile information error-prevention confirmation interaction method and can guide the processor to perform calculations and processing according to pre-designed steps.
（3）程序指令执行：程序指令存储在存储器中，当处理器执行这些指令时，它会按照事先设计的步骤和算法，运算、处理数据，实现移动信息防止误操作确认交互方法中的各个阶段。这些指令将引导处理器完成整个移动信息防止误操作确认交互方法。(3) Program instruction execution: Program instructions are stored in the memory. When the processor executes these instructions, it will calculate and process data according to the pre-designed steps and algorithms to implement each stage of the mobile information error prevention confirmation interaction method. These instructions will guide the processor to complete the entire mobile information error prevention confirmation interaction method.
总结性的，与现有技术相比，本发明所提供的一种移动信息防止误操作确认交互方法的有益效果是：In summary, compared with the prior art, the mobile information error-prevention confirmation interaction method provided by the present invention has the following beneficial effects:
（1）采用“手指口述”技术解决方案，可以实时将现场情况高清传送到远程专家面前，由专家进行会诊，专家无论在PC上还是智能手机上均可与现场人员实时多方高清视频指导，利用冻屏、白板、文件传输，共享屏幕等工具，利用手指口述技术方案共同解决现场遇到的难题。(1) The “finger-to-mouth” technical solution can transmit the on-site situation in high definition to remote experts in real time for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on either PC or smartphone, and use tools such as screen freezing, whiteboard, file transfer, and screen sharing to jointly solve problems encountered on-site using the finger-to-mouth technical solution.
（2）采用移动作业信息交互方式，解决了作业过程中信息交互问题，作业过程全程记录，提升作业安全。(2) The mobile operation information interaction method is adopted to solve the information interaction problem during the operation process. The entire operation process is recorded to improve operation safety.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
图1为本发明的方法流程逻辑示意图；FIG1 is a logic diagram of the method flow of the present invention;
图2为本发明的SIP服务器与SIP代理间的交互逻辑示意简图；FIG2 is a schematic diagram of the interaction logic between the SIP server and the SIP proxy of the present invention;
图3为本发明的方法实施例三的数据传输交互流程示意图；FIG3 is a schematic diagram of a data transmission interaction process of a method embodiment 3 of the present invention;
图4为本发明的方法实施例三的数据传输交互逻辑框图；FIG4 is a data transmission interaction logic block diagram of a third method embodiment of the present invention;
图5为本发明的方法实施例四的流程逻辑示意图；FIG5 is a flow chart of a fourth embodiment of the method of the present invention;
图6为本发明的方法实施例五的流程逻辑示意图；FIG6 is a flow chart of a fifth embodiment of the method of the present invention;
图7为本发明的方法实施例六的流程逻辑示意图；FIG7 is a flow chart of a sixth embodiment of the method of the present invention;
图8为本发明的方法实施例七的流程逻辑示意图。FIG8 is a flow chart of the logic of the seventh embodiment of the method of the present invention.
具体实施方式Detailed ways
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图对本发明的具体实施方式做详细的说明。在下面的描述中阐述了很多具体细节以便于充分理解本发明。但是本发明能够以很多不同于在此描述的其它方式来实施，本领域技术人员可以在不违背本发明内涵的情况下做类似改进，因此本发明不受下面公开的具体实施例的限制；In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;
需要指出的是，本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
对于本领域技术人员而言，还可以进一步意识到结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
需要指出的是，结合本文中所公开的实施例描述的方法或算法的步骤可以直接用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。It should be noted that the steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
实施例一：请参阅图1，本实施例提供一种移动信息防止误操作确认交互方法，包括如下步骤T-1、T-2、T-3和T-4：Embodiment 1: Please refer to FIG. 1 . This embodiment provides a mobile information confirmation interaction method for preventing misoperation, including the following steps T-1, T-2, T-3 and T-4:
所述T-1、信令注册：前端设备首次接入平台时，会持续发送REGISTER消息直到服务器回应“200 OK”表示成功注册，若失败则会随机延迟后重新尝试注册，同时，所述前端设备会定期刷新注册；T-1, signaling registration: When the front-end device accesses the platform for the first time, it will continue to send REGISTER messages until the server responds with "200 OK" to indicate successful registration. If it fails, it will try to register again after a random delay. At the same time, the front-end device will refresh the registration periodically;
所述T-2、数据传输交互：信令注册成功后，服务器会发送目录信息给客户端，客户端构建内容并将其发送给服务器，服务器进行校验并返回INVITE响应；T-2, data transmission interaction: After the signaling registration is successful, the server will send the directory information to the client, the client will build the content and send it to the server, the server will verify it and return the INVITE response;
所述T-3、获取前端设备音视频编解码数据；其中，提取所述音视频编解码数据的关联信息，所述关联信息包括：T-3, obtaining audio and video codec data of the front-end device; wherein, extracting associated information of the audio and video codec data, the associated information includes:
S3.1、视频编码：把相机采集到的数据通过MediaCodec编码h264，再使用RTP协议打包PS进行传输；S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;
S3.2、音频编码：通过AudioRecord来采集声音进行G711编码RTP打包传输发送；S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711 format, and then transmit it in RTP format.
S3.3、混流编码：通过MediaMuxer把音视频混流，初始化MediaExtractor用于从视频文件和音频文件中提取视频和音频轨道信息并获取其格式；S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, initialize MediaExtractor to extract video and audio track information from video files and audio files and obtain their formats;
所述T-4，获取云平台的音频媒体流交互：前端设备与云平台通过信令来建立广播连接，最后实现音视频交互。The T-4 obtains the audio media stream interaction of the cloud platform: the front-end device and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction.
在本实施例中，关于T-1：信令注册：前端设备首次接入平台时，会持续发送REGISTER消息直到服务器回应“200 OK”表示成功注册，若失败则会随机延迟后重新尝试注册，同时，所述前端设备会定期刷新注册。T-1的具体实施方式请参阅实施例二。In this embodiment, regarding T-1: signaling registration: when the front-end device first accesses the platform, it will continue to send REGISTER messages until the server responds with "200 OK" indicating successful registration. If it fails, it will retry registration after a random delay. At the same time, the front-end device will refresh the registration periodically. For the specific implementation of T-1, please refer to Example 2.
请参阅图2，具体的，在本实施例中，以基本注册为例：基本注册采用IETFRFC3261规定的基于数字摘要的挑战应答式安全技术进行注册。Please refer to FIG. 2 . Specifically, in this embodiment, basic registration is taken as an example: basic registration is performed using a digital digest-based challenge-response security technology specified in IETF RFC3261.
具体的，所述前端设备包括智能头戴计算机、麦克风、布控摄像头、显示光机和智能眼镜处理器。Specifically, the front-end device includes an intelligent head-mounted computer, a microphone, a control camera, a display optical machine and an intelligent glasses processor.
具体的，当智能头戴计算机设备第一次接入平台端时，设备端将持续向平台端发送 REGISTER消息，直到 Server端回复“200 OK”代表注册成功。如果设备或系统注册不成功，建议延迟一定的随机时间后重新注册。每隔一定时间客户端都会再次向服务器刷新注册，防止注册过期导致连接断开。Specifically, when the smart head-mounted computer device is connected to the platform for the first time, the device will continue to send REGISTER messages to the platform until the server replies "200 OK" to indicate successful registration. If the device or system registration is unsuccessful, it is recommended to re-register after a certain random delay. The client will refresh the registration with the server again at regular intervals to prevent the connection from being disconnected due to registration expiration.
在本实施例中，关于T-2：数据传输交互：信令注册成功后，服务器会发送目录信息给客户端，客户端构建内容并将其发送给服务器，服务器进行校验并返回INVITE响应。T-2的具体实施方式请参阅实施例三。In this embodiment, regarding T-2: Data transmission interaction: After the signaling registration is successful, the server will send directory information to the client, the client will build the content and send it to the server, the server will verify and return the INVITE response. For the specific implementation of T-2, please refer to Example 3.
在本实施例中，关于T-3：获取前端设备音视频编解码数据；其中，提取所述音视频编解码数据的关联信息，所述关联信息包括：In this embodiment, regarding T-3: obtaining audio and video codec data of a front-end device; wherein, extracting associated information of the audio and video codec data, the associated information includes:
S3.1、视频编码：把相机采集到的数据通过MediaCodec编码h264，再使用RTP协议打包PS进行传输；S3.1, Video encoding: The data collected by the camera is encoded into h264 through MediaCodec, and then packaged into PS using RTP protocol for transmission;
S3.2、音频编码：通过AudioRecord来采集声音进行G711编码RTP打包传输发送；S3.2, Audio encoding: AudioRecord is used to collect sound, encode it in G711, and then transmit it in RTP packaging;
S3.3、混流编码：通过MediaMuxer把音视频混流，初始化MediaExtractor用于从视频文件和音频文件中提取视频和音频轨道信息并获取其格式。T-3的具体实施方式请参阅实施例四、实施例五和实施例六。S3.3, mixed stream encoding: Mix the audio and video streams through MediaMuxer, and initialize MediaExtractor to extract video and audio track information from the video file and audio file and obtain their formats. For the specific implementation of T-3, please refer to Example 4, Example 5 and Example 6.
在本实施例中，关于T-4：获取云平台的音频媒体流交互：前端设备与云平台通过信令来建立广播连接，最后实现音视频交互。T-4的具体实施方式请参阅实施例七。In this embodiment, regarding T-4: obtaining audio media stream interaction of the cloud platform: the front-end device and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction. For the specific implementation of T-4, please refer to Example 7.
总结性的，采用“手指口述”技术解决方案，可以实时将现场情况高清传送到远程专家面前，由专家进行会诊，专家无论在PC上还是智能手机上均可与现场人员实时多方高清视频指导，利用冻屏、白板、文件传输，共享屏幕等工具，利用手指口述技术方案共同解决现场遇到的难题。In summary, the "finger dictation" technical solution can transmit the on-site situation in real time and in high definition to remote experts for consultation. Experts can provide real-time multi-party high-definition video guidance with on-site personnel on both PCs and smartphones, and use tools such as freezing the screen, whiteboards, file transfer, and sharing the screen, and use the finger dictation technical solution to jointly solve problems encountered on-site.
实施例二：请参阅图2，在实施例一的基础上，本实施例将进一步公开T-1的具体实施方式：在T-1中，包括：Example 2: Please refer to Figure 2. Based on Example 1, this example will further disclose the specific implementation of T-1: In T-1, it includes:
1）客户端向SIP服务器发起注册请求，服务器收到请求后对注册信息进行校验，校验通过后向客户端发送响应401，并在响应的header WWW-Authenticate字段中给出客户端的认证体制和参数；1) The client initiates a registration request to the SIP server. After receiving the request, the server verifies the registration information. After the verification is passed, it sends a 401 response to the client and gives the client's authentication system and parameters in the header WWW-Authenticate field of the response.
2）客户端收到认证体制和参数数据后重新向 SIP 服务器发送注册请求，在请求的 Authorization 字段给出信任书并携带认证信息；2) After receiving the authentication system and parameter data, the client resends a registration request to the SIP server, providing a letter of trust and carrying authentication information in the Authorization field of the request;
3）SIP 服务器收到请求对请求再次进行校验，校验之后，若客户端身份合法，则向客户端发送成功响应“200 OK”，若客户端身份不合法则会发送本次服务拒绝服务应答消息。3) The SIP server receives the request and verifies it again. After verification, if the client identity is legal, it sends a successful response "200 OK" to the client. If the client identity is illegal, it sends a service denial response message.
具体的：公开T-1的具体实施方式的特点：Specific: Features of the specific implementation of T-1 are disclosed:
1）提高了安全性：通过多次身份验证和信息校验，提高了SIP服务器对请求者身份的确认度，使得非法用户更难通过验证。1) Improved security: Through multiple identity authentication and information verification, the SIP server improves the degree of confirmation of the requester's identity, making it more difficult for illegal users to pass the verification.
2）客户端和服务器双向交互：客户端和服务器的通信不仅仅是单向的，服务器首先发送认证体制和参数给客户端，然后客户端再根据这些信息回应。这样的双向交互提高了服务器和客户端间的信息同步程度。2) Two-way interaction between client and server: The communication between client and server is not just one-way. The server first sends the authentication system and parameters to the client, and then the client responds based on this information. This two-way interaction improves the degree of information synchronization between the server and the client.
3）增强了系统的稳健性：收到身份不合法的请求时，服务器不会尝试处理错误的请求，而是直接返回服务拒绝的应答消息，避免了可能的错误处理，保障了系统运行稳定。3) Enhanced system robustness: When receiving a request with an illegal identity, the server will not attempt to process the erroneous request, but will directly return a response message of service rejection, avoiding possible error handling and ensuring stable system operation.
4）灵活的应答模式：客户端根据服务器的应答以及携带的参数信息，以相应的方式给出响应，如若积极响应服务器需向客户端发送成功响应“200 OK”，使得整个过程更加机动灵活。4) Flexible response mode: The client responds in a corresponding manner based on the server's response and the parameter information it carries. If the response is positive, the server needs to send a successful response "200 OK" to the client, making the whole process more flexible.
总结性的，在本实施例中的T-1的具体实施方式使得验证过程更灵活，也大幅度提高了安全性和系统稳健性。In summary, the specific implementation of T-1 in this embodiment makes the verification process more flexible and greatly improves the security and system robustness.
实施例三：请参阅图3，在本实施例中，在实施例一的基础上，将进一步公开T-2的具体实施方式：Example 3: Please refer to Figure 3. In this example, based on Example 1, the specific implementation of T-2 will be further disclosed:
所述数据传输交互：具体步骤包括：The data transmission interaction: the specific steps include:
S2.1、信令注册成功后服务端会发送CATALOG目录信息给客户端；S2.1. After the signaling registration is successful, the server will send the CATALOG directory information to the client;
S2.2、客户端收到CATALOG消息并对内容进行构建，完成后发送给服务端；S2.2, the client receives the CATALOG message and constructs the content, and sends it to the server after completion;
S2.3、服务端收到CATALOG信息后对内容进行校验，校验通过后服务端发送INVITE响应给客户端；S2.3, after receiving the CATALOG information, the server verifies the content. After the verification passes, the server sends an INVITE response to the client;
S2.4、客户端收到INVITE响应后对INVITE信息与SDP信息进行封装并对服务端做出回复；客户端根据服务端发过来的INVITE信息与SDP信息获取到RTP传输的端口，其中所述SDP信息包括TCP、UDP模式和SSRC信息；S2.4. After receiving the INVITE response, the client encapsulates the INVITE information and SDP information and responds to the server. The client obtains the RTP transmission port according to the INVITE information and SDP information sent by the server, wherein the SDP information includes TCP, UDP mode and SSRC information.
S2.5、所有连接建立之后客户端把采集到的音视频数据，打包成符合国标GB/T28181协议的PS流，基于RTP封包的形式发送到服务端；S2.5. After all connections are established, the client packages the collected audio and video data into a PS stream that complies with the national standard GB/T28181 protocol and sends it to the server in the form of RTP packets;
对于所述国标GB/T28181协议中常用的数据流打包格式PS流方式，将SDP文件中的视频属性字段集合中描述流结构的字段写成：a=rtpmap：96MP2P/90000，a=rtpmap为固定格式，96为负载类型，MP2P为编码类型，90000为时钟频率；接着上一行a=rtpmap：96MP2P/90000，设计a=fmtp：96，其中a=fmtp为固定格式，96为负载类型；For the PS stream mode, a data stream packaging format commonly used in the national standard GB/T28181 protocol, the field describing the stream structure in the video attribute field set in the SDP file is written as: a=rtpmap:96MP2P/90000, a=rtpmap is a fixed format, 96 is the payload type, MP2P is the encoding type, and 90000 is the clock frequency; following the previous line a=rtpmap:96MP2P/90000, a=fmtp:96 is designed, where a=fmtp is a fixed format and 96 is the payload type;
在传输过程中客户端必须间歇发送KEEP ALIVE心跳包给服务端来维持长连接，避免断开导致数据传输失败。During the transmission process, the client must intermittently send KEEP ALIVE heartbeat packets to the server to maintain a long connection and avoid disconnection that causes data transmission failure.
S2.6、若客户端需要断开Invite会话连接，则会需要发送BYE请求给服务端，服务端返回“200 OK”之后断开连接；S2.6. If the client needs to disconnect the Invite session, it will need to send a BYE request to the server. The server will disconnect after returning "200 OK";
S2.7、若客户端发起注销：发送 Register命令消息给服务端，消息中的 Expire字段设置为0即代表注销，此时中止会话。S2.7. If the client initiates a logout: it sends a Register command message to the server, and the Expire field in the message is set to 0, which means logout, and the session is terminated at this time.
具体的，完整的数据传输和交互过程，具有安全、可控和高效的特点：Specifically, the complete data transmission and interaction process is safe, controllable and efficient:
1）安全性：在连接建立之前，服务器对客户端的请求信息进行了多轮校验，这有利于防止非法设备的接入，确保数据的安全传输。1) Security: Before the connection is established, the server performs multiple rounds of verification on the client's request information, which helps prevent access by illegal devices and ensures secure data transmission.
2）可控性：通过SIP协议，客户端与服务端可以进行详细的会话控制，包括会话的建立、修改和终止等，符合通信的实时性和可控性需求。2) Controllability: Through the SIP protocol, the client and the server can perform detailed session control, including session establishment, modification, and termination, meeting the real-time and controllability requirements of communication.
3）高效性：所有的音视频数据在传输前，都会被打包成符合国标GB/T28181协议的PS流，基于RTP封包的形式发送，保证了数据传输的高效和稳定。并且在SDP文件中详细描述了流结构的字段，提供了清晰的数据描述，便于后续的数据处理。3) Efficiency: All audio and video data will be packaged into PS streams that comply with the national standard GB/T28181 protocol before transmission, and sent in the form of RTP packets, ensuring efficient and stable data transmission. The fields of the stream structure are described in detail in the SDP file, providing a clear data description to facilitate subsequent data processing.
4）灵活性：当客户端需要断开连接或注销时，都有相应的指令可以发送，使得整个过程更加灵活，便于实用。4) Flexibility: When the client needs to disconnect or log out, corresponding instructions can be sent, making the whole process more flexible and practical.
请参阅图4，具体的：在所述T-2中，所述数据传输交互，还包括：数据源、设备接入端和国际平台端；Please refer to FIG. 4 , specifically: in the T-2, the data transmission interaction further includes: a data source, a device access terminal and an international platform terminal;
所述数据源包括外部编码前数据、Android摄像头（含Android Camera2）、屏幕数据、编码后数据H264、HEVC、RTSP或RTMP流数据，所述外部编码前数据类型为YV12、NV21、NV12、1420、RGB24、RGBA32、GB565中的任意一种；The data source includes data before external encoding, Android camera (including Android Camera2), screen data, encoded data H264, HEVC, RTSP or RTMP stream data, and the data type before external encoding is any one of YV12, NV21, NV12, 1420, RGB24, RGBA32, GB565;
所述设备接入端，用于国标GB/T 28181设备接入信令注册、流媒体传输、语音广播接收；The device access terminal is used for national standard GB/T 28181 device access signaling registration, streaming media transmission, and voice broadcast reception;
所述国际平台端为国标GB/T 28181平台；The international platform is the national standard GB/T 28181 platform;
所述数据源与设备接入端连接，所述设备接入端与国际平台端交互连接。The data source is connected to the device access terminal, and the device access terminal is interactively connected to the international platform terminal.
总结性的，在本实施例中的T-2的具体实施方式使得整个数据传输过程有良好的安全性、可控性和高效性，并且能够灵活应对不同的使用需求。In summary, the specific implementation of T-2 in this embodiment makes the entire data transmission process have good security, controllability and efficiency, and can flexibly respond to different usage requirements.
实施例四：请参阅图5，在本实施例中，在实施例一的基础上，本实施例将进一步公开S3.1中，所述视频编码：把相机采集到的数据通过MediaCodec编码h264，再使用RTP协议打包PS进行传输，具体实施方式：Embodiment 4: Please refer to FIG. 5. In this embodiment, based on Embodiment 1, this embodiment further discloses the video encoding in S3.1: the data collected by the camera is encoded into h264 by MediaCodec, and then packaged into PS for transmission using the RTP protocol. The specific implementation method is as follows:
S3.11、初始化获取设备Camera2通过PreviewFrame实时监听检测相机采集返回的原始数据；S3.11, initialize and obtain the original data returned by the device Camera2 through PreviewFrame real-time monitoring detection camera collection;
S3.12、把采集到的NV21数据转成h264编码为YUV420P数据；S3.12, convert the collected NV21 data into h264 encoding into YUV420P data;
S3.13、创建MediaCodec实例设置编码格式参数，对YUV数据进行h264硬编码处理；S3.13. Create a MediaCodec instance to set encoding format parameters and perform h264 hard encoding on YUV data;
S3.14、获取编码后的h264数据，对所述h264数据进行国标GB/T28181协议的PS格式封装；S3.14, obtaining the encoded h264 data, and encapsulating the h264 data in PS format of the national standard GB/T28181 protocol;
S3.15、基于RTP/UDP把封装后的数据进行发送到服务端。S3.15. Send the encapsulated data to the server based on RTP/UDP.
具体的示例，初始化获取设备Camera2实例，通过调用ImageReader.newInstance方法来获取相机开启预览返回的原始数据流；创建MediaCodec实例，设置编码格式等参数，用来对相机原始数据进行编码处理，当相机开启之后通过实现ImageReader.OnImageAvailableListener回调不断获取到原始YUV数据，创建byte数组对该数据进行一次缓冲处理避免堵塞导致数据回调中止，调用MediaCodec对YUV数据进行h264编码，当开始编解码操作时，开启编解码线程，处理相机预览返回的 YUV 数据，通过 dequeueInputBuffer 方法获取可用的输入缓冲区的索引；In the specific example, initialize and obtain the device Camera2 instance, call the ImageReader.newInstance method to obtain the original data stream returned by the camera preview; create a MediaCodec instance, set the encoding format and other parameters to encode the camera raw data, and when the camera is turned on, continuously obtain the original YUV data by implementing the ImageReader.OnImageAvailableListener callback, create a byte array to buffer the data to avoid blockage and data callback termination, call MediaCodec to encode the YUV data in h264, and when the encoding and decoding operation starts, start the encoding and decoding thread, process the YUV data returned by the camera preview, and obtain the index of the available input buffer through the dequeueInputBuffer method;
调用getInputBuffer获取输入缓存区，把数据写入缓存区，填充完数据后通过queueInputBuffer 方法入队，调用queueInputBuffer将填满数据的inputBuffer提交到编码队列进行编码，编码完成后调用dequeueOutputBuffer获取已成功编解码的输出缓冲区的索引，最后获取输出缓存区得到编码后的h264数据，完成后释放缓存区，再对h264数据进行PS流打包，按RTP协议发送即可。Call getInputBuffer to get the input buffer, write the data into the buffer, and queue it through the queueInputBuffer method after filling the data. Call queueInputBuffer to submit the inputBuffer filled with data to the encoding queue for encoding. After the encoding is completed, call dequeueOutputBuffer to obtain the index of the output buffer that has been successfully encoded and decoded. Finally, get the output buffer to get the encoded h264 data. After completion, release the buffer, package the h264 data into PS stream, and send it according to the RTP protocol.
总结性的，在本实施例中的S3.1的具体实施方式，具有下列特点：In summary, the specific implementation of S3.1 in this embodiment has the following features:
1）效率高：通过MediaCodec直接进行h264硬编码处理，避免了软编码的复杂计算过程，大幅提高了编码效率。1) High efficiency: Directly performing h264 hard encoding processing through MediaCodec avoids the complex calculation process of soft encoding and greatly improves the encoding efficiency.
2）实时性好：获取设备Camera原始数据，通过PreviewFrame实时监听，能够快速获取并转换相机采集返回的数据。2) Good real-time performance: Get the original data of the device Camera, and monitor it in real time through PreviewFrame, so as to quickly obtain and convert the data returned by the camera.
3）兼容性强：对采集到的NV21数据进行h264编码为YUV420P数据，并且采用了被广泛接受的国标GB/T28181协议的PS格式封装，这样不仅提高了传输效率，还增强了设备之间的兼容性。3) Strong compatibility: The collected NV21 data is encoded into YUV420P data using h264, and encapsulated in the PS format of the widely accepted national standard GB/T28181 protocol, which not only improves transmission efficiency but also enhances compatibility between devices.
4）易于传输：基于RTP/UDP把封装后的数据进行发送，能够保证数据在网络中顺利传输，使得数据能够更加顺利的进行传输，且在数据丢失的情况下还能保证视频的连续性。4) Easy to transmit: Sending encapsulated data based on RTP/UDP can ensure smooth data transmission in the network, making data transmission more smooth and ensuring video continuity in the event of data loss.
5）适应性强：因为视频数据被打包为PS流，这意味着可以对一系列的音视频编码进行打包，可以灵活处理多种格式的音视频数据。5) Strong adaptability: Because the video data is packaged as a PS stream, it means that a series of audio and video encodings can be packaged, and audio and video data in various formats can be flexibly processed.
实施例五：请参阅图6，在本实施例中，在实施例一的基础上，将进一步公开S3.2中，所述音频编码：通过AudioRecord来采集声音进行G711编码RTP打包传输发送，具体实施方式：Embodiment 5: Please refer to FIG. 6. In this embodiment, based on Embodiment 1, the audio encoding in S3.2 is further disclosed: the audio is collected through AudioRecord for G711 encoding and RTP packaging transmission and transmission. The specific implementation method is as follows:
S3.21、初始化AudioRecord设置编码格式参数，采集音频数据；S3.21, initialize AudioRecord to set encoding format parameters and collect audio data;
S3.22、开启录音后通过audioRecord.read将获取到的音频数据写入创建好的字节缓存区；S3.22, after starting recording, write the acquired audio data into the created byte buffer through audioRecord.read;
S3.23、读取缓存区内的数据并进行short类型转换，把转换后的short数据进行G711编码；S3.23, read the data in the buffer area and perform short type conversion, and encode the converted short data into G711;
S3.24、编码后的数据通过RTP协议打包发送服务端。S3.24. The encoded data is packaged and sent to the server via the RTP protocol.
具体的，初始化AudioRecord设置编码格式等参数，用来采集音频数据，当开启录音之后通过audioRecord.read 来实时监听是否有音频数据，当有数据时把数据写入创建好的byte字节缓存区，如果缓存区长度大于0时，读取缓存区内的数据并转为short类型，把转换后的short数据进行G711编码，编码完成后通过RTP协议打包发送服务端。Specifically, initialize AudioRecord to set parameters such as the encoding format to collect audio data. After starting recording, use audioRecord.read to monitor in real time whether there is audio data. When there is data, write the data into the created byte buffer. If the buffer length is greater than 0, read the data in the buffer and convert it to short type. Perform G711 encoding on the converted short data. After encoding, package it and send it to the server through the RTP protocol.
总结性的，在本实施例中的S3.2的具体实施方式，具有下列特点：In summary, the specific implementation of S3.2 in this embodiment has the following characteristics:
1）效率高：通过AudioRecord技术直接获取音频数据，并进行G711编码，相比于其他编码方式，G711编码可提供较高的音频质量并且对硬件要求低。1) High efficiency: Audio data is directly obtained through AudioRecord technology and G711 encoding is performed. Compared with other encoding methods, G711 encoding can provide higher audio quality and has low hardware requirements.
2）质量好：使用RTP协议进行音频数据的封装和传输，可以保证音频数据在网络中的顺利传输，降低数据丢失对音频质量的影响。2) Good quality: Using the RTP protocol to encapsulate and transmit audio data can ensure the smooth transmission of audio data in the network and reduce the impact of data loss on audio quality.
3）实时性强：方案在录音时就进行了数据的读取并且编码，实现了音频的实时传输。3) Strong real-time performance: The solution reads and encodes data during recording, realizing real-time transmission of audio.
4）操作简单：通过audioRecord.read就可以直接读取音频数据，随后只需对音频数据进行简单的short类型转换和编码即可，降低了操作复杂性。4) Simple operation: Audio data can be directly read through audioRecord.read, and then only simple short type conversion and encoding of the audio data is required, which reduces the complexity of operation.
5）兼容性好：因为采用了国际通用的G711音频编码方式和RTP协议，使得该方案具有良好的设备兼容性和系统兼容性。5) Good compatibility: Because it adopts the internationally accepted G711 audio encoding method and RTP protocol, the solution has good device compatibility and system compatibility.
S3.2的具体实施方式使得音频编码和传输过程实时性强，效率高且兼容性好。The specific implementation of S3.2 makes the audio encoding and transmission process highly real-time, efficient and compatible.
实施例六：请参阅图7，在本实施例中，在实施例一的基础上，将进一步公开S3.3中，所述混流编码：通过MediaMuxer把音视频混流，初始化MediaExtractor用于从视频文件和音频文件中提取视频和音频轨道信息并获取其格式；Embodiment 6: Please refer to FIG. 7. In this embodiment, based on Embodiment 1, the mixed stream encoding in S3.3 is further disclosed: the audio and video are mixed by MediaMuxer, and MediaExtractor is initialized to extract video and audio track information from video files and audio files and obtain their formats;
S3.31、初始化MediaExtractor从视频文件和音频文件中提取视频和音频轨道信息；S3.31, initialize MediaExtractor to extract video and audio track information from video files and audio files;
S3.32、用MediaExtractor.readSampleData从视频文件和音频文件各自读取一帧数据到缓存区；S3.32, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file respectively into the buffer area;
S3.33、实例化MediaMuxer对象将缓存区内的音频和视频数据合并输出；S3.33, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output them;
S3.34、设置缓存区信息，包含当前帧的时间戳和标志位，判定是否为关键帧；S3.34, set the buffer information, including the timestamp and flag of the current frame, to determine whether it is a key frame;
S3.35、基于RTP/UDP使用DatagramSocket进行发送。S3.35. Send using DatagramSocket based on RTP/UDP.
具体的，使用MediaExtractor.readSampleData 从视频文件和音频文件各读取一帧数据到缓存区，当数据大于0时，通过实例化MediaMuxer对象将缓存区内的音频和视频数据合并输出，输出时设置缓存区信息，包含当前帧的时间戳和标志位是否为关键帧等，然后继续移动到下一帧进行合并，使用DatagramSocket.send 进行发送。Specifically, use MediaExtractor.readSampleData to read one frame of data from the video file and audio file into the buffer. When the data is greater than 0, instantiate the MediaMuxer object to merge the audio and video data in the buffer and output it. Set the buffer information during output, including the timestamp of the current frame and whether the flag is a key frame, etc. Then move to the next frame for merging and use DatagramSocket.send to send it.
总结性的，S3.3中，具体实施方式，所述混流编码的特点：In summary, in S3.3, the specific implementation method, the characteristics of the mixed stream encoding are:
1）高效性：通过使用MediaExtractor和MediaMuxer，可以在不需要解码和重新编码的情况下，直接从原始文件中提取音频和视频数据，进行混流，这大大提高了处理效率。1) Efficiency: By using MediaExtractor and MediaMuxer, audio and video data can be extracted directly from the original files and mixed without decoding and re-encoding, which greatly improves processing efficiency.
2）灵活性：方案中对每一帧数据都进行了详细的设置，包括时间戳和关键帧标志，这不仅便于后续的数据处理和解码，也使得视频流的控制更为灵活。2) Flexibility: The solution provides detailed settings for each frame of data, including timestamps and key frame flags, which not only facilitates subsequent data processing and decoding, but also makes the control of the video stream more flexible.
3）操作简便：只需要初始化相应的对象就可以实现对音频视频数据的读取和混流，操作简便易行，无需复杂的数据处理步骤。3) Easy operation: You only need to initialize the corresponding objects to read and mix audio and video data. The operation is simple and easy, and no complicated data processing steps are required.
4）兼容性强：采用了被广泛接受的RTP/UDP协议进行数据的传输，基于DatagramSocket发送，与各种设备和系统具有良好的兼容性。4) Strong compatibility: It uses the widely accepted RTP/UDP protocol for data transmission, and is sent based on DatagramSocket, which has good compatibility with various devices and systems.
5）可扩展性强：基于这个方案，可以容易地进行拓展，例如添加新的音频轨道，修改视频质量等。5) Strong scalability: Based on this solution, it can be easily expanded, such as adding new audio tracks, modifying video quality, etc.
实施例七：请参阅图8，在本实施例中，在实施例一的基础上，将进一步公开T-4：获取云平台的音频媒体流交互：前端设备与云平台通过信令来建立广播连接，最后实现音视频交互的具体实施方式：Embodiment 7: Please refer to FIG. 8. In this embodiment, based on Embodiment 1, T-4: Obtaining the audio media stream interaction of the cloud platform: The front-end device and the cloud platform establish a broadcast connection through signaling, and finally realize the specific implementation method of audio and video interaction:
所述获取云平台的音频媒体流交互：The audio media stream interaction of the cloud platform is obtained:
S4.1、云平台发MESSAGE语音广播命令到前端设备；S4.1. The cloud platform sends a MESSAGE voice broadcast command to the front-end device;
S4.2、前端设备发MESSAGE消息应答语音广播命令；S4.2, the front-end device sends a MESSAGE message to respond to the voice broadcast command;
S4.3、前端设备收到MESSAGE响应“200 OK”后，发送INVITE消息：S4.3. After receiving the MESSAGE response "200 OK", the front-end device sends an INVITE message:
S4.4、前端设备收到INVITE和“200 OK”响应；S4.4, the front-end device receives the INVITE and the "200 OK" response;
S4.5、前端设备回复ACK消息后,开始读取RTP数据包，解析RTP数据包，运行编码逻辑对音频数据解码，输出到前端设备播放，完成交互。S4.5. After the front-end device replies with an ACK message, it starts to read the RTP data packet, parse the RTP data packet, run the encoding logic to decode the audio data, and output it to the front-end device for playback to complete the interaction.
具体的，公开T-4的具体实施方式的特点：Specifically, the features of the specific implementation of T-4 are disclosed:
1）有效性：通过清晰的命令响应过程，云平台和前端设备之间的通信得以有效进行。云平台发出指令，前端设备应答指令，并且根据收到的响应进行下一步动作，这保证了整个音频媒体流获取过程的有效性。1) Effectiveness: Through a clear command response process, the communication between the cloud platform and the front-end device is carried out effectively. The cloud platform issues a command, the front-end device responds to the command, and takes the next step based on the response received, which ensures the effectiveness of the entire audio media stream acquisition process.
2）实时性：在整个交互过程中，前端设备能够实时响应云平台的指令并进行下一步操作，这保证了获取云平台的音频媒体流交互的实时性。2) Real-time: During the entire interaction process, the front-end device can respond to the cloud platform's instructions in real time and perform the next operation, which ensures the real-time interaction of the audio media stream of the cloud platform.
3）稳定性：收到“200 OK”响应后，前端设备回复ACK消息并进行后续处理，这一过程体现了稳定的通信过程。即使在网络波动或其他情况下，通过回复ACK消息保证了通讯的可靠性。3) Stability: After receiving the "200 OK" response, the front-end device replies with an ACK message and performs subsequent processing. This process reflects a stable communication process. Even in the event of network fluctuations or other situations, the reliability of communication is guaranteed by replying with an ACK message.
4）易于解码：在收到RTP数据包后，前端设备解析数据包并运行编码逻辑进行音频数据解码，这使得音频数据可以在前端设备上顺利播放，完成交互。4) Easy to decode: After receiving the RTP data packet, the front-end device parses the data packet and runs the encoding logic to decode the audio data, which allows the audio data to be played smoothly on the front-end device and complete the interaction.
实施例八：本实施例进一步公开一种移动信息防止误操作确认交互系统；该系统包括处理器、与处理器耦接的存储器，存储器中存储有程序指令，程序指令被处理器执行时，使处理器执行如上述的移动信息防止误操作确认交互方法。具体的：Embodiment 8: This embodiment further discloses a mobile information error-prevention operation confirmation interaction system; the system includes a processor and a memory coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the mobile information error-prevention operation confirmation interaction method as described above. Specifically:
存储器存储程序指令，程序指令被处理器执行时，使得处理器执行上述实施例中的移动信息防止误操作确认交互方法的步骤。The memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the steps of the mobile information prevention misoperation confirmation interaction method in the above embodiment.
处理器：处理器是该交互系统的核心组件，负责执行移动信息防止误操作确认交互方法中的各个步骤。它能够处理输入数据并根据程序指令执行相应的操作，从而实现整个交互方法的运行。Processor: The processor is the core component of the interactive system and is responsible for executing each step in the mobile information to prevent misoperation confirmation interactive method. It can process input data and perform corresponding operations according to program instructions, thereby realizing the operation of the entire interactive method.
其中，处理器还可以称为CPU(Central Processing Unit，中央处理单元)。处理器可能是一种集成电路芯片，具有信号的处理能力。处理器还可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
（2）存储器：存储器与处理器耦接，用于存储程序指令。这些程序指令是实现移动信息防止误操作确认交互方法的关键，能够指导处理器按照事先设计好的步骤进行运算和处理。(2) Memory: The memory is coupled to the processor and is used to store program instructions. These program instructions are the key to implementing the mobile information error-prevention confirmation interaction method and can guide the processor to perform calculations and processing according to pre-designed steps.
（3）程序指令执行：程序指令存储在存储器中，当处理器执行这些指令时，它会按照事先设计的步骤和算法，运算、处理数据，实现移动信息防止误操作确认交互方法中的各个阶段。这些指令将引导处理器完成整个基于元胞自动机的移动信息防止误操作确认交互方法。(3) Program instruction execution: Program instructions are stored in the memory. When the processor executes these instructions, it will calculate and process data according to the pre-designed steps and algorithms to implement each stage of the mobile information error prevention confirmation interaction method. These instructions will guide the processor to complete the entire mobile information error prevention confirmation interaction method based on cellular automation.
本申请实施例的存储介质存储有程序指令，程序指令被处理器执行以实现上述实施例中的移动信息防止误操作确认交互方法，其中，该计算机程序可以以软件产品的形式存储在上述存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random AccessMemory)、磁碟或者光盘等各种可以存储程序代码的介质，或者是计算机、服务器、手机、平板等计算机设备。服务器可以是独立的服务器，也可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(ContentDelivery Network，CDN)、以及大数据和人工智能平台等基础云计算服务的云服务器。The storage medium of the embodiment of the present application stores program instructions, and the program instructions are executed by the processor to implement the mobile information prevention misoperation confirmation interaction method in the above embodiment, wherein the computer program can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, and a tablet. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
（4）系统工作流程：(4) System workflow:
P1、输入数据导入：获取前端设备音视频编解码数据作为输入被导入系统。P1. Input data import: The audio and video codec data of the front-end device is obtained and imported into the system as input.
P2、程序指令执行：处理器根据存储器中的程序指令执行移动信息防止误操作确认交互方法，包括T-1、T-2、T-3和T-4的各个步骤。P2. Program instruction execution: The processor executes the mobile information to prevent misoperation confirmation interaction method according to the program instructions in the memory, including steps T-1, T-2, T-3 and T-4.
P3、获取云平台的音频媒体流交互：前端设备与云平台通过信令来建立广播连接，最后实现音视频交互。P3. Obtain audio media stream interaction from the cloud platform: The front-end device and the cloud platform establish a broadcast connection through signaling, and finally realize audio and video interaction.
进一步的，下文将公开本实施例中存储器内存储的控制程序（仅供参考），本实施例以C++伪代码的形式仅展示其运算的逻辑：Further, the control program stored in the memory in this embodiment will be disclosed below (for reference only). This embodiment only shows the logic of its operation in the form of C++ pseudo code:
// P1: 输入数据导入class Device {// P1: Input data import class Device {
public:public:
MediaStream getMediaData();MediaStream getMediaData();
};};
Device frontendDevice;Device frontendDevice;
MediaStream inputData = frontendDevice.getMediaData();MediaStream inputData = frontendDevice.getMediaData();
// P2: 程序指令执行class Processor// P2: Program instruction execution class Processor
{public:{public:
void executeIns(T instruction); // T代表指令的类型};void executeIns(T instruction); // T represents the type of instruction};
Processor cpu;Processor cpu;
T instruction;T instruction;
cpu.executeIns(instruction);cpu.executeIns(instruction);
// P3: 获取云平台音视频媒体流交互// P3: Get the audio and video media stream interaction of the cloud platform
class CloudPlatformclass CloudPlatform
{public:{public:
void connect(Device device);void connect(Device device);
void interact(MediaStream mediaStream);};void interact(MediaStream mediaStream);};
CloudPlatform cloudPlatform;CloudPlatform cloudPlatform;
cloudPlatform.connect(frontendDevice);cloudPlatform.connect(frontendDevice);
MediaStream mediaStreamFromCloud = cloudPlatform.interact(inputData);MediaStream mediaStreamFromCloud = cloudPlatform.interact(inputData);
上述代码着重展示了从输入设备获取编解码数据;处理器根据程序指令完成操作和一系列步骤；设备与云平台建立广播连接，最后实现音视频交互的过程。The above code focuses on the process of obtaining codec data from the input device; the processor completes operations and a series of steps according to program instructions; the device establishes a broadcast connection with the cloud platform, and finally realizes audio and video interaction.
实施例九Embodiment 9
在实施例八的基础上，公开一种移动信息防止误操作确认交互系统；该系统还包括：确认交互运行流畅评估模块，用于在监测防止误操作确认交互时段内将所有局部交互监测信息进行整合并对确认交互实施的整体运行状态进行流畅度评估，得到所有确认交互目标对应的流畅评估数据；包括：On the basis of the eighth embodiment, a mobile information misoperation prevention confirmation interaction system is disclosed; the system also includes: a confirmation interaction operation smoothness evaluation module, which is used to integrate all local interaction monitoring information during the monitoring misoperation prevention confirmation interaction period and perform a smoothness evaluation on the overall operation state of the confirmation interaction implementation, and obtain smoothness evaluation data corresponding to all confirmation interaction targets; including:
对确认交互实施的整体运行状态进行流畅度评估时，对防止误操作确认交互时段内所有局部交互监测信息进行遍历，防止误操作确认交互时段的单位为天，具体的可以为30天，同时获取跳转正常标签、跳转轻度异常标签和跳转重度异常标签对应的正常标签权重、轻度异常标签权重和重度异常标签权重并分别标记为CZ、QZ和ZZ；When evaluating the fluency of the overall operation status of the confirmation interaction implementation, all local interaction monitoring information within the period of preventing misoperation confirmation interaction is traversed. The unit of the period of preventing misoperation confirmation interaction is day, which can be 30 days. At the same time, the normal label weight, slight abnormal label weight and severe abnormal label weight corresponding to the jump normal label, jump slight abnormal label and jump severe abnormal label are obtained and marked as CZ, QZ and ZZ respectively;
其中，跳转正常标签、跳转轻度异常标签和跳转重度异常标签对应的正常标签权重、轻度异常标签权重和重度异常标签权重可以根据确认交互运行的历史大数据来获取，也可以根据运营维护人员的经验进行自定义；Among them, the normal label weight, the mild abnormal label weight, and the severe abnormal label weight corresponding to the jump normal label, the jump mild abnormal label, and the jump severe abnormal label can be obtained based on the historical big data of confirming the interactive operation, or can be customized based on the experience of the operation and maintenance personnel;
正常标签权重、轻度异常标签权重和重度异常标签权重用于对不同的标签进行数字化、差异化表示，可以为后续确认交互流畅度的整体分析提供单次确认交互方面的局部数据支持；Normal label weights, mildly abnormal label weights, and severely abnormal label weights are used to digitize and differentiate different labels, and can provide local data support for a single confirmation interaction for the overall analysis of subsequent confirmation interaction fluency;
以及，统计跳转正常标签、跳转轻度异常标签和跳转重度异常标签出现的总数并分别标记为CZ0、QZ0和ZZ0；And, count the total number of jump normal labels, jump mild abnormal labels and jump severe abnormal labels and mark them as CZ0, QZ0 and ZZ0 respectively;
获取确认交互目标对应的确认交互权重并标记为JQ；Get the confirmation interaction weight corresponding to the confirmation interaction target and mark it as JQ;
其中，确认交互权重可以根据确认交互目标对应的盈利总额来获取，也可以根据确认交互目标对应的交易总次数来获取，确认交互权重的数值越大，表示对应确认交互目标的重要性越大，确认交互权重可以对不同的确认交互目标进行数字化、差异化的表示，可以有效提高后续不同确认交互目标流畅度分析以及管控的准确性；Among them, the confirmation interaction weight can be obtained based on the total profit corresponding to the confirmation interaction target, or based on the total number of transactions corresponding to the confirmation interaction target. The larger the value of the confirmation interaction weight, the greater the importance of the corresponding confirmation interaction target. The confirmation interaction weight can digitize and differentiate different confirmation interaction targets, which can effectively improve the accuracy of subsequent fluency analysis and control of different confirmation interaction targets;
提取标记的各项数据的数值并联立计算，通过公式计算获取确认交互目标对应的交互流畅系数；交互流畅系数的计算公式为：The numerical values of the marked data are extracted and calculated in parallel, and the interaction smoothness coefficient corresponding to the confirmed interaction target is obtained through the formula calculation; the calculation formula of the interaction smoothness coefficient is:
式中，为交互流畅系数；j1、j2均为大于零的常量系数且2*j1=j2；j1可以取值为1.306，j2可以取值为2.612；In the formula, is the interaction smoothness coefficient; j1 and j2 are both constant coefficients greater than zero and 2*j1=j2; j1 can be 1.306 and j2 can be 2.612;
需要说明的是，交互流畅系数是用于将不同确认交互的单次跳转分析数据进行整合来对其整体流畅度进行评估的数值；交互流畅系数越小，表示对应确认交互的流畅度越正常、反应越迅速；It should be noted that the interaction smoothness coefficient is a value used to integrate the single jump analysis data of different confirmation interactions to evaluate their overall fluency. The smaller the interaction smoothness coefficient, the more normal the fluency of the corresponding confirmation interaction and the faster the response.
根据交互流畅系数对确认交互目标在监测基础评估时段内的流畅度进行评估时，将交互流畅系数与对应的交互流畅阈值进行比对；交互流畅阈值根据确认交互目标的历史交互大数据来获取；When evaluating the fluency of the confirmed interaction target within the monitoring basic evaluation period according to the interaction fluency coefficient, the interaction fluency coefficient is compared with the corresponding interaction fluency threshold; the interaction fluency threshold is obtained according to the historical interaction big data of the confirmed interaction target;
若交互流畅系数小于交互流畅阈值，则判定确认交互目标的流畅度正常并生成交互流畅正常信号，根据交互流畅正常信号将对应的确认交互目标标记为一类流畅目标；If the interaction fluency coefficient is less than the interaction fluency threshold, the fluency of the confirmed interaction target is determined to be normal and a normal interaction fluency signal is generated. According to the normal interaction fluency signal, the corresponding confirmed interaction target is marked as a type of fluency target;
若交互流畅系数不小于交互流畅阈值且出现的总次数小于N，则判定确认交互目标的流畅度轻度异常并生成交互流畅轻度异常信号，根据交互流畅轻度异常信号将对应的确认交互目标标记为二类流畅目标；N为正整数；If the interaction fluency coefficient is not less than the interaction fluency threshold and the total number of occurrences is less than N, the fluency of the confirmed interaction target is determined to be slightly abnormal and an interaction fluency slightly abnormal signal is generated. According to the interaction fluency slightly abnormal signal, the corresponding confirmed interaction target is marked as a second-class fluency target; N is a positive integer;
若交互流畅系数不小于交互流畅阈值且出现的总次数不小于N，则判定确认交互目标的流畅度重度异常并生成交互流畅重度异常信号，根据交互流畅重度异常信号将对应的确认交互目标标记为三类流畅目标；If the interaction fluency coefficient is not less than the interaction fluency threshold and the total number of occurrences is not less than N, the fluency of the confirmed interaction target is determined to be severely abnormal and an interaction fluency severe abnormality signal is generated. According to the interaction fluency severe abnormality signal, the corresponding confirmed interaction target is marked as three types of fluency targets;
交互流畅系数以及对应的交互流畅正常信号和一类流畅目标、交互流畅轻度异常信号和二类流畅目标或者交互流畅重度异常信号和三类流畅目标构成流畅评估数据并上传至云平台。The interaction fluency coefficient and the corresponding interaction fluency normal signal and first-class fluency target, interaction fluency slightly abnormal signal and second-class fluency target, or interaction fluency severely abnormal signal and third-class fluency target constitute the fluency evaluation data and are uploaded to the cloud platform.
本发明实施例中，通过将不同确认交互的单次跳转分析数据进行整合获取交互流畅系数，根据交互流畅系数可以对对应确认交互目标的整体流畅度进行分析和分类，可以有效提高确认交互运行的监管效果，还可以为后续确认交互运行的质量管理提供运行流畅方面的数据支持。In an embodiment of the present invention, the interaction fluency coefficient is obtained by integrating the single jump analysis data of different confirmation interactions. The overall fluency of the corresponding confirmation interaction target can be analyzed and classified according to the interaction fluency coefficient, which can effectively improve the supervision effect of the confirmation interaction operation and provide data support for the subsequent quality management of the confirmation interaction operation in terms of operation fluency.
实施例十Embodiment 10
在实施例八的基础上，公开一种移动信息防止误操作确认交互系统；该系统还包括：Based on the eighth embodiment, a mobile information misoperation prevention confirmation interactive system is disclosed; the system also includes:
确认交互运行异常监测模块，用于在监测基础评估时段内对确认交互目标被反馈异常和维护发现异常的数据进行监测统计，得到异常监测统计数据；包括：The confirmation interaction operation abnormality monitoring module is used to monitor and count the data of abnormal feedback of the confirmation interaction target and abnormal discovery of maintenance within the monitoring basic evaluation period to obtain abnormal monitoring statistical data; including:
统计确认交互目标对应的每次被反馈异常类型和每次维护发现异常类型；设定不同的异常类型均对应一个不同的异常权重，将获取的每次被反馈异常类型和每次维护发现异常类型分别与数据库中预存储的所有异常类型进行比对获取对应的异常权重；Statistics are collected to confirm the abnormality type reported each time and the abnormality type found each time during maintenance corresponding to the interaction target; different abnormality types are set to correspond to different abnormality weights, and the obtained abnormality type reported each time and the abnormality type found each time during maintenance are compared with all the abnormality types pre-stored in the database to obtain the corresponding abnormality weights;
其中，被反馈指的是用户方提交的异常反馈，维护发现指的是确认交互目标在现有的运维方案中通过运营维护人员主动维护检查发现的异常；异常类型包括但不限于触控不灵敏、触控没反应、显示屏黑屏、网络异常等等；Among them, feedback refers to abnormal feedback submitted by users, and maintenance discovery refers to the abnormalities found by operation and maintenance personnel in the existing operation and maintenance plan. The types of abnormalities include but are not limited to insensitive touch, no touch response, black screen, network abnormality, etc.
异常权重用于对不同的被反馈异常类型和维护发现异常类型进行数字化表示，可以有效提高不同方面异常监测统计以及后续数据分析的整体效果；The anomaly weight is used to digitally represent different types of reported anomalies and maintenance-discovered anomalies, which can effectively improve the overall effect of anomaly monitoring statistics in different aspects and subsequent data analysis;
统计被反馈异常类型对应的被反馈总数和维护发现异常类型对应的维护发现总数；Count the total number of feedbacks corresponding to the reported abnormality type and the total number of maintenance findings corresponding to the maintenance-discovered abnormality type;
每次被反馈异常类型对应的异常权重和被反馈总数、每次维护发现异常类型对应的异常权重和维护发现总数构成异常监测统计数据并上传至云平台。The abnormal weight corresponding to each reported abnormality type and the total number of reported abnormalities, and the abnormal weight corresponding to each maintenance discovered abnormality type and the total number of maintenance discovered abnormalities constitute abnormal monitoring statistical data and are uploaded to the cloud platform.
本发明实施例中，通过从不同的维度对确认交互目标的运行异常进行监测统计，可以为后续确认交互目标的运行稳定质量分析提供不同维度的数据支持，可以有效提高确认交互目标的运行稳定质量分析的准确性。In an embodiment of the present invention, by monitoring and counting the operational anomalies of the confirmed interactive target from different dimensions, data support of different dimensions can be provided for the subsequent operational stability quality analysis of the confirmed interactive target, which can effectively improve the accuracy of the operational stability quality analysis of the confirmed interactive target.
确认交互运行异常评估模块，用于在监测基础评估时段内对异常监测统计数据进行数据处理，并对处理后的数据实施计算和分析判断对应确认交互目标对应的运行稳定状态，得到确认交互目标对应的隐患评估数据；包括：The confirmation interaction operation abnormality assessment module is used to process the abnormal monitoring statistical data within the monitoring basic assessment period, and to calculate and analyze the processed data to determine the operation stability state corresponding to the confirmation interaction target, and obtain the hidden danger assessment data corresponding to the confirmation interaction target; including:
对异常监测统计数据进行数据处理时，获取异常监测统计数据中的每次被反馈异常类型对应的异常权重和被反馈总数以及每次维护发现异常类型对应的异常权重和维护发现总数；When processing the abnormal monitoring statistical data, the abnormal weight and total number of abnormalities reported corresponding to each abnormality type reported in the abnormal monitoring statistical data, as well as the abnormal weight and total number of abnormalities found during maintenance corresponding to each abnormality type found during maintenance are obtained;
将每次被反馈异常类型对应的异常权重标记为BQ，以及将被反馈总数标记为BZ；并将每次维护反向异常类型对应的异常权重标记为WQ，以及将维护发现总数标记为WZ；The anomaly weight corresponding to each feedback anomaly type is marked as BQ, and the total number of feedbacks is marked as BZ; the anomaly weight corresponding to each maintenance reverse anomaly type is marked as WQ, and the total number of maintenance findings is marked as WZ;
提取标记的各项数据的数值并联立计算，通过公式计算获取确认交互目标对应的交互隐患系数；交互隐患系数的计算公式为：The values of the marked data are extracted and calculated in parallel, and the interactive hidden danger coefficient corresponding to the confirmed interactive target is obtained through the formula calculation; the calculation formula of the interactive hidden danger coefficient is:
式中，为交互隐患系数，y1、y2均为大于零的常量系数且y1＜y2，y1可以取值为0.813，y2可以取值为1.625；In the formula, is the interaction hidden danger coefficient, y1 and y2 are both constant coefficients greater than zero and y1＜y2, y1 can be 0.813, and y2 can be 1.625;
需要说明的是，交互隐患系数是用于将确认交互目标不同方面的异常数据进行整合来对其整体运行稳定状态进行评估的数值；交互隐患系数越大，表示对应确认交互目标的运行稳定状态越不佳；It should be noted that the interaction hidden danger coefficient is a value used to integrate abnormal data of different aspects of the confirmed interaction target to evaluate its overall operational stability; the larger the interaction hidden danger coefficient, the worse the operational stability of the corresponding confirmed interaction target;
根据交互隐患系数对对应确认交互目标的运行稳定状态进行评估时，将交互隐患系数与对应的交互隐患范围进行比对；When evaluating the operation stability state of the corresponding confirmed interactive target according to the interactive hidden danger coefficient, the interactive hidden danger coefficient is compared with the corresponding interactive hidden danger range;
若交互隐患系数小于交互隐患范围的最小值，则判定确认交互目标对应的运行稳定状态处于正常状态并生成第一隐患信号，根据第一隐患信号将对应确认交互目标标记为一类隐患目标；If the interaction hidden danger coefficient is less than the minimum value of the interaction hidden danger range, it is determined that the operation stable state corresponding to the confirmed interaction target is in a normal state and a first hidden danger signal is generated, and the corresponding confirmed interaction target is marked as a type-one hidden danger target according to the first hidden danger signal;
若交互隐患系数不小于交互隐患范围的最小值且不大于交互隐患范围的最大值，则判定确认交互目标对应的运行稳定状态处于轻度异常状态并生成第二隐患信号，根据第二隐患信号将对应确认交互目标标记为二类隐患目标；If the interaction hidden danger coefficient is not less than the minimum value of the interaction hidden danger range and not greater than the maximum value of the interaction hidden danger range, it is determined that the operation stability state corresponding to the confirmed interaction target is in a slightly abnormal state and a second hidden danger signal is generated. According to the second hidden danger signal, the corresponding confirmed interaction target is marked as a second-class hidden danger target;
若交互隐患系数大于交互隐患范围的最大值，则判定确认交互目标对应的运行稳定状态处于重度异常状态并生成第三隐患信号，根据第三隐患信号将对应确认交互目标标记为三类隐患目标；If the interaction hidden danger coefficient is greater than the maximum value of the interaction hidden danger range, it is determined that the operation stability state corresponding to the confirmed interaction target is in a severe abnormal state and a third hidden danger signal is generated. According to the third hidden danger signal, the corresponding confirmed interaction target is marked as a third-category hidden danger target;
交互隐患系数以及对应的第一隐患信号和一类隐患目标、第二隐患信号和二类隐患目标或者第三隐患信号和三类隐患目标构成确认交互目标对应的隐患评估数据并上传至云平台。The interactive hidden danger coefficient and the corresponding first hidden danger signal and first category hidden danger target, second hidden danger signal and second category hidden danger target, or third hidden danger signal and third category hidden danger target constitute the hidden danger assessment data corresponding to the confirmed interactive target and are uploaded to the cloud platform.
本发明实施例中，通过将确认交互目标出现异常的被反馈方面和维护主动发现方面进行统计以及实施数据计算和分析分类，既可以直观高效的获取到不同确认交互目标对应的运行稳定状态，又可以为后续确认交互目标异常隐患维护的动态管控提供可靠的数据支持。In the embodiment of the present invention, by statistically analyzing the feedback aspects and maintenance active discovery aspects of abnormalities in confirmed interaction targets and implementing data calculation and analysis classification, it is possible to intuitively and efficiently obtain the operating stability states corresponding to different confirmed interaction targets, and provide reliable data support for the dynamic management and control of subsequent maintenance of abnormal hidden dangers of confirmed interaction targets.
以上所有的实施例仅表达了本发明的相关实际应用的实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。All the above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410571921.7ACN118264649B (en) | 2024-05-10 | 2024-05-10 | A mobile information confirmation interaction method to prevent misoperation |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410571921.7ACN118264649B (en) | 2024-05-10 | 2024-05-10 | A mobile information confirmation interaction method to prevent misoperation |
| Publication Number | Publication Date |
|---|---|
| CN118264649Atrue CN118264649A (en) | 2024-06-28 |
| CN118264649B CN118264649B (en) | 2025-02-07 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410571921.7AActiveCN118264649B (en) | 2024-05-10 | 2024-05-10 | A mobile information confirmation interaction method to prevent misoperation |
| Country | Link |
|---|---|
| CN (1) | CN118264649B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100241845A1 (en)* | 2009-03-18 | 2010-09-23 | Daniel Cuende Alonso | Method and system for the confidential recording, management and distribution of meetings by means of multiple electronic devices with remote storage |
| CN102368819A (en)* | 2011-10-24 | 2012-03-07 | 南京大学 | System for collection, transmission, monitoring and publishment of mobile video |
| CN104065933A (en)* | 2014-06-30 | 2014-09-24 | 苏州科达科技股份有限公司 | Intelligent data acquisition method and system in video surveillance system |
| CN113923391A (en)* | 2021-09-08 | 2022-01-11 | 荣耀终端有限公司 | Method, apparatus, storage medium and program product for video processing |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100241845A1 (en)* | 2009-03-18 | 2010-09-23 | Daniel Cuende Alonso | Method and system for the confidential recording, management and distribution of meetings by means of multiple electronic devices with remote storage |
| CN102368819A (en)* | 2011-10-24 | 2012-03-07 | 南京大学 | System for collection, transmission, monitoring and publishment of mobile video |
| CN104065933A (en)* | 2014-06-30 | 2014-09-24 | 苏州科达科技股份有限公司 | Intelligent data acquisition method and system in video surveillance system |
| CN113923391A (en)* | 2021-09-08 | 2022-01-11 | 荣耀终端有限公司 | Method, apparatus, storage medium and program product for video processing |
| Title |
|---|
| INFOQ腾讯云音视频: "技术解码 | GB28181协议简介及实践", pages 1 - 11, Retrieved from the Internet <URL:https://www.easygbs.con/download.easygbs/GBT28181-2011.pdf>* |
| JS-S: "Camera2+MediaCodec+AudioRecord+MediaMuxer实现录制功能", pages 6 - 39, Retrieved from the Internet <URL:https://blog.csdn.net/weixin_44752167/article/details/>* |
| Publication number | Publication date |
|---|---|
| CN118264649B (en) | 2025-02-07 |
| Publication | Publication Date | Title |
|---|---|---|
| US11128893B2 (en) | Live streaming method and system, server, and storage medium | |
| US11792130B2 (en) | Audio/video communication method, terminal, server, computer device, and storage medium | |
| US20200382741A1 (en) | Method and system for video recording | |
| CN102263959B (en) | Direct broadcast transfer method and system | |
| CN104618690B (en) | A kind of method and system of HD video real-time VOD and history playback | |
| CN115243074B (en) | Video stream processing method and device, storage medium, and electronic device | |
| CN112636880B (en) | Side channel retransmission high-reliability ultra-high-definition video satellite transmission system | |
| CN109327493A (en) | A kind of remote medical monitoring system based on cloud and monitoring method | |
| CN112035081A (en) | Screen projection method and device, computer equipment and storage medium | |
| CN114339146B (en) | Audio and video monitoring method, device, electronic device, and computer-readable storage medium | |
| US10819951B2 (en) | Recording video from a bitstream | |
| CN114470745A (en) | Cloud game implementation method, device and system based on SRT | |
| CN107087138A (en) | A Video Conference System Based on H.265 Video Coding Format | |
| CN103442203A (en) | Network video monitoring system and method based on DM365 | |
| CN116800371A (en) | A data processing method, device, equipment and readable storage medium | |
| CN113038182B (en) | Device and method for realizing video safety transmission through hardware coding and decoding technology | |
| US11089334B1 (en) | Methods and systems for maintaining quality of experience in real-time live video streaming | |
| CN118264649A (en) | A mobile information confirmation interaction method to prevent misoperation | |
| CN110795008B (en) | Picture transmission method and device and computer readable storage medium | |
| CN116248972A (en) | A video streaming method, device and storage medium | |
| CN115942000A (en) | Video stream transcoding method, device, equipment and medium based on H.264 format | |
| CN108924088A (en) | A kind of 4K web camera transmission realizing method | |
| CN115208864A (en) | Data transmission method, device, equipment, vehicle and storage medium | |
| Bailey | Live video streaming from android-enabled devices to web browsers | |
| CN111314659B (en) | Method, device, system, electronic equipment and medium for monitoring code stream push |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |