







技术领域technical field
本申请实施例涉及自动驾驶技术领域,尤其涉及一种自移动设备的通信方法、自移动设备、服务端及巡检系统。The embodiments of the present application relate to the technical field of automatic driving, and in particular, relate to a communication method of an autonomous mobile device, an autonomous mobile device, a server, and an inspection system.
背景技术Background technique
在自动驾驶领域中,除具有在外部环境如道路上自动行驶的车辆、行驶机器人,在天空中飞行的无人机等设备外,大部分时间运行于区域环境或局部环境中(如酒店、餐厅、厂房等)的自动巡检机器人也是其中的一个重要部分。In the field of autonomous driving, in addition to equipment such as vehicles that drive automatically on the road, driving robots, and drones flying in the sky, most of the time they operate in regional environments or local environments (such as hotels, restaurants, etc.) , plant, etc.) is also an important part of the automatic inspection robot.
对于这部分巡检机器人来说,因出于其工作场景的安全性和管理方式的考虑,目前的巡检机器人无法感知管理端的信息,因而无法发起主动呼叫。但是,在实际使用环境中,如防疫酒店巡检场景中,又时常存在着需要巡检机器人主动呼叫管理端的场景,例如,巡检机器人所在工作场景中出现了应急事件处理需求等。For this part of the inspection robot, due to the safety of its work scene and the consideration of the management method, the current inspection robot cannot perceive the information of the management terminal, so it cannot initiate an active call. However, in the actual use environment, such as the inspection scene of the epidemic prevention hotel, there are often scenarios that require the inspection robot to actively call the management terminal.
因此,如何在保障安全性和高效管理性的前提下,还能够使得巡检机器人能够主动呼叫管理端,成为亟待解决的问题。Therefore, how to enable the inspection robot to actively call the management terminal under the premise of ensuring safety and efficient management has become an urgent problem to be solved.
发明内容Contents of the invention
有鉴于此,本申请实施例提供一种自移动设备的通信方案,以至少部分解决上述问题。In view of this, an embodiment of the present application provides a communication solution from a mobile device, so as to at least partly solve the above problems.
根据本申请实施例的第一方面,提供了一种自移动设备的通信方法,包括:自移动设备接收用于呼叫管理端的呼叫指令,其中,所述自移动设备为被限制与管理端进行主动通话连接的设备;根据所述呼叫指令生成呼叫请求,并将所述呼叫请求发送至服务端,以通过所述服务端向所述管理端发送所述呼叫请求;接收所述管理端响应所述呼叫请求发起的通话连接,并基于所述通话连接与所述管理端进行通话。According to the first aspect of the embodiment of the present application, there is provided a communication method of a self-mobile device, including: receiving a call instruction for calling a management terminal from the mobile device, wherein the self-mobile device is restricted from actively communicating with the management terminal A device connected to the call; generate a call request according to the call instruction, and send the call request to the server, so as to send the call request to the management terminal through the server; receive the response from the management terminal calling the call connection initiated by the call request, and talking with the management terminal based on the call connection.
根据本申请实施例的第二方面,提供了另一种自移动设备的通信方法,包括:接收自移动设备发送的呼叫请求,其中,所述自移动设备为被限制与管理端进行主动通话连接的设备;根据所述呼叫请求,确定并呼叫响应所述呼叫请求的管理端,以使所述管理端向发送所述呼叫请求的自移动设备发起通话连接并进行通话。According to the second aspect of the embodiment of the present application, another communication method from a mobile device is provided, including: receiving a call request sent from a mobile device, wherein the mobile device is restricted from actively communicating with a management terminal According to the call request, determine and call the management terminal that responds to the call request, so that the management terminal initiates a call connection to the self-mobile device that sends the call request and conducts a call.
根据本申请实施例的第三方面,提供了一种自移动设备,包括:用于与用户交互的交互系统、和用于对所述自移动设备进行操作控制的控制系统,其中,所述自移动设备为被限制与管理端进行主动通话连接的设备;所述交互系统,用于接收用于呼叫管理端的呼叫指令;根据所述呼叫指令生成呼叫请求,并将所述呼叫请求发送至服务端,以通过所述服务端向所述管理端发送所述呼叫请求;所述控制系统,用于接收所述服务端根据所述呼叫请求返回的所述通话任务信息,并根据所述通话任务信息更新所述呼叫请求所请求的通话对应的通话任务状态,并向所述交互系统推送更新后的所述通话任务状态,以通过所述交互系统向用户展示更新后的所述通话任务状态;所述交互系统,还用于在根据所述通话任务状态确定响应所述呼叫请求的管理端后,接收所述管理端响应所述呼叫请求发起的通话连接,并基于所述通话连接与所述管理端进行通话。According to the third aspect of the embodiments of the present application, there is provided an autonomous mobile device, including: an interaction system for interacting with users, and a control system for operating and controlling the autonomous mobile device, wherein the autonomous The mobile device is a device that is restricted to actively communicate with the management terminal; the interactive system is used to receive a call instruction for calling the management terminal; generate a call request according to the call instruction, and send the call request to the server , so as to send the call request to the management end through the server; the control system is configured to receive the call task information returned by the server according to the call request, and according to the call task information updating the call task state corresponding to the call requested by the call request, and pushing the updated call task state to the interactive system, so as to present the updated call task state to the user through the interactive system; The interactive system is further configured to receive the call connection initiated by the management end in response to the call request after determining the management end responding to the call request according to the state of the call task, and based on the call connection and the management end terminal to make a call.
根据本申请实施例的第四方面,提供了一种服务端,包括:处理器和通信接口;所述处理器,用于接收自移动设备发送的呼叫请求,其中,所述自移动设备为被限制与管理端进行主动通话连接的设备;根据所述呼叫请求,确定响应所述呼叫请求的管理端;所述通信接口,用于对响应所述呼叫请求的管理端进行通话呼叫,以使所述管理端向所述自移动设备发起通话连接并进行通话。According to a fourth aspect of the embodiments of the present application, there is provided a server, including: a processor and a communication interface; the processor is configured to receive a call request sent from a mobile device, wherein the mobile device is Restricting the equipment for active conversation connection with the management terminal; according to the call request, determining the management terminal that responds to the call request; the communication interface is used to make a call to the management terminal that responds to the call request, so that all The management terminal initiates a call connection to the self-mobile device and conducts a call.
根据本申请实施例的第五方面,提供了一种巡检系统,包括:自移动设备、服务端和管理端;其中,所述自移动设备为被限制与管理端进行主动通话连接的设备;所述自移动设备,用于接收用于呼叫管理端的呼叫指令,根据所述呼叫指令生成呼叫请求,并将所述呼叫请求发送至服务端;所述服务端,用于接收所述自移动设备发送的所述呼叫请求,根据所述呼叫请求,确定并呼叫响应所述呼叫请求的管理端;所述管理端,用于根据所述服务端的呼叫,向发送所述呼叫请求的所述自移动设备发起通话连接;所述自移动设备,还用于接收所述管理端发起的通话连接,并基于所述通话连接与所述管理端进行通话。According to the fifth aspect of the embodiments of the present application, a patrol inspection system is provided, including: a self-mobile device, a server and a management terminal; wherein, the self-mobile device is a device that is restricted from actively communicating with the management terminal; The self-mobile device is used to receive a call instruction for calling the management terminal, generate a call request according to the call instruction, and send the call request to a server; the server is used to receive the self-mobile device The call request sent, according to the call request, determine and call the management terminal that responds to the call request; the management terminal is used to send the call request to the self-mobile The device initiates a call connection; the self-mobile device is further configured to receive the call connection initiated by the management terminal, and communicate with the management terminal based on the call connection.
根据本申请实施例提供的方案,在限制了自移动设备向管理端主动发起通话的场景中,借助于服务端,在自移动设备有主动向服务端发起通话的需求时,由自移动设备向服务端来发送呼叫请求,以通过服务端来确定和告知管理端这一主动通话需求,进而,管理端可以连接自移动设备,实现自移动设备和管理端之间的直接通话。上述借助于服务端通知管理端并由管理端发起通话的过程对自移动设备端屏蔽,从自移动设备端来看,其可以无感知地实现与管理端的通话连接,实现了类似主动发起通话的功能。由此,因实际上自移动设备端的主动通话功能仍然受限,因而仍然可以保障巡检系统和管理端的安全性,以及对自移动设备进行高效管理;又因,自移动设备可通过服务端实现与管理端之间的直接通话,从而实现了自移动设备的类似主动通话的功能,满足了自移动设备端的主动通话需求,也提升了自移动设备端的用户的使用体验。According to the solution provided by the embodiment of this application, in the scenario where the mobile device is restricted from actively initiating a call to the management terminal, with the help of the server, when the mobile device needs to initiate a call to the server, the The server sends a call request to determine and inform the management terminal of the active call requirement through the server, and then the management terminal can connect to the mobile device to realize a direct call between the mobile device and the management terminal. The above-mentioned process of notifying the management terminal by the server and initiating a call from the management terminal is shielded from the mobile device side. From the perspective of the mobile device side, it can realize the call connection with the management terminal without perception, and realizes a similar initiative to initiate a call. Function. Therefore, since the active call function of the self-mobile device is still limited, the security of the inspection system and the management terminal can still be guaranteed, and the self-mobile device can be managed efficiently; and because the self-mobile device can be implemented through the server The direct call with the management terminal realizes the function similar to the active call from the mobile device, meets the active call demand from the mobile device end, and improves the user experience of the user from the mobile device end.
附图说明Description of drawings
为了更清楚地说明本申请实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请实施例中记载的一些实施例,对于本领域普通技术人员来讲,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the embodiments of the present application, and those skilled in the art can also obtain other drawings based on these drawings.
图1A为根据本申请实施例一的一种自移动设备的通信方法的步骤流程图;FIG. 1A is a flow chart of steps of a communication method from a mobile device according to Embodiment 1 of the present application;
图1B为一种基于WebRTC通信协议的通话过程示意图;FIG. 1B is a schematic diagram of a call process based on the WebRTC communication protocol;
图2为根据本申请实施例二的一种自移动设备的通信方法的步骤流程图;FIG. 2 is a flowchart of steps of a communication method from a mobile device according to Embodiment 2 of the present application;
图3为根据本申请实施例三的一种自移动设备的通信方法的步骤流程图;FIG. 3 is a flowchart of steps of a communication method from a mobile device according to Embodiment 3 of the present application;
图4为根据本申请实施例四的一种自移动设备的结构框图;FIG. 4 is a structural block diagram of a mobile device according to Embodiment 4 of the present application;
图5为根据本申请实施例五的一种服务端的结构框图;FIG. 5 is a structural block diagram of a server according to Embodiment 5 of the present application;
图6A为根据本申请实施例六的一种巡检系统的结构示意图;FIG. 6A is a schematic structural diagram of an inspection system according to Embodiment 6 of the present application;
图6B为图6A所示巡检系统的通信交互的交互示意图。FIG. 6B is an interactive schematic diagram of communication interaction of the inspection system shown in FIG. 6A .
具体实施方式Detailed ways
为了使本领域的人员更好地理解本申请实施例中的技术方案,下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅是本申请实施例一部分实施例,而不是全部的实施例。基于本申请实施例中的实施例,本领域普通技术人员所获得的所有其他实施例,都应当属于本申请实施例保护的范围。In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the embodiments of the present application shall fall within the protection scope of the embodiments of the present application.
下面结合本申请实施例附图进一步说明本申请实施例具体实现。The specific implementation of the embodiment of the present application will be further described below in conjunction with the accompanying drawings of the embodiment of the present application.
实施例一Embodiment one
参照图1A,示出了根据本申请实施例一的一种自移动设备的通信方法的步骤流程图。Referring to FIG. 1A , it shows a flow chart of steps of a communication method from a mobile device according to Embodiment 1 of the present application.
本实施例从自移动设备的角度,对本申请实施例提供的方案进行说明。该自移动设备通常为在区域内进行巡检的巡检场景下的自移动设备,包括但不限于巡检机器人、服务机器人等。基于安全性和高效管理性的考虑,这些自移动设备不具有与其管理端进行主动通话的功能。This embodiment describes the solution provided by the embodiment of the present application from the perspective of a mobile device. The self-mobile device is usually a self-mobile device in an inspection scene that performs inspections in an area, including but not limited to an inspection robot, a service robot, and the like. Based on security and efficient management considerations, these self-mobile devices do not have the function of actively communicating with their management terminals.
基于上述情形,本实施例提供的自移动设备的通信方法包括以下步骤:Based on the above situation, the communication method from the mobile device provided in this embodiment includes the following steps:
步骤S102:自移动设备接收用于呼叫管理端的呼叫指令。Step S102: Receive a call instruction for calling the management terminal from the mobile device.
如前所述,本申请实施例中,自移动设备为被限制与管理端进行主动通话连接的设备。As mentioned above, in the embodiment of the present application, the self-mobile device is a device that is restricted to actively communicate with the management terminal.
以WebRTC(Web Real-Time Communications,网页实时通信)协议为例,其是一种实时通信技术,允许网络应用或者站点,在不借助中间媒介的情况下,建立浏览器之间点对点的连接,实现视频流、音频流、其他数据中的部分或全部的传输。基于该WebRTC协议的一种示例性通话过程如图1B所示,该通话系统中包括Web服务器、浏览器A和浏览器B。当浏览器A需要与浏览器B进行通话时:首先,浏览器A将自身的会话描述对象(offer,提议)发送至Web服务器;Web服务器将浏览器A的会话描述对象发送至浏览器B;浏览器B向自身的会话描述对象(answer,应答)发送至Web服务器;Web服务器转发该应答至浏览器A;浏览器A和B基于此开始交互,确定访问对方的方式;基于确定的方式,浏览器A和B开始协商通信密钥;进而,浏览器A和B建立通信连接并开始交换如语音数据、视频数据或其它数据,以实现彼此间的通话。反之,若浏览器B需要与浏览器A进行通话时,也同样采用上述方式。浏览器A和B中的任何一端都可以成为通话的主动发起端。Take the WebRTC (Web Real-Time Communications) protocol as an example, which is a real-time communication technology that allows network applications or sites to establish point-to-point connections between browsers without intermediaries, realizing Transmission of some or all of video streams, audio streams, and other data. An exemplary call process based on the WebRTC protocol is shown in FIG. 1B , and the call system includes a Web server, a browser A, and a browser B. When browser A needs to communicate with browser B: first, browser A sends its own session description object (offer, proposal) to the web server; the web server sends browser A's session description object to browser B; Browser B sends its own session description object (answer, response) to the web server; the web server forwards the response to browser A; browsers A and B start interacting based on this, and determine the way to access each other; based on the determined way, The browsers A and B start negotiating the communication key; then, the browsers A and B establish a communication connection and start exchanging voice data, video data or other data, so as to realize the conversation between each other. Conversely, if browser B needs to communicate with browser A, the above method is also adopted. Either of browsers A and B can be the active initiator of the call.
但是,在将上述WebRTC通信方式应用于某些场景如巡检场景中时,作为自移动设备端如巡检机器人端,上述过程将会对其禁用,由此导致自移动设备端的主动通话功能被限制。而对于自移动设备的对端,即管理端,则可正常进行上述过程,实现主动通话功能。从而,可以有效保障此种场景下,对自移动设备的高效管理,以及对管理端和整个巡检系统的安全保障。However, when the above-mentioned WebRTC communication method is applied to certain scenarios such as inspection scenarios, as the self-mobile device side such as the inspection robot side, the above process will disable it, thus causing the active call function of the self-mobile device side to be disabled. limit. However, for the opposite end of the self-mobile device, that is, the management end, the above process can be carried out normally to realize the active call function. Therefore, it can effectively guarantee the efficient management of self-mobile devices in such a scenario, as well as the security of the management terminal and the entire inspection system.
而本申请实施例提供的方案,正是在上述场景的基础上,提供的针对自移动设备的通信方式,以使其可主动发起呼叫,借助于服务端通知管理端,再由管理端呼叫自移动设备。对于自移动设备来说,其通过发起呼叫即可实现与管理端的通话,实现了与上述传统方式类似的主动通话功能。The solution provided by the embodiment of this application is based on the above-mentioned scenario, providing a communication method for the self-mobile device, so that it can initiate a call actively, and notify the management terminal by means of the server, and then the management terminal calls the self-mobile device. Mobile devices. For the self-mobile device, it can realize the communication with the management terminal by initiating a call, and realizes the active communication function similar to the above-mentioned traditional method.
本步骤中,自移动设备接收的用于呼叫管理端的呼叫指令可以为任意适当的可指示自移动设备发起呼叫的指令,包括但不限于:用户通过语音方式发出的呼叫指令,如“XXX(可为唤醒词等),请呼叫管理端”;或者,用户通过自移动设备上的交互设置(如物理交互按钮、或交互界面中的交互按键等)发出的呼叫指令。In this step, the call instruction received from the mobile device for the call management terminal can be any appropriate instruction that can instruct the mobile device to initiate a call, including but not limited to: a call instruction sent by the user through voice, such as "XXX (can be wake-up word, etc.), please call the management terminal”; or, the user sends a call command through the interaction settings on the mobile device (such as physical interaction buttons, or interaction buttons in the interaction interface, etc.).
在一种可行方式中,在自移动设备接收到用于呼叫管理端的呼叫指令之后,自移动设备还可以采集自移动设备所在环境的图像,并上传至服务端,以通过服务端将采集的环境图像发送至管理端,使管理端根据环境图像获取自移动设备的环境信息。通过该环境图像,可以使管理端快速了解自移动设备发起呼叫的原因及其位置等信息,从而为快速响应自移动设备提供依据,并且提高了管理端的响应效率。需要说明的是,该环境图像无需实时上传,且可以与后续的呼叫请求异步上传至服务端,能够在服务端呼叫管理端之前上传至服务端即可。In a feasible manner, after the self-mobile device receives a call instruction for calling the management terminal, the self-mobile device can also collect an image of the environment where the mobile device is located, and upload it to the server, so that the collected environment can be uploaded to the server through the server. The image is sent to the management terminal, so that the management terminal obtains the environmental information from the mobile device according to the environmental image. Through the environment image, the management terminal can quickly understand the reason for initiating a call from the mobile device and information such as its location, thereby providing a basis for quick response to the mobile device, and improving the response efficiency of the management terminal. It should be noted that the environment image does not need to be uploaded in real time, and can be uploaded to the server asynchronously with subsequent call requests, and can be uploaded to the server before the server calls the management terminal.
步骤S104:自移动设备根据呼叫指令生成呼叫请求,并将呼叫请求发送至服务端,以通过服务端向管理端发送呼叫请求。Step S104: Generate a call request from the mobile device according to the call instruction, and send the call request to the server, so as to send the call request to the management end through the server.
自移动设备在接收到呼叫指令后,可据此生成呼叫请求,本申请实施例中,对呼叫请求的具体实现及生成方式均不进行限制。在自移动设备通过WebRTC连接与服务端通信的场景下,该呼叫请求可实现为HTTP请求。After the mobile device receives the call instruction, it can generate a call request accordingly. In the embodiment of the present application, there is no limitation on the specific implementation and generation method of the call request. In the scenario where the mobile device communicates with the server through the WebRTC connection, the call request can be implemented as an HTTP request.
服务端在接收到该呼叫请求后,从向一个或多个管理端发送,以确定出可响应该呼叫请求的管理端。示例性地,若某个自移动设备具有固定的管理端,则该呼叫请求将由服务端发送给该管理端;若某个自移动设备对应有多个管理端,则服务端可从中选择某个管理端响应该呼叫请求,或者,服务端可以将最先反馈的管理端作为响应该呼叫请求的管理端,并将呼叫请求发送至该管理端。After receiving the call request, the server sends it to one or more management terminals to determine the management terminal that can respond to the call request. Exemplarily, if a self-mobile device has a fixed management terminal, the call request will be sent to the management terminal by the server; if a certain self-mobile device corresponds to multiple management terminals, the server can choose a The management terminal responds to the call request, or, the server may use the management terminal that feeds back first as the management terminal that responds to the call request, and send the call request to the management terminal.
需要说明的是,本申请实施例中,若无特殊说明,“多个”、“多种”等与“多”有关的数量均意指两个及两个以上。It should be noted that, in the embodiments of the present application, unless otherwise specified, the quantities related to "multiple" such as "multiple" and "multiple" all mean two or more.
在一种可行方式中,在自移动设备将呼叫请求发送至服务端之后,还会接收服务端根据呼叫请求返回的通话任务信息,其中,该通话任务信息包括与呼叫请求相对应的通话任务标识信息和通话任务状态信息;根据通话任务信息更新呼叫请求所请求的通话对应的通话任务状态。服务端在接收到呼叫请求后,会为其分配一个通话任务标识,以便于对其进行管理和处理状态监查。与此同时,服务端还会根据管理端的响应来生成该通话任务的通话任务状态信息,因自移动设备直接与用户进行交互,因此,该通话任务状态信息还会发送给自移动设备,以更新自移动设备中的信息,使用户能够及时了解通话的处理进展,提升用户的交互感受。In a feasible manner, after the call request is sent from the mobile device to the server, the call task information returned by the server according to the call request will also be received, wherein the call task information includes the call task identifier corresponding to the call request information and call task state information; update the call task state corresponding to the call requested by the call request according to the call task information. After receiving the call request, the server will assign it a call task ID to facilitate its management and processing status monitoring. At the same time, the server will also generate the call task status information of the call task according to the response of the management terminal. Since the self-mobile device directly interacts with the user, the call task status information will also be sent to the self-mobile device to update From the information in the mobile device, the user can keep abreast of the processing progress of the call and improve the user's interactive experience.
又,为了使得自移动设备能够高效、快速地进行信息和交互处理,可以将自移动设备划分为用于与用户交互的交互系统、和用于对自移动设备进行操作控制的控制系统。此种情况下,自移动设备可以通过其交互系统接收用户发起的呼叫指令,并根据呼叫指令生成呼叫请求并发送至服务端;通过其控制系统接收服务端根据呼叫请求返回的通话任务信息,并根据通话任务信息更新通话任务状态并向交互系统推送,以通过交互系统向用户展示更新后的通话任务状态。需要说明的是,在实际应用中,自移动设备中的系统也可以不作此区分,而使用同一系统实现上述交互系统和控制系统的功能,其均在本申请的保护范围内。Also, in order to enable the self-mobile device to efficiently and quickly process information and interactions, the self-mobile device can be divided into an interaction system for interacting with users and a control system for operating and controlling the self-mobile device. In this case, the self-mobile device can receive the call instruction initiated by the user through its interactive system, and generate a call request according to the call instruction and send it to the server; through its control system, it can receive the call task information returned by the server according to the call request, and The call task status is updated according to the call task information and pushed to the interactive system, so as to display the updated call task status to the user through the interactive system. It should be noted that in practical applications, the system in the self-mobile device may not make this distinction, and use the same system to realize the functions of the above-mentioned interaction system and control system, which are all within the protection scope of the present application.
由此,用户在通过自移动设备了解通话的处理状态的同时,后台服务端和管理端对该呼叫请求进行处理,该过程对用户是无感知的,可以进一步提升用户的交互体验。Thus, while the user understands the processing status of the call from the mobile device, the background server and the management end process the call request. This process is insensitive to the user and can further improve the user's interactive experience.
步骤S106:自移动设备接收管理端响应呼叫请求发起的通话连接,并基于通话连接与管理端进行通话。Step S106: Receive the call connection initiated by the management terminal in response to the call request from the mobile device, and communicate with the management terminal based on the call connection.
管理端在接收到服务端发送的呼叫请求后,可对其进行响应,以表明自身可回应该呼叫请求。通过该响应,一方面,服务端可更新通话任务状态并向自移动设备发送;另一方面,管理端将会直接向自移动设备发起通话连接,自移动设备在接收到该通话连接,并成功建立起自移动设备与管理端的通话连接后,即可基于该通话连接与管理端进行通话。After receiving the call request sent by the server, the management terminal can respond to it to indicate that it can respond to the call request. Through this response, on the one hand, the server can update the status of the call task and send it to the self-mobile device; on the other hand, the management end will directly initiate a call connection to the self-mobile device. After the call connection between the mobile device and the management terminal is established, a call can be made with the management terminal based on the call connection.
在采用如图1B所示的WebRTC通信方式的情况下,自移动设备与服务端之间、服务端与管理端之间、以及管理端与自移动设备之间,均通过该方式进行通信。基于此,本步骤中,自移动设备会接收管理端响应呼叫请求而发起的、基于WebRTC协议的通话连接,并基于该通话连接与管理端进行通话。由此,可以实现自移动设备与管理端之间的快速、安全的即时通话。进一步地,也提升了用户与自移动设备之间的交互体验。In the case of adopting the WebRTC communication method as shown in FIG. 1B , communication is performed between the mobile device and the server, between the server and the management terminal, and between the management terminal and the mobile device through this method. Based on this, in this step, the mobile device will receive the call connection based on the WebRTC protocol initiated by the management terminal in response to the call request, and communicate with the management terminal based on the call connection. Thus, a fast and safe instant conversation between the mobile device and the management terminal can be realized. Further, the interactive experience between the user and the mobile device is also improved.
可见,通过本实施例,在限制了自移动设备向管理端主动发起通话的场景中,借助于服务端,在自移动设备有主动向服务端发起通话的需求时,由自移动设备向服务端来发送呼叫请求,以通过服务端来确定和告知管理端这一主动通话需求,进而,管理端可以连接自移动设备,实现自移动设备和管理端之间的直接通话。上述借助于服务端通知管理端并由管理端发起通话的过程对自移动设备端屏蔽,从自移动设备端来看,其可以无感知地实现与管理端的通话连接,实现了类似主动发起通话的功能。由此,因实际上自移动设备端的主动通话功能仍然受限,因而仍然可以保障巡检系统和管理端的安全性,以及对自移动设备进行高效管理;又因,自移动设备可通过服务端实现与管理端之间的直接通话,从而实现了自移动设备的类似主动通话的功能,满足了自移动设备端的主动通话需求,也提升了自移动设备端的用户的使用体验。It can be seen that through this embodiment, in the scenario where the self-mobile device actively initiates a call to the management terminal, with the help of the server, when the self-mobile device has a demand for actively initiating a call to the server, the self-mobile device initiates a call to the server. To send a call request to determine and inform the management terminal of this active call requirement through the server, and then the management terminal can connect to the mobile device to realize a direct call between the mobile device and the management terminal. The above-mentioned process of notifying the management terminal by the server and initiating a call from the management terminal is shielded from the mobile device side. From the perspective of the mobile device side, it can realize the call connection with the management terminal without perception, and realizes a similar initiative to initiate a call. Function. Therefore, since the active call function of the self-mobile device is still limited, the security of the inspection system and the management terminal can still be guaranteed, and the self-mobile device can be managed efficiently; and because the self-mobile device can be implemented through the server The direct call with the management terminal realizes the function similar to the active call from the mobile device, meets the active call demand from the mobile device end, and improves the user experience of the user from the mobile device end.
实施例二Embodiment two
参照图2,示出了根据本申请实施例二的一种自移动设备的通信方法的步骤流程图。Referring to FIG. 2 , it shows a flowchart of steps of a communication method from a mobile device according to Embodiment 2 of the present application.
本实施例从服务端的角度,对本申请实施例提供的方案进行说明。本实施例提供的自移动设备的通信方法包括以下步骤:This embodiment describes the solution provided by the embodiment of the present application from the perspective of the server. The communication method from the mobile device provided in this embodiment includes the following steps:
步骤S202:接收自移动设备发送的呼叫请求。Step S202: Receive a call request sent from the mobile device.
本实施例中,自移动设备仍为被限制与管理端进行主动通话连接的设备。自移动设备在接收到用户发出的呼叫指令后,会生成相应的呼叫请求发送给服务端,服务端可接收到该呼叫请求。该呼叫请求中可携带有自移动设备的标识信息和/或位置信息等,在某些场景下,该呼叫请求中也可能携带有待呼叫的管理端的信息,如管理端的标识信息或位置信息等。In this embodiment, the self-mobile device is still a device that is restricted from actively communicating with the management terminal. After receiving the call instruction from the user, the mobile device will generate a corresponding call request and send it to the server, and the server can receive the call request. The call request may carry identification information and/or location information of the mobile device, and in some scenarios, the call request may also carry information of the management terminal to be called, such as identification information or location information of the management terminal.
步骤S204:根据呼叫请求,确定并呼叫响应呼叫请求的管理端,以使管理端向发送呼叫请求的自移动设备发起通话连接并进行通话。Step S204: According to the call request, determine and call the management terminal that responds to the call request, so that the management terminal initiates a call connection to the self-mobile device that sends the call request and conducts a call.
服务端在接收到呼叫请求后,会对其进行处理,并向管理端发起呼叫,以确定能够响应该呼叫请求的管理端。After receiving the call request, the server will process it and initiate a call to the management terminal to determine the management terminal that can respond to the call request.
在一种可行方式中,自移动设备具有与其对应的管理端,即预设管理端。自移动设备与该预设管理端之间可以是一一对应关系,即一个自移动设备对应有一个固定的管理端;也可以是多对一关系,即多个自移动设备对应一个固定的管理端。此种情况下,服务端可以根据呼叫请求中携带的自移动设备的信息,来确定与该自移动设备对应的预设管理端,并向该管理端发起呼叫。若管理端应答该呼叫,则表明该管理端可与自移动设备通话,后续也将由该管理端向自移动设备发起通话。而若该管理端未应答该呼叫,则服务端可向自移动设备反馈类似“呼叫失败”之类的提示信息,并且,更新相应的通话任务状态。通过这种方式,服务端可以高效、快速地为自移动设备确定可通话的管理端。In a feasible manner, the self-mobile device has a corresponding management terminal, that is, a preset management terminal. There can be a one-to-one relationship between the self-mobile device and the preset management terminal, that is, one self-mobile device corresponds to a fixed management terminal; it can also be a many-to-one relationship, that is, multiple self-mobile devices correspond to a fixed management terminal. end. In this case, the server can determine the preset management terminal corresponding to the mobile device according to the information of the mobile device carried in the call request, and initiate a call to the management terminal. If the management terminal answers the call, it indicates that the management terminal can communicate with the self-mobile device, and the management terminal will initiate a conversation with the self-mobile device subsequently. And if the management terminal does not answer the call, the server can feed back prompt information such as "call failure" to the mobile device, and update the corresponding call task status. In this way, the server can efficiently and quickly determine the management terminal that can be called from the mobile device.
但不限于此,在另一种可行方式中,针对每一个自移动设备,其可能对应有多个管理端,需要多个管理端中挑选出一个与其通话。则,此种情况下,服务端可以根据呼叫请求中携带的自移动设备的信息,确定与自移动设备对应的多个管理端,并向多个管理端同时发起呼叫;根据多个管理端对呼叫的响应,确定响应呼叫请求的管理端。一种方式中,服务端可以将最先应答呼叫的管理端确定为响应呼叫请求的管理端,以简单、快速地实现管理端确定;另一种方式中,服务端可以向多个管理端发送呼叫选择,如呼叫弹窗,由管理端的管理员自行选择是否应答,并将最先选择应答的管理端确定为响应呼叫请求的管理端,以为管理端的管理员提供更多的操作空间。But not limited thereto, in another feasible manner, for each self-mobile device, it may correspond to multiple management terminals, and it is necessary to select one of the multiple management terminals to communicate with it. Then, in this case, the server can determine multiple management terminals corresponding to the mobile device according to the information of the mobile device carried in the call request, and initiate calls to multiple management terminals at the same time; The response of the call, which determines the management end that responds to the call request. In one way, the server can determine the management terminal that answers the call first as the management terminal that responds to the call request, so as to realize the determination of the management terminal simply and quickly; in another way, the server can send multiple management terminals Call selection, such as a call pop-up window, is up to the administrator of the management terminal to choose whether to answer, and the management terminal that chooses to answer first is determined as the management terminal that responds to the call request, so as to provide more operating space for the administrator of the management terminal.
若自移动设备还向服务端上传了其采集的环境图像,则在服务端成功呼叫到响应呼叫请求的管理端后,还会向该管理端发送自移动设备采集的环境图像,以使该管理端能够根据环境图像获取自移动设备的环境信息,从而为管理端后续呼叫自移动设备,以处理自移动设备可能产生的问题提供信息参考和依据。If the mobile device also uploads the collected environmental image to the server, after the server successfully calls to the management terminal that responds to the call request, it will also send the environmental image collected from the mobile device to the management terminal, so that the management The terminal can obtain the environmental information of the mobile device according to the environment image, so as to provide information reference and basis for the management terminal to call the mobile device subsequently to deal with the problems that may arise from the mobile device.
此外,如前所述,服务端还会根据自移动设备发起的呼叫请求,为其设定通话任务,并生成相应的通话任务信息,包括通话任务标识信息和通话任务状态信息,此种情况下,服务端在成功呼叫到响应呼叫请求的管理端后,还会更新呼叫请求对应的通话任务信息中的通话任务状态信息,并向自移动设备发送,以使自移动设备可以基于该更新后的通话任务信息与用户交互,使用户及时了解该次呼叫请求被处理的动态进展。In addition, as mentioned above, the server will also set call tasks for it according to the call request initiated from the mobile device, and generate corresponding call task information, including call task identification information and call task status information. , after the server successfully calls to the management terminal that responds to the call request, it will also update the call task status information in the call task information corresponding to the call request, and send it to the self-mobile device, so that the self-mobile device can The call task information interacts with the user, so that the user can keep abreast of the dynamic progress of the call request being processed.
但在某些情况下,还存在着场景异常的情况,例如,若服务端检测到自移动设备在等待与管理端进行通话的过程中出现异常(如任务异常、电量异常、自移动设备硬件异常,等),则生成与呼叫请求对应的任务异常信息,并将任务异常信息发送至管理端,以通知管理端终止通话任务。由此,可以保障管理端处理呼叫请求的有效性,避免系统资源浪费。However, in some cases, there are still abnormal scenarios. For example, if the server detects that the mobile device is waiting for an abnormality in the process of talking with the management terminal (such as task abnormality, power abnormality, hardware abnormality of the mobile device) , etc.), generate task exception information corresponding to the call request, and send the task exception information to the management terminal to notify the management terminal to terminate the call task. In this way, the effectiveness of the management terminal in processing the call request can be guaranteed, and the waste of system resources can be avoided.
在采用如图1B所示的WebRTC通信方式的情况下,自移动设备与服务端之间、服务端与管理端之间、以及管理端与自移动设备之间,均通过该方式进行通信。基于此,上述过程中,服务端接收自移动设备发送的呼叫请求可以实现为:服务端接收自移动设备通过与服务端之间的WebRTC连接发送的呼叫请求。相应地,服务端确定并呼叫响应呼叫请求的管理端可实现为:服务端确定响应呼叫请求的管理端,并通过服务端与管理端之间的WebRTC连接呼叫管理端。由此,实现各端之间的高效互连和通信。In the case of adopting the WebRTC communication method as shown in FIG. 1B , communication is performed between the mobile device and the server, between the server and the management terminal, and between the management terminal and the mobile device through this method. Based on this, in the above process, the server receives the call request sent from the mobile device may be implemented as: the server receives the call request sent from the mobile device through the WebRTC connection with the server. Correspondingly, the server determines and calls the management terminal that responds to the call request can be implemented as follows: the server determines the management terminal that responds to the call request, and calls the management terminal through the WebRTC connection between the server and the management terminal. Thereby, efficient interconnection and communication between each end are realized.
可见,通过本实施例的服务端,可以在接收到呼叫请求后,通过呼叫方式将该呼叫请求PUSH至管理端,管理端收到该呼叫请求后,主动向自移动设备发起通话。从而,实现了在只有管理端可单向主动连接自移动设备通话的情况下,实现双向通话的能力。It can be seen that, through the server of this embodiment, after receiving the call request, the call request can be PUSHed to the management terminal through a calling method, and the management terminal can actively initiate a call to the mobile device after receiving the call request. Thus, the ability to realize two-way conversation is realized under the condition that only the management terminal can actively connect to the mobile device for one-way conversation.
需要说明的是,本申请实施例中,服务端可实现为服务器集群、或服务器集群中的服务器、或云服务,等等。It should be noted that, in the embodiment of the present application, the server may be implemented as a server cluster, or a server in a server cluster, or a cloud service, or the like.
实施例三Embodiment three
参照图3,示出了根据本申请实施例三的一种自移动设备的通信方法的步骤流程图。Referring to FIG. 3 , it shows a flow chart of steps of a communication method from a mobile device according to Embodiment 3 of the present application.
本实施例从自移动设备、服务端和管理端交互的角度,对本申请实施例的通信方法进行说明。并且,本实施例中,以自移动设备为巡检机器人为示例。但本领域技术人员应当明了,其它类似自移动设备均可适用本实施例的方案。This embodiment describes the communication method in the embodiment of the present application from the perspective of interaction between the mobile device, the server, and the management end. Moreover, in this embodiment, the self-mobile device is used as an example of an inspection robot. However, those skilled in the art should understand that the solution of this embodiment can be applied to other similar mobile devices.
本实施例的自移动设备的通信方法包括以下步骤:The communication method from the mobile device of this embodiment includes the following steps:
步骤S302:巡检机器人接收到其所在环境的用户发出的指令。Step S302: The inspection robot receives an instruction from a user in its environment.
在实际应用中,用户会向巡检机器人发出各种指令,比如“今天天气如何?”等等,但也有可能发出呼叫管理端的指令,例如,“小X,请连接管理端通话”,或者,“小X,我要与管理端通话”等。这些语音指令将会由巡检机器人首先将其转换为文本指令,再进行处理。In practical applications, the user will send various instructions to the inspection robot, such as "how is the weather today?", etc., but it is also possible to send instructions to call the management terminal, for example, "Xiao X, please connect to the management terminal to talk", or, "Little X, I want to talk to the management terminal" and so on. These voice commands will be converted into text commands by the inspection robot first, and then processed.
步骤S304:巡检机器人判断用户发出的指令是否为用于呼叫管理端的呼叫指令;若为呼叫指令,则巡检机器人一方面对用户的呼叫进行回应,另一方面在根据呼叫指令生成呼叫请求并上传服务端的同时,采集现场环境图像或视频,并可与呼叫请求异步上传至服务端。Step S304: The inspection robot judges whether the instruction sent by the user is a call instruction for calling the management terminal; if it is a call instruction, the inspection robot responds to the user's call on the one hand, and generates a call request and While uploading to the server, the image or video of the scene environment is collected, and can be uploaded to the server asynchronously with the call request.
对用户发出指令是否为呼叫管理端的呼叫指令的判断,可由本领域技术人员采用任意适当方式实现,包括但不限于:关键字识别方式、语义识别方式等等。在判断为呼叫指令后,一方面,巡检机器人对用户的呼叫进行回应,该也可以为任意适当的回应,如,“好的,开始呼叫…”,“呼叫中……”等等。另一方面,巡检机器人会根据该呼叫指令生成相应的呼叫请求并上传服务端,在该呼叫请求中可以携带相应的信息,如巡检机器人的标识信息、位置信息等,若巡检机器人与管理端的对应关系是确定的,则还可以在呼叫请求中携带该巡检机器人对应的管理端的信息。再一方面,巡检机器人还会采集其所在现场环境的图像或视频,并将图像或视频异步上传至服务端。The judgment of whether the command issued by the user is a call command from the call management terminal can be realized by those skilled in the art in any appropriate way, including but not limited to: keyword recognition method, semantic recognition method and so on. After judging that it is a call instruction, on the one hand, the inspection robot responds to the user's call, which can also be any appropriate response, such as, "OK, start calling...", "Calling..." and so on. On the other hand, the inspection robot will generate a corresponding call request according to the call instruction and upload it to the server. The call request can carry corresponding information, such as the identification information and location information of the inspection robot. If the corresponding relationship of the management terminal is determined, the information of the management terminal corresponding to the inspection robot can also be carried in the call request. On the other hand, the inspection robot will also collect images or videos of its on-site environment, and upload the images or videos to the server asynchronously.
步骤S306:服务端在接收到呼叫请求后,检测是否可以进行呼叫任务;若否,则上报呼叫失败事件,并转至步骤S314执行;若是,则通过PUSH通道通知管理端。Step S306: After receiving the call request, the server checks whether the call task can be performed; if not, reports the call failure event, and proceeds to step S314; if so, notifies the management end through the PUSH channel.
例如,服务端在接收到呼叫请求后,会检测当前是否有管理端在线,或者,会检测巡检机器人对应的那个或那些管理端是否在线。如果不在线,则不能进行呼叫任务,该次呼叫失败。而若相应的管理端在线,则可通过PUSH方式和相应的通信通道通知管理端。仍以采用WebRTC方式为例,服务端可通过呼叫的方式来通知管理端。For example, after receiving the call request, the server will detect whether there is currently a management terminal online, or will detect whether the management terminal or terminals corresponding to the inspection robot are online. If not online, the call task cannot be performed, and the call fails. And if the corresponding management terminal is online, the management terminal can be notified through the PUSH method and the corresponding communication channel. Still taking the WebRTC method as an example, the server can notify the management terminal by calling.
步骤S308:服务端判断管理端是否接听了服务端发出的、用于通知管理端的呼叫;若未接听,则转至步骤S314执行;若接听,则确定接听的管理端。Step S308: The server judges whether the management terminal has answered the call sent by the server to notify the management terminal; if not answered, proceed to step S314; if answered, determine the management terminal to answer.
步骤S310:管理端发起WebRTC通信,并进行通话。Step S310: the management end initiates WebRTC communication and makes a call.
管理端通过接听服务端的呼叫,告知其将响应巡检机器人发出的呼叫请求,与巡检机器人通话。进而,管理端向巡检机器人重新发起WebRTC通信,以连接巡检机器人,实现与巡检机器人,具体为巡检机器人对应的用户的通话。By answering the call from the server, the management terminal informs it that it will respond to the call request sent by the inspection robot and talk to the inspection robot. Furthermore, the management end re-initiates WebRTC communication to the inspection robot to connect to the inspection robot, and implements a call with the inspection robot, specifically the user corresponding to the inspection robot.
步骤S312:通话结束后,管理端挂断通话。Step S312: After the call ends, the management end hangs up the call.
步骤S314:服务端在检测到通话结束后,记录本次呼叫事件,并结束本次呼叫流程。Step S314: After detecting that the call is over, the server records the call event and ends the call process.
通过本实施例,由服务端端维护巡检机器人执行任务,巡检机器人收到用户的语音发起呼叫之后,向服务端发起呼叫请求。服务端对本次呼叫请求所请求的呼叫可进行任务状态判断及维护,通过PUSH通道,通知管理端。管理端收到呼叫请求之后,会主动向巡检机器人发起WebRTC通信。通过这种利用服务端和管理端双向通信的能力,实现了在巡检机器人通信能力受限的情况下,实现双向通信的能力,达到了巡检机器人可以进行主动呼叫的效果。Through this embodiment, the inspection robot is maintained by the server to perform tasks, and the inspection robot initiates a call request to the server after receiving the user's voice to initiate a call. The server can judge and maintain the task status of the call requested by this call request, and notify the management terminal through the PUSH channel. After receiving the call request, the management terminal will actively initiate WebRTC communication to the inspection robot. By using the two-way communication capability of the server and the management end, the ability of two-way communication is realized when the communication ability of the inspection robot is limited, and the effect that the inspection robot can make active calls is achieved.
实施例四Embodiment Four
参照图4,示出了根据本申请实施例四的一种自移动设备的结构框图。本实施例的自移动设备仍为被限制与管理端进行主动通话连接的设备,其可实现前述多个实施例中自移动设备的功能,该自移动设备包括:用于与用户交互的交互系统402、和用于对自移动设备进行操作控制的控制系统404。需要说明的是,该自移动设备的交互系统402和控制系统404既可实现为纯软件系统,也可实现为具有相应功能的硬件系统,或者结合相应软件实现相应功能的硬件系统。Referring to FIG. 4 , it shows a structural block diagram of an autonomous mobile device according to Embodiment 4 of the present application. The self-mobile device in this embodiment is still a device that is restricted to actively communicate with the management terminal, and it can realize the functions of the self-mobile device in the aforementioned multiple embodiments. The self-mobile device includes: an interactive system for interacting with users 402, and a control system 404 for operating and controlling the mobile device. It should be noted that the interactive system 402 and the control system 404 of the self-mobile device can be implemented as a pure software system, or as a hardware system with corresponding functions, or a hardware system with corresponding functions combined with corresponding software.
其中:in:
交互系统402,用于接收用于呼叫管理端的呼叫指令;根据呼叫指令生成呼叫请求,并将呼叫请求发送至服务端,以通过服务端向所述管理端发送所述呼叫请求。The interaction system 402 is configured to receive a call instruction for calling the management terminal; generate a call request according to the call instruction, and send the call request to the server, so as to send the call request to the management terminal through the server.
控制系统404,用于接收服务端根据呼叫请求返回的通话任务信息,并根据通话任务信息更新呼叫请求所请求的通话对应的通话任务状态,并向交互系统402推送更新后的通话任务状态,以通过交互系统402向用户展示更新后的通话任务状态。The control system 404 is configured to receive the call task information returned by the server according to the call request, and update the call task state corresponding to the call requested by the call request according to the call task information, and push the updated call task state to the interaction system 402, to The updated call task status is displayed to the user through the interactive system 402 .
相应地,交互系统402,还用于在根据通话任务状态确定响应呼叫请求的管理端后,接收管理端响应呼叫请求发起的通话连接,并基于通话连接与所述管理端进行通话。Correspondingly, the interaction system 402 is further configured to receive the call connection initiated by the management end in response to the call request after determining the management end responding to the call request according to the call task status, and to communicate with the management end based on the call connection.
可选地,交互系统402还可以在接收用于呼叫管理端的呼叫指令之后,采集自移动设备所在环境的图像,并上传至服务端,以通过服务端将图像发送至管理端,使管理端根据图像获取自移动设备的环境信息。Optionally, the interactive system 402 can also collect images from the environment where the mobile device is located after receiving the call instruction for calling the management terminal, and upload them to the server, so that the image can be sent to the management terminal through the server, so that the management terminal according to The image is obtained from the environment information of the mobile device.
可选地,通话任务信息包括与呼叫请求相对应的通话任务标识信息和通话任务状态信息。Optionally, the call task information includes call task identification information and call task state information corresponding to the call request.
可选地,交互系统402还向用户展示更新后的通话任务状态。Optionally, the interactive system 402 also displays the updated call task status to the user.
可选地,自移动设备与服务端之间、服务端与管理端之间、以及管理端与自移动设备之间,均基于WebRTC协议进行通信。基于此,交互系统402接收管理端响应呼叫请求发起的、基于WebRTC协议的通话连接,并基于该通话连接与管理端进行通话。Optionally, the communication between the self-mobile device and the server, between the server and the management terminal, and between the management terminal and the self-mobile device is based on the WebRTC protocol. Based on this, the interaction system 402 receives the call connection based on the WebRTC protocol initiated by the management terminal in response to the call request, and communicates with the management terminal based on the call connection.
本实施例的自移动设备用于实现前述多个实施例中自移动设备的功能,并具有相应的有益效果,在此不再详述。本实施例的自移动设备的各个部分执行的功能描述较为简单,其具体实现可参照前述方法实施例中相关部分的描述,在此亦不再赘述。The self-moving device in this embodiment is used to implement the functions of the self-moving device in the foregoing embodiments, and has corresponding beneficial effects, which will not be described in detail here. The description of the functions executed by each part of the self-mobile device in this embodiment is relatively simple, and for its specific implementation, reference may be made to the description of relevant parts in the foregoing method embodiments, which will not be repeated here.
实施例五Embodiment five
参照图5,示出了根据本申请实施例五的一种服务端的结构框图。Referring to FIG. 5 , it shows a structural block diagram of a server according to Embodiment 5 of the present application.
本实施例的服务端包括:处理器502和通信接口504。The server in this embodiment includes: a processor 502 and a communication interface 504 .
其中:in:
处理器502,用于接收自移动设备发送的呼叫请求,该自移动设备为被限制与管理端进行主动通话连接的设备,示例性地,该自移动设备为实施例四中所述的自移动设备。处理器502根据呼叫请求,确定响应呼叫请求的管理端。进而,通过通信接口504,对响应呼叫请求的管理端进行通话呼叫,以使管理端向自移动设备发起通话连接并进行通话。The processor 502 is configured to receive a call request sent from a mobile device, the mobile device is a device that is restricted from active call connection with the management terminal, for example, the mobile device is the mobile device described in Embodiment 4 equipment. The processor 502 determines the management terminal that responds to the call request according to the call request. Furthermore, through the communication interface 504, a call is made to the management terminal that responds to the call request, so that the management terminal initiates a call connection to the self-mobile device and conducts a call.
可选地,处理器502根据呼叫请求中携带的自移动设备的信息,确定与自移动设备对应的预设管理端,并向管理端发起呼叫;或者,根据呼叫请求中携带的自移动设备的信息,确定与自移动设备对应的多个管理端,并向多个管理端同时发起呼叫;根据多个管理端对所述呼叫的响应,确定响应呼叫请求的管理端。Optionally, the processor 502 determines the preset management terminal corresponding to the mobile device according to the information of the mobile device carried in the call request, and initiates a call to the management terminal; or, according to the information of the mobile device carried in the call request Information, determine multiple management terminals corresponding to the self-mobile device, and initiate calls to multiple management terminals at the same time; determine the management terminal that responds to the call request according to the responses of the multiple management terminals to the call.
可选地,处理器502还用于在成功呼叫响应呼叫请求的管理端后,通过通信接口504向管理端发送自移动设备采集的环境图像,以使管理端根据环境图像获取自移动设备的环境信息。Optionally, the processor 502 is further configured to send the environmental image collected from the mobile device to the management terminal through the communication interface 504 after successfully calling the management terminal that responds to the call request, so that the management terminal obtains the environmental image from the mobile device according to the environmental image. information.
可选地,处理器502还用于在成功呼叫响应呼叫请求的管理端后,更新呼叫请求对应的通话任务信息中的通话任务状态信息,并通过通信接口504向自移动设备发送。Optionally, the processor 502 is further configured to update the call task status information in the call task information corresponding to the call request after successfully calling the management terminal that responds to the call request, and send it to the self-mobile device through the communication interface 504 .
可选地,处理器502接收自移动设备通过与服务端之间的WebRTC连接发送的呼叫请求;以及,确定响应呼叫请求的管理端,并通过服务端与管理端之间的WebRTC连接呼叫管理端。Optionally, the processor 502 receives a call request sent from the mobile device through a WebRTC connection with the server; and, determines the management terminal that responds to the call request, and calls the management terminal through the WebRTC connection between the server and the management terminal .
可选地,处理器502还用于若检测到自移动设备在等待与管理端进行通话的过程中出现异常,则生成与呼叫请求对应的任务异常信息,并将任务异常信息发送至管理端,以通知管理端终止通话任务。Optionally, the processor 502 is also configured to generate task exception information corresponding to the call request if it detects that the mobile device is abnormal while waiting for a call with the management terminal, and send the task exception information to the management terminal, To notify the management terminal to terminate the call task.
本实施例的服务端用于实现前述多个实施例中服务端的功能,并具有相应的有益效果,在此不再详述。本实施例的服务端的各个部分执行的功能描述较为简单,其具体实现可参照前述方法实施例中相关部分的描述,在此亦不再赘述。The server in this embodiment is used to implement the functions of the server in the foregoing embodiments, and has corresponding beneficial effects, which will not be described in detail here. The description of the functions executed by each part of the server in this embodiment is relatively simple, and its specific implementation can refer to the description of the relevant parts in the foregoing method embodiments, which will not be repeated here.
实施例六Embodiment six
参照图6A,示出了根据本申请实施例六的一种巡检系统的结构示意图。Referring to FIG. 6A , it shows a schematic structural diagram of an inspection system according to Embodiment 6 of the present application.
本实施例中的巡检系统包括自移动设备602、服务端604和管理端606。The inspection system in this embodiment includes a self-mobile device 602 , a server 604 and a management terminal 606 .
示例性地,自移动设备602可为如实施例四中所示的自移动设备,即,被限制与管理端进行主动通话连接的设备,在一种实现方式中,可实现为巡检机器人;服务端604可为如实施例五中所示的服务端;管理端606可实现为前述多个实施例中涉及的管理端。Exemplarily, the self-mobile device 602 can be the self-mobile device as shown in Embodiment 4, that is, a device that is restricted to actively communicate with the management terminal, and in one implementation, it can be implemented as an inspection robot; The server 604 can be the server shown in the fifth embodiment; the management terminal 606 can be implemented as the management terminal involved in the above-mentioned multiple embodiments.
在该巡检系统中,自移动设备602可用于接收用于呼叫管理端的呼叫指令,根据呼叫指令生成呼叫请求,并将呼叫请求发送至服务端604;服务端604,用于接收自移动设备602发送的呼叫请求,根据呼叫请求,确定并呼叫响应呼叫请求的管理端606;管理端606,用于根据服务端的呼叫,向发送呼叫请求的自移动设备602发起通话连接;自移动设备602,还用于接收管理端606发起的通话连接,并基于通话连接与管理端606进行通话。In this inspection system, the self-mobile device 602 can be used to receive a call instruction for the call management terminal, generate a call request according to the call instruction, and send the call request to the server 604; According to the call request sent, determine and call the management terminal 606 that responds to the call request; the management terminal 606 is used to initiate a call connection to the self-mobile device 602 that sends the call request according to the call of the server; the self-mobile device 602 also It is used to receive the call connection initiated by the management terminal 606, and communicate with the management terminal 606 based on the call connection.
上述巡检系统的一个具体通信交互示例如图6B中所示,该交互过程包括:A specific communication interaction example of the above inspection system is shown in Figure 6B, the interaction process includes:
1.自移动设备602所在环境中的用户向自移动设备602,具体可为自移动设备602的交互系统6024发出用于呼叫管理端606的语音呼叫指令。1. The user in the environment where the mobile device 602 is located sends a voice call instruction for calling the management terminal 606 to the mobile device 602 , specifically the interaction system 6024 of the mobile device 602 .
2.交互系统6024根据该呼叫指令采集自移动设备602所在环境的图像,并传递给自移动设备602的控制系统6024。2. The interactive system 6024 collects the image of the environment where the mobile device 602 is located according to the call instruction, and transmits it to the control system 6024 of the mobile device 602 .
3.控制系统6024向交互系统6022返回图像的存储URL,并将在后续与呼叫请求异步上传服务端604。3. The control system 6024 returns the storage URL of the image to the interactive system 6022, and uploads it to the server 604 asynchronously with the call request.
4.交互系统6022根据呼叫指令生成HTTP呼叫请求,并上传服务端604。4. The interactive system 6022 generates an HTTP call request according to the call instruction, and uploads it to the server 604 .
5.服务端604根据该呼叫请求为本次呼叫任务分配任务id,并将任务id的信息返回控制系统6024。5. The server 604 assigns a task id to the call task according to the call request, and returns the task id information to the control system 6024 .
6.服务端604会根据该呼叫请求检测是否有管理端在线,若没有,则向自移动设备602,如自移动设备602的交互系统6022返回错误信息,相应地,该错误信息中还携带有任务id的信息。若有管理端在线,交互系统6022向控制系统6024订阅任务状态,即通话任务状态,该任务状态会由服务端604负责更新并推送至控制系统6024。6. The server 604 will detect whether there is a management terminal online according to the call request. If not, it will return an error message to the mobile device 602, such as the interactive system 6022 of the mobile device 602. Correspondingly, the error message also carries Information about the task id. If the management terminal is online, the interactive system 6022 subscribes to the control system 6024 for the task status, that is, the call task status, and the task status will be updated by the server 604 and pushed to the control system 6024.
7.在可能的情形(一)中,服务端604在分配任务id后,会向管理端606单机呼叫,其中携带任务id信息。需要说明的是,与服务端604连接的管理端606均为通过身份认证,并存在基于WebRTC协议进行PUSH通道互连的管理端。若服务端604呼叫的管理端606接听了该呼叫,则服务端604会更新任务状态,并推送给控制系统6024,并由控制系统6024推送给交互系统6022,以指示通话任务被接起。与此同时,管理端606会向交互系统6022发起WebRTC通话,交互系统6022接收该通话后,停止呼叫进程,并实时采集自移动设备602所在环境的音视频推送给管理端606,管理端606接收该音视频,并基于该通话连接进行用户与管理员间的通话对讲,直至通话结束。服务端604在检测到通话结束后,会向控制系统6024推送通话任务完成的状态通知,进而控制系统6024会再将该状态传递给交互系统6022,以明确告知用户本次通话结束。7. In possible situation (1), after assigning the task id, the server 604 will make a stand-alone call to the management terminal 606, which carries the task id information. It should be noted that the management terminal 606 connected to the server terminal 604 has passed the identity authentication, and there is a management terminal for PUSH channel interconnection based on the WebRTC protocol. If the management terminal 606 called by the server terminal 604 answers the call, the server terminal 604 will update the task status and push it to the control system 6024, and the control system 6024 will push it to the interaction system 6022 to indicate that the call task is picked up. At the same time, the management terminal 606 will initiate a WebRTC call to the interactive system 6022. After the interactive system 6022 receives the call, it stops the call process, and real-time audio and video collected from the environment where the mobile device 602 is located is pushed to the management terminal 606, and the management terminal 606 receives the call. The audio and video, and based on the call connection, the call intercom between the user and the administrator will be carried out until the call ends. After the server 604 detects that the call is over, it will push a status notification of the completion of the call task to the control system 6024, and then the control system 6024 will pass the status to the interaction system 6022 to clearly inform the user that the call is over.
在可能的情形(二)中,服务端604在分配任务id后,向管理端606单机呼叫,其中携带任务id信息。若服务端604呼叫的管理端606拒绝接听该呼叫,则服务端604会尝试呼叫其他管理端,若呼叫的所有管理端都拒绝接听,则服务端604会生成相应的通话状态,如任务被忽略abort等,并发送给控制系统6024。结束本次呼叫流程。In possible situation (2), after assigning the task id, the server 604 makes a stand-alone call to the management terminal 606, which carries the task id information. If the management terminal 606 called by the server 604 refuses to answer the call, the server 604 will try to call other management terminals. If all the management terminals of the call refuse to answer, the server 604 will generate a corresponding call state, such as the task is ignored abort etc., and send to the control system 6024. End this call flow.
在可能的情形(三)中,服务端604在分配任务id后,同时向多个管理端606发起呼叫,其中携带任务id信息。该多个管理端606中任意一个管理端接起,将服务端604都会更新任务状态,并推送给控制系统6024,并由控制系统6024推送给交互系统6022,以指示通话任务被接起。与此同时,管理端606会向交互系统6022发起WebRTC通话,交互系统6022接收该通话后,停止呼叫进程,并实时采集自移动设备602所在环境的音视频推送给管理端606,管理端606接收该音视频,并基于该通话连接进行用户与管理员间的通话对讲,直至通话结束。服务端604在检测到通话结束后,会向控制系统6024推送通话任务完成的状态通知,进而控制系统6024会再将该状态传递给交互系统6022,以明确告知用户本次通话结束。In possible situation (3), after assigning the task id, the server 604 simultaneously initiates calls to multiple management terminals 606, which carry task id information. When any one of the multiple management terminals 606 is connected, the server 604 will update the task status and push it to the control system 6024, and the control system 6024 will push it to the interaction system 6022 to indicate that the call task is picked up. At the same time, the management terminal 606 will initiate a WebRTC call to the interactive system 6022. After the interactive system 6022 receives the call, it stops the call process, and real-time audio and video collected from the environment where the mobile device 602 is located is pushed to the management terminal 606, and the management terminal 606 receives the call. The audio and video, and based on the call connection, the call intercom between the user and the administrator will be carried out until the call ends. After the server 604 detects that the call is over, it will push a status notification of the completion of the call task to the control system 6024, and then the control system 6024 will pass the status to the interaction system 6022 to clearly inform the user that the call is over.
此外,在呼叫及通话过程中,上述任何一方出现异常,都会导致任务结束。例如,若自移动设备端异常(包括但不限于任务异常、电量异常、硬件异常等),这些异常信息会由控制系统6024上报给服务端604,则服务端604会通知管理端任务异常退出。再例如,若自移动设备端采集或处理音视频数据异常,则交互系统6022会通过服务端604通知管理端604任务异常退出。又例如,若管理端606异常或服务端604异常,产生诸如心跳停止或失败的信息,则服务端604会根据该信息向控制系统6024推送结束任务的信息,再由控制系统6024将该结束任务的信息通知给交互系统6022,最终通知到用户。In addition, during the call and conversation, any abnormality of any of the above parties will lead to the end of the task. For example, if there is an abnormality from the mobile device (including but not limited to task abnormality, power abnormality, hardware abnormality, etc.), the abnormal information will be reported to the server 604 by the control system 6024, and the server 604 will notify the management terminal to exit the task abnormally. For another example, if the collection or processing of audio and video data from the mobile device is abnormal, the interactive system 6022 will notify the management terminal 604 through the server 604 to exit the task abnormally. For another example, if the management terminal 606 is abnormal or the server terminal 604 is abnormal and generates information such as heartbeat stop or failure, the server terminal 604 will push the information of ending the task to the control system 6024 according to the information, and then the control system 6024 will end the task. The information is notified to the interactive system 6022, and finally notified to the user.
通过本实施例的巡检系统,由服务端端维护用于巡检的自移动设备执行任务,自移动设备收到用户的语音发起呼叫之后,向服务端发起呼叫请求。服务端对本次呼叫请求所请求的呼叫可进行任务状态判断及维护,通过PUSH通道,通知管理端。管理端收到呼叫请求之后,会主动向自移动设备发起WebRTC通信。通过这种利用服务端和管理端双向通信的能力,实现了在自移动设备通信能力受限的情况下,实现双向通信的能力,达到了自移动设备可以进行主动呼叫的效果。Through the inspection system of this embodiment, the self-mobile device maintained by the server for inspection performs tasks, and after the mobile device receives the user's voice to initiate a call, it initiates a call request to the server. The server can judge and maintain the task status of the call requested by this call request, and notify the management terminal through the PUSH channel. After receiving the call request, the management terminal will actively initiate WebRTC communication to the mobile device. By utilizing the two-way communication capability of the server and the management terminal, the two-way communication capability is realized when the communication capability of the mobile device is limited, and the effect that the mobile device can make an active call is achieved.
需要指出,根据实施的需要,可将本申请实施例中描述的各个部件/步骤拆分为更多部件/步骤,也可将两个或多个部件/步骤或者部件/步骤的部分操作组合成新的部件/步骤,以实现本申请实施例的目的。It should be pointed out that, according to the needs of implementation, each component/step described in the embodiment of the present application can be divided into more components/steps, and two or more components/steps or partial operations of components/steps can also be combined into New components/steps to achieve the purpose of the embodiment of the present application.
上述根据本申请实施例的方法可在硬件、固件中实现,或者被实现为可存储在记录介质(诸如CD ROM、RAM、软盘、硬盘或磁光盘)中的软件或计算机代码,或者被实现通过网络下载的原始存储在远程记录介质或非暂时机器可读介质中并将被存储在本地记录介质中的计算机代码,从而在此描述的方法可被存储在使用通用计算机、专用处理器或者可编程或专用硬件(诸如ASIC或FPGA)的记录介质上的这样的软件处理。可以理解,计算机、处理器、微处理器控制器或可编程硬件包括可存储或接收软件或计算机代码的存储组件(例如,RAM、ROM、闪存等),当所述软件或计算机代码被计算机、处理器或硬件访问且执行时,实现在此描述的方法。此外,当通用计算机访问用于实现在此示出的方法的代码时,代码的执行将通用计算机转换为用于执行在此示出的方法的专用计算机。The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by Computer code downloaded from a network that is originally stored on a remote recording medium or a non-transitory machine-readable medium and will be stored on a local recording medium so that the methods described herein can be stored on a computer code using a general-purpose computer, a dedicated processor, or a programmable Such software processing on a recording medium of dedicated hardware such as ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when When accessed and executed by a processor or hardware, implements the methods described herein. Furthermore, when a general purpose computer accesses code for implementing the methods shown herein, execution of the code transforms the general purpose computer into a special purpose computer for performing the methods shown herein.
本领域普通技术人员可以意识到,结合本文中所公开的实施例描述的各示例的单元及方法步骤,能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请实施例的范围。Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the embodiments of the present application.
以上实施方式仅用于说明本申请实施例,而并非对本申请实施例的限制,有关技术领域的普通技术人员,在不脱离本申请实施例的精神和范围的情况下,还可以做出各种变化和变型,因此所有等同的技术方案也属于本申请实施例的范畴,本申请实施例的专利保护范围应由权利要求限定。The above implementations are only used to illustrate the embodiments of the application, rather than to limit the embodiments of the application. Those of ordinary skill in the relevant technical fields can also make various implementations without departing from the spirit and scope of the embodiments of the application Changes and modifications, so all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211405849.8ACN115914191A (en) | 2022-11-10 | 2022-11-10 | Communication method, self-moving equipment, server and inspection system |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211405849.8ACN115914191A (en) | 2022-11-10 | 2022-11-10 | Communication method, self-moving equipment, server and inspection system |
| Publication Number | Publication Date |
|---|---|
| CN115914191Atrue CN115914191A (en) | 2023-04-04 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211405849.8APendingCN115914191A (en) | 2022-11-10 | 2022-11-10 | Communication method, self-moving equipment, server and inspection system |
| Country | Link |
|---|---|
| CN (1) | CN115914191A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118337961A (en)* | 2024-05-22 | 2024-07-12 | 深圳市友联天美科技有限公司 | Visual intercom method, system and access control machine |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1863157A (en)* | 2005-10-28 | 2006-11-15 | 华为技术有限公司 | Method and apparatus for implementing network communication through NAT |
| CN101170687A (en)* | 2007-11-28 | 2008-04-30 | 武汉烽火网络有限责任公司 | Method for front recording order to penetrate NAT based on video monitoring |
| US20080201480A1 (en)* | 2005-10-28 | 2008-08-21 | Huawei Technologies Co., Ltd. | Method for establishing peer-to-peer connection, method, apparatus and system for traversing nat to realize network communication |
| CN102347990A (en)* | 2011-11-17 | 2012-02-08 | 杭州华三通信技术有限公司 | Method and equipment for establishing TCP (Transmission Control Protocol) connection |
| CN104580630A (en)* | 2015-01-06 | 2015-04-29 | 陈包容 | Callback type call communication method and system |
| US20150140981A1 (en)* | 2013-11-19 | 2015-05-21 | Avaya Inc. | Method and system to manage mobile data network usage for voip calls |
| CN106027816A (en)* | 2016-05-16 | 2016-10-12 | 腾讯科技(深圳)有限公司 | Call method and device, as well as call data processing method and device |
| CN109104536A (en)* | 2017-06-20 | 2018-12-28 | 重庆无线绿洲通信技术有限公司 | A kind of car-mounted terminal and alarm management server |
| CN109447539A (en)* | 2018-09-17 | 2019-03-08 | 北京云迹科技有限公司 | A kind of notification method and distributed robot |
| US20190306203A1 (en)* | 2018-04-03 | 2019-10-03 | Telia Company Ab | Solution for establishing a communication session |
| CN110868500A (en)* | 2019-10-12 | 2020-03-06 | 中国平安财产保险股份有限公司 | Call communication method, device, computer equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1863157A (en)* | 2005-10-28 | 2006-11-15 | 华为技术有限公司 | Method and apparatus for implementing network communication through NAT |
| US20080201480A1 (en)* | 2005-10-28 | 2008-08-21 | Huawei Technologies Co., Ltd. | Method for establishing peer-to-peer connection, method, apparatus and system for traversing nat to realize network communication |
| CN101170687A (en)* | 2007-11-28 | 2008-04-30 | 武汉烽火网络有限责任公司 | Method for front recording order to penetrate NAT based on video monitoring |
| CN102347990A (en)* | 2011-11-17 | 2012-02-08 | 杭州华三通信技术有限公司 | Method and equipment for establishing TCP (Transmission Control Protocol) connection |
| US20150140981A1 (en)* | 2013-11-19 | 2015-05-21 | Avaya Inc. | Method and system to manage mobile data network usage for voip calls |
| CN104580630A (en)* | 2015-01-06 | 2015-04-29 | 陈包容 | Callback type call communication method and system |
| CN106027816A (en)* | 2016-05-16 | 2016-10-12 | 腾讯科技(深圳)有限公司 | Call method and device, as well as call data processing method and device |
| CN109104536A (en)* | 2017-06-20 | 2018-12-28 | 重庆无线绿洲通信技术有限公司 | A kind of car-mounted terminal and alarm management server |
| US20190306203A1 (en)* | 2018-04-03 | 2019-10-03 | Telia Company Ab | Solution for establishing a communication session |
| CN109447539A (en)* | 2018-09-17 | 2019-03-08 | 北京云迹科技有限公司 | A kind of notification method and distributed robot |
| CN110868500A (en)* | 2019-10-12 | 2020-03-06 | 中国平安财产保险股份有限公司 | Call communication method, device, computer equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118337961A (en)* | 2024-05-22 | 2024-07-12 | 深圳市友联天美科技有限公司 | Visual intercom method, system and access control machine |
| Publication | Publication Date | Title |
|---|---|---|
| EP3926917B1 (en) | Data transmission method, device and computer storage medium | |
| CN108881779A (en) | Method, system and server for answering and transferring video calls between smart devices | |
| US8345556B2 (en) | Congestion control system, service edge node, guidance server, congestion control method, program therefor, and recording medium recorded therewith | |
| WO2011127706A1 (en) | Device, system and method for realizing the intelligent home application | |
| WO2013185655A1 (en) | Method and device for distributing mobile attendant call | |
| US20170237938A1 (en) | Information terminal, recording medium, communication control method, and communication system | |
| CN110365931B (en) | Control method and device for multi-party call, electronic device, and storage medium | |
| BRPI0806105B1 (en) | method and system for network multimedia conference access | |
| WO2018133767A1 (en) | Call control method, terminal apparatus, and data storage medium | |
| CN115914191A (en) | Communication method, self-moving equipment, server and inspection system | |
| CN113765745A (en) | Service performance testing method and device, terminal equipment and storage medium | |
| CN113037807B (en) | Remote control method, device, server and media | |
| CN112217850B (en) | Remote control processing method, device, equipment, server and storage medium | |
| CN116915935A (en) | Intercommunication processing method, intercommunication processing system, server and storage medium | |
| CN115134422A (en) | Service processing method, video network terminal, master control device, electronic device and medium | |
| CN107404591B (en) | Smart doorbell monitoring method and device | |
| KR100587945B1 (en) | Method and system of providing call transfer service | |
| CN120528901A (en) | Call method, device, electronic device and computer-readable storage medium | |
| JP7260569B2 (en) | MEDIA DOWNLINK TRANSMISSION CONTROL METHOD AND RELATED DEVICE | |
| KR102479463B1 (en) | Method for providing messaging services and application for the same | |
| HK40037802A (en) | Remote control processing method and apparatus, device, server, and storage medium | |
| HK40037802B (en) | Remote control processing method and apparatus, device, server, and storage medium | |
| JP2007067535A (en) | Participation control method in electronic conference | |
| CN118158277A (en) | Remote help seeking method, device, equipment, vehicle and storage medium | |
| CN110324557B (en) | Method and device for controlling terminal to carry out visual communication |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |