技术领域technical field
本发明涉及通信领域，尤其涉及一种基于Web RTC多方通话建立的方法、设备和系统。The present invention relates to the field of communications, in particular to a method, device and system for establishing a multi-party call based on Web RTC.
背景技术Background technique
当前由双方通话变更为多方通话的方法主要通过终端混音模式和会场混音模式实现，其中前者主要应用与参与人数较少的例如双方通话变更为三方通话的场景，而后者则应用于较多人数参与的场景。The current method of changing from two-party calls to multi-party calls is mainly realized through the terminal audio mixing mode and the venue audio mixing mode. The former is mainly used in scenarios with a small number of participants, such as changing a two-party call to a three-party call, while the latter is used more. The scene of the participation of the number of people.
终端混音模式的实现是在业务方Ａ与用户Ｂ处于通话保持状态，同时与用户Ｃ正在通话时，通过用户A的终端设备分别将A与B、A与C的媒体流进行混音，接着将混音后的媒体流再分别发送至用户B和用户C，使得用户B和用户C能够接收到A与C、A与B的图像和声音，从而间接实现三方通话的效果。The realization of the terminal audio mixing mode is to mix the media streams of A and B, A and C respectively through the terminal equipment of user A when the business party A and user B are in the call hold state and are talking with user C at the same time, and then The mixed media stream is sent to user B and user C respectively, so that user B and user C can receive the image and sound of A and C, A and B, thus indirectly realizing the effect of three-way calling.
会场混音模式的实现是有业务方Ａ与用户Ｂ处于通话保持状态，同时与用户Ｃ正在通话时，业务方A首先通过会场服务器（MediaResource Server，MRS）申请多方会议的会场资源，接着将A与B、A与C的通话分别通过会话重协商，分别转移到与会场服务器建立的媒体通道中，最终通过与会场服务器的连接，实现多方通话的效果。The realization of the venue mixing mode is that when business party A and user B are on hold and are talking with user C at the same time, business party A first applies for a multi-party conference site resource through the site server (MediaResource Server, MRS), and then sends A The calls with B, A, and C are respectively transferred to the media channels established with the site server through session renegotiation, and finally realize the effect of multi-party calls through the connection with the site server.
虽然上述的终端混音模式和会场混音模式均能实现有双方通话变更为多方通话的要求，但是前者需要能够进行本地混音的终端设备的支持，并且如果进行的多方通话中包含视频时，会对混音设备的性能有很高的要求；后者的实现更是需要MRS的支持，否则无法实现，进一步的，通过会场混音模式实现多方通话每一次都需要申请会场资源，步骤繁琐，当由多方通话恢复成双方通话时，申请的会场资源也不能及时释放，导致资源浪费。Although both the terminal audio mixing mode and the venue audio mixing mode mentioned above can meet the requirement of changing a two-party call to a multi-party call, the former requires the support of a terminal device capable of local audio mixing, and if the multi-party call includes video, There will be high requirements on the performance of the audio mixing device; the implementation of the latter requires the support of MRS, otherwise it cannot be realized. Further, each time a multi-party call is implemented through the venue mixing mode, it is necessary to apply for venue resources, and the steps are cumbersome. When the multi-party call is restored to a two-party call, the requested site resources cannot be released in time, resulting in waste of resources.
发明内容Contents of the invention
本发明的实施例提供一种基于Web RTC多方通话建立的方法、设备和系统，能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。Embodiments of the present invention provide a method, device and system for establishing a multi-party call based on Web RTC, which can reduce the requirement for high performance of the device due to local audio mixing, and also eliminate the need to contact the site server for cumbersome site resource applications , to further save the waste of site resources caused when the multi-party call is restored to a two-party call, and finally reduce the establishment steps of the multi-party call and improve the use efficiency of communication resources.
为达到上述目的，本发明的实施例采用如下技术方案：In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:
第一方面，提供一种基于Web RTC多方通话建立的方法，所述方法包括：In the first aspect, a method for establishing a multi-party call based on Web RTC is provided, the method comprising:
接收正在通话的第一用户发送的多方通话建立请求，所述请求包括第一扩展消息，所述第一扩展消息中有待与所述第一用户建立多方通话的第二用户的信息；receiving a multi-party call establishment request sent by the first user who is talking, the request includes a first extended message, and the first extended message includes information about a second user who is to establish a multi-party call with the first user;
判断所述第一用户建立所述多方通话的权限；judging the authority of the first user to establish the multi-party call;
当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，所述第三扩展消息中包括参加所述多方通话成员的列表信息；When the first user has the authority to establish the multiparty call, send a second extended message confirming the establishment of the multiparty call to the first user, and send a second extended message to the third user who is talking with the first user A third extended message, where the third extended message includes list information of members participating in the multi-party call;
向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息；sending a request to the second user to join the multi-party call, where the request includes list information of members participating in the multi-party call;
接收所述第二用户发送的确认加入的信息；receiving the joining confirmation information sent by the second user;
分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道；Establishing media channels for multi-party calls between the first user and the second user, and between the third user and the second user, respectively;
通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话。The multi-party call is performed through the established media channel for the multi-party call.
在第一种可能的实现方式中，结合第一方面，所述多方通话至少包括音频流和视频流的传输。In a first possible implementation manner, with reference to the first aspect, the multiparty call includes at least transmission of audio streams and video streams.
在第二种可能的实现方式中，结合第一方面，所述方法还包括：In a second possible implementation manner, in combination with the first aspect, the method further includes:
当所述多方通话基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。When the multiparty call is based on Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second extended message, and the third extended message are extended messages based on the SIP.
在第三种可能的实现方式中，结合第一方面，所述第二用户为至少一个用户终端。In a third possible implementation manner, with reference to the first aspect, the second user is at least one user terminal.
第二方面，提供一种基于Web RTC多方通话建立的方法，所述方法包括：In the second aspect, a method for establishing a multi-party call based on Web RTC is provided, the method comprising:
向应用服务器发送多方通话建立请求，所述请求包括第一扩展消息，所述第一扩展消息中待建立多方通话的第二用户的消息；Sending a multi-party call establishment request to the application server, where the request includes a first extended message, and in the first extended message, a message of a second user whose multi-party call is to be established;
接收所述应用服务器发送的确认建立多方通话的第二扩展消息；receiving a second extended message sent by the application server to confirm the establishment of the multi-party call;
建立与所述第二用户的用于多方通话的媒体通道；establishing a media channel for a multi-party call with the second user;
通过所述媒体通道进行多方通话。A multi-party conversation is performed through the media channel.
在第一种可能的实现方式中，结合第二方面，所述建立与所述第二用户的用于多方通话的媒体通道包括：In a first possible implementation manner, in combination with the second aspect, the establishing a media channel for a multiparty call with the second user includes:
向所述第二用户发送建立所述媒体通道的邀请信息；sending invitation information for establishing the media channel to the second user;
接收所述第二用户发送的回复邀请的信息，建立与所述第二用户的媒体通道。receiving the reply invitation information sent by the second user, and establishing a media channel with the second user.
在第二种可能的实现方式中，结合第二方面，所述方法包括：In a second possible implementation manner, in combination with the second aspect, the method includes:
获取本地的媒体流，保存所述本地的媒体流；Obtain a local media stream, and save the local media stream;
将所述本地的媒体流通过与所述第二用户间的媒体通道发送至所述第二用户，从与所述第二用户间的媒体通道接收所述第二用户的媒体流。sending the local media stream to the second user through the media channel with the second user, and receiving the media stream of the second user through the media channel with the second user.
在第三种可能的实现方式中，结合第二方面，所述方法还包括：In a third possible implementation manner, in combination with the second aspect, the method further includes:
将所述本地的媒体流通过与第三用户间的媒体通道发送至所述第三用户，从与所述第三用户间的媒体通道接收所述第三用户的媒体流。The local media stream is sent to the third user through the media channel with the third user, and the media stream of the third user is received through the media channel with the third user.
第三方面，提供一种基于Web RTC多方通话的设备，所述设备包括：In a third aspect, a device based on Web RTC multi-party call is provided, and the device includes:
第一接收单元，用于接收正在通话的第一用户发送的多方通话建立请求，所述请求包括第一扩展消息，所述第一扩展消息中有待与所述第一用户建立多方通话的第二用户的信息；The first receiving unit is configured to receive a multi-party call establishment request sent by the first user who is talking, the request includes a first extended message, and in the first extended message, the second party to establish a multi-party call with the first user user information;
权限判断单元，用于判断所述第一用户建立所述多方通话的权限；an authority judgment unit, configured to determine the authority of the first user to establish the multi-party call;
第一消息发送单元，用于当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，所述第三扩展消息中包括参加所述多方通话成员的列表信息；A first message sending unit, configured to send a second extended message to the first user confirming establishment of the multi-party call when the first user has the authority to establish the multi-party call, and communicate with the first user A third user whose user is talking sends a third extended message, and the third extended message includes list information of members participating in the multi-party call;
第一请求发送单元，用于向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息，并接收所述第二用户发送的确认加入的信息；A first request sending unit, configured to send a request to the second user to join the multi-party call, the request includes list information of members participating in the multi-party call, and receive confirmation of joining sent by the second user information;
第一通道建立单元，用于分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道；a first channel establishing unit, configured to establish media channels for multi-party calls between the first user and the second user, and between the third user and the second user;
第一多方通话单元，用于通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话。The first multi-party call unit is configured to conduct the multi-party call through the established media channel for the multi-party call.
在第一种可能的实现方式中，结合第三方面，所述多方通话至少包括音频流和视频流的传输。In a first possible implementation manner, with reference to the third aspect, the multiparty call includes at least transmission of audio streams and video streams.
在第二种可能的实现方式中，结合第三方面，在所述设备中，当所述多方通话基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。In a second possible implementation manner, in combination with the third aspect, in the device, when the multiparty call is based on Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second The extended message, the third extended message is an extended message based on the SIP.
在第三种可能的实现方式中，结合第三方面，所述第二用户为至少一个用户终端。In a third possible implementation manner, with reference to the third aspect, the second user is at least one user terminal.
第四方面，提供一种基于Web RTC多方通话建立的系统，所述设备至少包括：In the fourth aspect, a system based on Web RTC multi-party call establishment is provided, and the device at least includes:
如第一方面所示的会议应用服务器，或如第三方面所述的会议应用服务器；The conference application server as described in the first aspect, or the conference application server as described in the third aspect;
如第二方面所示的第一用户。The first user as shown in the second aspect.
本发明实施例提供一种基于Web RTC多方通话建立的方法、设备和系统，通过第一用户向会议应用服务器发送建立多方通话的请求，会议应用服务器判断第一用户是否有建立多方通话的权限，在确定第一用户具有建立多方通话的请求后，向待参加多方通话的用户发送加入多方会议的请求并附带有该多方通话的成员列表信息，以便于其他用户对是否加入多方通话进行判断，待其他用户向会议应用服务器发送加入多方会议的信息后，令第一用户向其他用户发送建立媒体通道的邀请，并在其他用户回复接收建立媒体通道的信息，从而成功建立第一用户与其他用户的媒体通道。最终通过建立的媒体通道，并结合媒体流复用以及浏览器内音视频标签的技术，从而实现多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。Embodiments of the present invention provide a method, device and system for establishing a multi-party call based on Web RTC. The first user sends a request for establishing a multi-party call to the conference application server, and the conference application server determines whether the first user has the authority to establish a multi-party call. After determining that the first user has a request to establish a multi-party call, send a request to join the multi-party conference to the user who is waiting to join the multi-party call with the member list information of the multi-party call attached, so that other users can judge whether to join the multi-party call. After other users send the information of joining the multi-party conference to the conference application server, the first user sends an invitation to establish a media channel to other users, and receives the information of establishing a media channel when other users reply, thereby successfully establishing a relationship between the first user and other users. media channel. Finally, through the established media channel, combined with media stream multiplexing and the technology of audio and video tags in the browser, multi-party calls can be realized; it can reduce the high performance requirements of the equipment caused by local audio mixing, and can also eliminate the need to contact the venue The server performs cumbersome site resource application, which further saves the waste of site resources when the multi-party call is restored to a two-party call, and finally reduces the establishment steps of the multi-party call and improves the use efficiency of communication resources.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.
图1为本发明实施例提供的浏览器承载Web RTC应用的层次图；Fig. 1 is the hierarchical diagram of the browser carrying Web RTC application that the embodiment of the present invention provides;
图2为本发明实施例提供的一种基于Web RTC多方通话建立的方法的流程图；Fig. 2 is the flow chart of a kind of method based on Web RTC multi-party call establishment provided by the embodiment of the present invention;
图3为本发明实施例提供的一种基于Web RTC多方通话建立的方法的详细流程图；Fig. 3 is a detailed flowchart of a method for establishing a Web RTC-based multiparty call provided by an embodiment of the present invention;
图4为本发明实施例提供的一种基于Web RTC多方通话建立的方法的流程图；Fig. 4 is the flow chart of a kind of method based on Web RTC multi-party call establishment provided by the embodiment of the present invention;
图5为本发明实施例提供的一种基于Web RTC多方通话建立的方法的详细流程图；Fig. 5 is a detailed flowchart of a method for establishing a Web RTC-based multiparty call provided by an embodiment of the present invention;
图6为本发明实施例提供一种基于Web RTC多方通话建立的设备的结构示意图；FIG. 6 is a schematic structural diagram of a device based on Web RTC multi-party call establishment provided by an embodiment of the present invention;
图7为本发明实施例提供另一种基于Web RTC多方通话建立的装置的结构示意图；7 is a schematic structural diagram of another device for establishing a Web RTC-based multi-party call according to an embodiment of the present invention;
图8为本发明实施例提供另一种基于Web RTC多方通话建立的系统的结构示意图。FIG. 8 is a schematic structural diagram of another system for establishing a multi-party call based on Web RTC according to an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
本发明实施例基于Web RTC协议，通过该协议提供的基于web浏览器的实时的点对点技术（Peer to Peer，P2P），实现包括语音、视频、实时协作、数据传输等的通信。与已有的基于web的通信技术不同的是，该协议不需要浏览器安装任何插件及附加软件，通过浏览器内置的音视频解码器，以及信令面和媒体面的协议栈，再加上对JavaScript控制会话建立过程协议即JSEP协议的支持，就可以实现基于Web RTC的网络语音电话业务（Voice over Internet Phone，VoIP）。The embodiment of the present invention is based on the Web RTC protocol, and the real-time peer-to-peer technology (Peer to Peer, P2P) based on the web browser provided by the protocol realizes communication including voice, video, real-time collaboration, data transmission, etc. Different from the existing web-based communication technologies, this protocol does not require the browser to install any plug-ins and additional software, through the built-in audio and video decoders in the browser, as well as the protocol stack of the signaling plane and the media plane, plus Supporting the JavaScript-controlled session establishment process protocol, that is, the JSEP protocol, can realize the Voice over Internet Phone (VoIP) service based on Web RTC.
具体的在Web RTC的应用中，信令通道、信令消息的生成和解析、会话的控制和管理均是由JavaScript应用层实现的；而媒体参数协商、ICE协商、RTP协议栈、媒体编解码等是由浏览器底层实现的。详细的浏览器承载Web RTC应用的层次图如图1所示。Specifically, in the application of Web RTC, the signaling channel, the generation and analysis of signaling messages, and the control and management of sessions are all implemented by the JavaScript application layer; while media parameter negotiation, ICE negotiation, RTP protocol stack, media codec etc. are implemented by the bottom layer of the browser. The detailed hierarchical diagram of browser hosting Web RTC application is shown in Figure 1.
在图1所示的Web RTC应用中，媒体协商、NAT穿越、RTP协议栈、媒体流的获取呈现、编码解码均是有浏览器底层实现的，浏览器通过开放JavaScript API供应用程序调用，应用程序因此可以根据当前会话的状态控制浏览器底层做出相应的动作。除此之外，应用程序还负责信令通道的维护，将来自浏览器底层的媒体参数封装成相应的信令信息，或解析来自对方的信令消息等。浏览器底层通过上层的应用程序来了和通信另一方交换媒体参数，从而完成媒体协商。In the Web RTC application shown in Figure 1, media negotiation, NAT traversal, RTP protocol stack, acquisition and presentation of media streams, and encoding and decoding are all implemented by the bottom layer of the browser. Therefore, the program can control the bottom layer of the browser to make corresponding actions according to the state of the current session. In addition, the application program is also responsible for the maintenance of the signaling channel, packaging the media parameters from the bottom layer of the browser into corresponding signaling information, or parsing the signaling messages from the other party, etc. The bottom layer of the browser exchanges media parameters with the other party through the application program of the upper layer, so as to complete the media negotiation.
本发明实施例提供一种基于Web RTC多方通话建立的方法，如图2所示，该方法包括：The embodiment of the present invention provides a method for establishing a multi-party call based on Web RTC, as shown in Figure 2, the method includes:
在本实施例的起始阶段，第一用户正在与第三用户进行两方通话，但第一用户需要建立多方通话，新增的用户为第二用户。In the initial stage of this embodiment, the first user is conducting a two-party call with the third user, but the first user needs to establish a multi-party call, and the newly added user is the second user.
101、第一用户向会议应用服务器发送多方通话建立请求。该请求中包括第一扩展消息，所述第一扩展消息中有待于第一用户建立多方通话的第二用户的信息。101. The first user sends a multi-party call establishment request to a conference application server. The request includes a first extended message, and the first extended message includes information of the second user to be established by the first user to establish a multi-party call.
上述多方通话至少包括音频流和视频流的传输。进一步的，当上述多方通话是基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。The above-mentioned multi-party call includes at least the transmission of audio streams and video streams. Further, when the above-mentioned multiparty call is based on Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second extended message, and the third extended message are extended messages based on the SIP.
具体的，当上述多方通话中的信令协议采用的是SIP协议时，通过重新定义的XML数据格式，扩展SIP协议中的Message消息体，利用SIP协议的路由机制对Message消息进行路由，进而实现应用服务器与客户端之间的信息交互。Specifically, when the signaling protocol in the above-mentioned multi-party call adopts the SIP protocol, the Message body in the SIP protocol is extended through the redefined XML data format, and the routing mechanism of the SIP protocol is used to route the Message message, thereby realizing Information exchange between application server and client.
示例性的，以第一用户为例，客户端发起多方通话请求到应用服务器的扩展消息如下所示：Exemplarily, taking the first user as an example, the extended message sent from the client to the application server by initiating a multi-party call request is as follows:
在上述消息中，包括了该消息的序列号“4001”、消息发送者的名称“A”、连接类型及对应的会话参与者“web RTC---B”、“web---C”等信息。In the above message, the sequence number "4001" of the message, the name of the sender "A", the connection type and the corresponding session participants "web RTC---B", "web---C" etc. are included. information.
102、会议应用服务器接收正在通话的第一用户发送的多方通话建立请求。该请求中包括第一扩展消息，所述第一扩展消息中有待与所述第一用户建立多方通话的第二用户的信息。102. The conference application server receives a multi-party call establishment request sent by the first user who is talking. The request includes a first extended message, and the first extended message includes information of a second user to establish a multiparty call with the first user.
103、会议应用服务器判断第一用户建立多方通话的权限。103. The conference application server determines the authority of the first user to establish a multi-party call.
104、当第一用户具有建立多方通话的权限时，应用服务器向第一用户发送建立多方通话的第二扩展消息，并向与第一用户正在通话的第三用户发送第三扩展消息。该第三扩展消息中包括参加所述多方通话成员的列表信息。104. When the first user has the right to establish a multi-party call, the application server sends a second extended message for establishing a multi-party call to the first user, and sends a third extended message to a third user who is in a call with the first user. The third extended message includes list information of members participating in the multi-party call.
与步骤101中信息类似的，应用服务器向第一用户发送建立多方通话的相应消息举例如下。Similar to the information in step 101, an example of the application server sending a corresponding message for establishing a multi-party call to the first user is as follows.
上述消息包括了该消息的序号cmd、多方通话建立的结果res、以及该多方通话的序号id，具体的，以上述消息为例，cmd的值4002为建立的多方通话的消息序号，res的值0表示该多方通话已成功建立，id的值0001表示该多方通话的序号为0001，若res的值不为0时，表明该多方通话由于其他的原因未能成功建立。The above message includes the serial number cmd of the message, the result res of the multi-party call establishment, and the serial number id of the multi-party call. Specifically, taking the above message as an example, the value of cmd 4002 is the message serial number of the established multi-party call, and the value of res 0 means that the multi-party call has been successfully established, and the value of id 0001 means that the serial number of the multi-party call is 0001. If the value of res is not 0, it means that the multi-party call has not been successfully established due to other reasons.
105、会议应用服务器向第二用户发送加入多方通话的请求，该请求中包括参加所述多方通话成员的列表信息。105. The conference application server sends a request for joining the multi-party call to the second user, where the request includes list information of members participating in the multi-party call.
该多方通话的请求是会议应用服务器在确认了多方通话的发起者具有建立多方通话的权限后，向需要加入多方通话的第三方发送加入多方会议的请求，并携带多方通话成员的列表信息，以便于第三方对会议发起者进行权限的判断。The request for the multi-party call is that the conference application server sends a request to join the multi-party conference to a third party that needs to join the multi-party call after confirming that the initiator of the multi-party call has the authority to establish the multi-party call, and carries the list information of the multi-party call members, so that It depends on the judgment of the authority of the conference initiator by the third party.
在本实施例中，会议应用服务器向第二用户发送由第一用户发起的多方通话的加入请求，并且在该加入请求中还包括了与此多方通话相关的成员列表信息，以便于第二用户对此多方通话加入的必要性进行判断，并根据判断结果确定是否加入该多方通话。In this embodiment, the conference application server sends to the second user a request to join the multi-party call initiated by the first user, and the join request also includes member list information related to the multi-party call, so that the second user can Judging the necessity of joining the multi-party call, and determining whether to join the multi-party call according to the judgment result.
106、第二用户接收应用服务器发送的多方通话的请求，该请求中包括参加所述多方通话的列表信息。106. The second user receives a request for a multi-party call sent by the application server, where the request includes list information for participating in the multi-party call.
在接收到应用服务器发送的多方通话的请求，根据请求中包括的参加多方通话的列表信息对该多方通话的必要性进行判断。After receiving the multi-party call request sent by the application server, the necessity of the multi-party call is judged according to the list information of participating in the multi-party call included in the request.
107、第二用户在确认该多方通话请求的正确性后，向应用服务器发送确认加入多方通话的信息。107. After confirming the correctness of the multi-party call request, the second user sends information confirming joining the multi-party call to the application server.
该确认加入的信息表示第二用户经过对加入该多方通话的必要性进行判断后，决定加入由第一用户发起的该多方通话。The joining confirmation information indicates that the second user decides to join the multi-party call initiated by the first user after judging the necessity of joining the multi-party call.
若第二用户经过判断后，决定不加入该多方通话，则向应用服务器发送拒绝加入的消息。If the second user decides not to join the multi-party call after judgment, a message of refusal to join is sent to the application server.
108、应用服务器接收第二用户发送的确认加入的信息。108. The application server receives the joining confirmation information sent by the second user.
109、应用服务器分别在第一用户与第二用户间、第三用户与第二用户间建立用于多方通话的媒体通道。109. The application server respectively establishes media channels for the multiparty call between the first user and the second user, and between the third user and the second user.
这里建立的媒体通道分别用于第一用户与第二用户、第三用户与第二用户进行通话，使得在该多方通话中，任意两方之间均有直接的点到点的用于传输音频流和视频流的传输通道，最终使得在整个多方通话中，每一个用户均有与其他任意用户相连的通道，进一步的，每一个用户均作为一个Web RTC客户端，在获取到用户自身的至少包括音频流和视频流的媒体流，将该媒体流于本地进行保存后，将获取到的媒体流可以通过与其他用户相连的媒体通道发送至其他用户处，从而实现了多方通话。The media channels established here are respectively used for the first user to communicate with the second user, and the third user to communicate with the second user, so that in the multi-party conversation, there is direct point-to-point audio transmission between any two parties. The transmission channel of streaming and video stream finally makes each user have a channel to connect with any other user in the whole multi-party call. Further, each user acts as a Web RTC client, after obtaining at least the user's own Media streams including audio streams and video streams, after the media streams are stored locally, the acquired media streams can be sent to other users through media channels connected to other users, thereby realizing multi-party calls.
进一步的，每个用户即每一个Web RTC客户端在接收到其他客户端通过对应的媒体通道传输过来的媒体流时，在用户浏览器的页面上新建多个HTML5的<Audio>标签，每个标签的源Source设置为其中一个客户端的媒体流，这样通过多个<Audio>标签就可以同时播放多个媒体流，也就是在该客户端中可以同时听到多个客户端的声音；对应媒体流中的视频流的处理方法类似，通过建立多个<Video>标签，使得在客户端中可以同时播放多个客户端的视频图像，通过上述对媒体流进行复用及同时处理多路的媒体流，从而实现多路通话。Furthermore, each user, that is, each Web RTC client, when receiving the media stream transmitted by other clients through the corresponding media channel, creates multiple HTML5 <Audio> tags on the page of the user's browser, each The source Source of the tag is set to the media stream of one of the clients, so that multiple media streams can be played at the same time through multiple <Audio> tags, that is, the sound of multiple clients can be heard in the client at the same time; the corresponding media stream The processing method of the video stream in is similar. By creating multiple <Video> tags, the video images of multiple clients can be played simultaneously in the client. Through the above-mentioned multiplexing of media streams and simultaneous processing of multiple media streams, Thereby realizing multi-way call.
与步骤101及104中的消息格式类似，会议应用服务器控制待加入的用户接入多方通话的信息示例性的为：Similar to the message format in steps 101 and 104, the information that the conference application server controls the user to join to access the multi-party call is exemplarily:
上述信息包括了该消息的序号cmd、该多方通话的主控制方（也称为该多方通话的主席）的序号userid、以及该多方通话的其他参与者的序号attendee。The above information includes the serial number cmd of the message, the serial number userid of the main controller of the multi-party call (also called the chairperson of the multi-party call), and the serial number attendee of other participants in the multi-party call.
具体的，以上述消息为例，cmd的值4003为该多方通话的序号，userID的值为a表明该多方通话的主控制方（或该多方通话的主席）的序号，attendee的值分别为b、c表明该多方通话的其他参与者的序号。Specifically, taking the above message as an example, the value of cmd is 4003 is the serial number of the multi-party call, the value of userID is a indicating the serial number of the main controller of the multi-party call (or the chairperson of the multi-party call), and the values of attendee are respectively b , c indicate the sequence numbers of other participants in the multi-party call.
根据上述描述，如图3所示，建立媒体通道的具体步骤包括：According to the above description, as shown in Figure 3, the specific steps for establishing a media channel include:
1091、第一用户向第二用户发送建立媒体通道的邀请信息。1091. The first user sends invitation information for establishing a media channel to the second user.
1092、第二用户接收第一用户发送的建立媒体通道的邀请信息。1092. The second user receives invitation information for establishing a media channel sent by the first user.
1093、第二用户判断邀请信息发送方的正确性。1093. The second user judges the correctness of the sender of the invitation information.
这里是第二用户根据事先从应用服务器接收到的参加多方通话成员的列表信息，判断邀请信息的发送方也就是第一用户的正确性，或者是第一用户是否具有发起多方通话的权限。Here, the second user judges whether the sender of the invitation message, that is, the first user, is correct according to the list information of the multiparty call members received from the application server in advance, or whether the first user has the authority to initiate the multiparty call.
1094、第二用户确定邀请信息发送方正确后，向第一用户回复接收邀请的信息，并建立于第一用户的媒体通道。1094. After confirming that the sender of the invitation information is correct, the second user replies to the first user with information about receiving the invitation, and establishes a media channel with the first user.
由于第二用户已经判断过邀请信息发送方也就是第一用户发起多方通话的权限，因此，第二用户在建立于第一用户的媒体通道后，再接收到来自于第一用户的消息后，采取有别于正常通话消息的处理方式。Since the second user has judged that the sender of the invitation information, that is, the authority of the first user to initiate a multiparty call, after the second user establishes the media channel of the first user and receives the message from the first user, Treat messages differently than normal call messages.
在正常的通话方式下，由于第二用户已经处于多方通话的环境中，也就是处于通话状态，因此，再接收到来自于其他用户的通话请求后，默认是要向其他用户返回“您所拨打的用户正在通话中”等类似的占线信息；但是当前由于第一用户是处于第二用户事先从会议应用服务器的多方通话成员的信息列表中的，因此，对于像第一用户这样来自于多方通话成员的信息列表中的用户发送的通话请求及通话信息，第二用户会采取会议内会话请求处理方式，即不弹出对话窗口或者不触发振铃模式，直接自动接收通话。In a normal call mode, since the second user is already in a multi-party call environment, that is, in a call state, after receiving a call request from another user, the default is to return to the other user "You dialed The user is in a call" and other similar busy information; but currently, because the first user is in the information list of the multi-party call members from the conference application server in advance by the second user, therefore, for the first user who comes from the multi-party call For the call request and call information sent by the user in the member information list, the second user will automatically receive the call directly without popping up the dialog window or triggering the ringing mode.
尤其是在第二用户与第一用户建立了媒体通道后的第一条通话时，采取上述“静默”处理的优点在于可以令第二用户更快的进入到已接入多方通话的状态，也就是仅仅是在第二用户向会议应用处理器返回加入多方通话后，针对于由第一用户或其他用户发送的建立媒体通道的请求，不再进行多余的提示，这样有助于令第二用户尽快的进入多方通话的状态中，避免第二用户会接收到多次加入多方通话的提示，从而提升第二用户的体验。Especially in the first call after the second user establishes a media channel with the first user, the advantage of adopting the above-mentioned "silent" processing is that the second user can enter the state of having connected to the multi-party call faster, and also That is, after the second user returns to the conference application processor to join the multi-party call, no redundant prompts will be given for the request for establishing a media channel sent by the first user or other users, which will help the second user Enter the multi-party call state as soon as possible to prevent the second user from receiving multiple prompts to join the multi-party call, thereby improving the experience of the second user.
至此，第一用户与第二用户之间的信息传输通道已经完全建立完毕，至于之前与第一用户正在进行双方通话的第三用户，它与第二用户建立媒体通道的方式与上述针对于第一用户和第二用户建立媒体通道的方法完全一致，并且值得一提的是，这里的第三用户与第二用户建立媒体通道的时间和第一用户与第二用户建立媒体通道在时间上是同时进行的，这样可以最大程度上节省建立会议媒体通道的时间，在本实施例中为了表述清晰，故将二者分开进行描述，并在此进行说明。So far, the information transmission channel between the first user and the second user has been completely established. As for the third user who was in a two-way conversation with the first user before, the way it establishes a media channel with the second user is the same as the above-mentioned method for the second user. The method for establishing a media channel between a user and a second user is exactly the same, and it is worth mentioning that the time when the third user establishes a media channel with the second user is the same as the time when the first user establishes a media channel with the second user. They are performed at the same time, which can save the time of establishing the conference media channel to the greatest extent. In this embodiment, for the sake of clarity, the two are described separately, and will be described here.
110、按照步骤1091至1094所示的方法，依次完成第一用户、第三用户与所有待参加多方通话用户的媒体通道的建立，并通过建立好的媒体通道，进行多方通话。110. According to the method shown in steps 1091 to 1094, complete the establishment of media channels between the first user, the third user and all users to participate in the multi-party call in sequence, and conduct the multi-party call through the established media channels.
本发明实施例提供一种基于Web RTC多方通话建立的方法，通过第一用户向会议应用服务器发送建立多方通话的请求，会议应用服务器判断第一用户是否有建立多方通话的权限，在确定第一用户具有建立多方通话的请求后，向待参加多方通话的用户发送加入多方会议的请求并附带有该多方通话的成员列表信息，以便于其他用户对是否加入多方通话进行判断，待其他用户向会议应用服务器发送加入多方会议的信息后，令第一用户向其他用户发送建立媒体通道的邀请，并在其他用户回复接收建立媒体通道的信息，从而成功建立第一用户与其他用户的媒体通道。最终通过建立的媒体通道，并结合媒体流复用以及浏览器内音视频标签的技术，从而实现多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。The embodiment of the present invention provides a method for establishing a multi-party call based on Web RTC. The first user sends a request for establishing a multi-party call to the conference application server, and the conference application server judges whether the first user has the authority to establish a multi-party call. After the user has a request to establish a multi-party call, send a request to join the multi-party conference to the user who is waiting to participate in the multi-party call and attach the member list information of the multi-party call, so that other users can judge whether to join the multi-party call. After the application server sends the information of joining the multi-party conference, the first user sends an invitation to establish a media channel to other users, and receives the information of establishing a media channel when other users reply, thereby successfully establishing the media channel between the first user and other users. Finally, through the established media channel, combined with media stream multiplexing and the technology of audio and video tags in the browser, multi-party calls can be realized; it can reduce the high performance requirements of the equipment caused by local audio mixing, and can also eliminate the need to contact the venue The server performs cumbersome site resource application, which further saves the waste of site resources when the multi-party call is restored to a two-party call, and finally reduces the establishment steps of the multi-party call and improves the use efficiency of communication resources.
本发明实施例提供一种基于Web RTC多方通话建立的方法，如图4所示，该方法包括：The embodiment of the present invention provides a method for establishing a multi-party call based on Web RTC, as shown in Figure 4, the method includes:
本实施例用于将正在通话的两方，变更成三方通话的通话形式。由于使用范围较小，因此仅作为上一实施例将双方通话变更为多方通话的特例，也作为一种基于Web RTC协议，实现多方通话方法的补充。This embodiment is used to change the two parties in the conversation into a three-party conversation. Due to the small scope of use, it is only used as a special case of changing the two-party call to a multi-party call in the previous embodiment, and also as a supplement to the method for realizing a multi-party call based on the Web RTC protocol.
在本实施例的起始阶段，第一用户与第二用户处于通话保持状态，同时第一用户与第三用户处于正在通话中。In the initial stage of this embodiment, the first user and the second user are on hold, and at the same time, the first user and the third user are in the middle of a call.
201、第一用户向应用服务器发送建立多方通话的第一信息。201. A first user sends first information for establishing a multiparty call to an application server.
上述多方通话至少包括音频流和视频流的传输。进一步的，当上述多方通话是基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。The above-mentioned multi-party call includes at least the transmission of audio streams and video streams. Further, when the above-mentioned multiparty call is based on Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second extended message, and the third extended message are extended messages based on the SIP.
具体的，当上述多方通话中的信令协议采用的是SIP协议时，通过重新定义的XML数据格式，扩展SIP协议中的Message消息体，利用SIP协议的路由机制对Message消息进行路由，进而实现应用服务器与客户端之间的信息交互。Specifically, when the signaling protocol in the above-mentioned multi-party call adopts the SIP protocol, the Message body in the SIP protocol is extended through the redefined XML data format, and the routing mechanism of the SIP protocol is used to route the Message message, thereby realizing Information exchange between application server and client.
202、应用服务器判断第一用户建立多方通话的权限。202. The application server determines the authority of the first user to establish a multi-party call.
203、当第一用户具有建立多方通话的权限后，应用服务器向第一用户发送成功建立多方通话的第二消息。203. After the first user has the authority to establish the multiparty call, the application server sends a second message that the multiparty call is successfully established to the first user.
204、应用服务器向第三用户发送第三消息，所述第三消息中包括第二用户的相关信息，和第一用户成功建立多方通话的消息。204. The application server sends a third message to the third user, where the third message includes relevant information of the second user and a message that the first user successfully establishes a multiparty call.
205、第三用户接收到第三消息后，将自身的状态变更为“多方通话”。205. After receiving the third message, the third user changes its status to "multi-party call".
206、应用服务器向第二用户发送第四消息，所述第四消息中包括第三用户的相关信息，和第一用户成功建立多方通话的消息。206. The application server sends a fourth message to the second user, where the fourth message includes relevant information of the third user and a message that the first user successfully establishes a multiparty call.
207、第二用户将自身的状态变更为“多方通话”，并与第三用户建立媒体通道。207. The second user changes his status to "multi-party call", and establishes a media channel with the third user.
208、在第二用户与第三用户建立媒体通道后，结合已经处于通话状态的第一用户和第三用户，实现多方通话。208. After the second user establishes a media channel with the third user, combine the first user and the third user who are already in a call state to implement a multi-party call.
这里建立的媒体通道分别用于第二用户与第三用户进行通话，使得在该多方通话中，任意两方之间均有直接的点到点的用于传输音频流和视频流的传输通道，最终使得在整个多方通话中，每一个用户均有与其他任意用户相连的通道，进一步的，每一个用户均作为一个WebRTC客户端，在获取到用户自身的至少包括音频流和视频流的媒体流，将该媒体流于本地进行保存后，将获取到的媒体流可以通过与其他用户相连的媒体通道发送至其他用户处，从而实现了多方通话。The media channels established here are respectively used for the second user to communicate with the third user, so that in the multi-party call, there is a direct point-to-point transmission channel for transmitting audio streams and video streams between any two parties. Finally, in the entire multi-party call, each user has a channel connected to any other user. Further, each user acts as a WebRTC client, and obtains the user's own media stream including at least audio stream and video stream. , after saving the media stream locally, the obtained media stream can be sent to other users through the media channel connected to other users, thus realizing multi-party conversation.
进一步的，每个用户即每一个Web RTC客户端在接收到其他客户端通过对应的媒体通道传输过来的媒体流时，在用户浏览器的页面上新建多个HTML5的<Audio>标签，每个标签的源Source设置为其中一个客户端的媒体流，这样通过多个<Audio>标签就可以同时播放多个媒体流，也就是在该客户端中可以同时听到多个客户端的声音；对应媒体流中的视频流的处理方法类似，通过建立多个<Video>标签，使得在客户端中可以同时播放多个客户端的视频图像，通过上述对媒体流进行复用及同时处理多路的媒体流，从而实现多路通话。Furthermore, each user, that is, each Web RTC client, when receiving the media stream transmitted by other clients through the corresponding media channel, creates multiple HTML5 <Audio> tags on the page of the user's browser, each The source Source of the tag is set to the media stream of one of the clients, so that multiple media streams can be played at the same time through multiple <Audio> tags, that is, the sound of multiple clients can be heard in the client at the same time; the corresponding media stream The processing method of the video stream in is similar. By creating multiple <Video> tags, the video images of multiple clients can be played simultaneously in the client. Through the above-mentioned multiplexing of media streams and simultaneous processing of multiple media streams, Thereby realizing multi-way call.
在步骤207中，如图5所述，第二用户与第三用户建立媒体通道具体包括：In step 207, as shown in FIG. 5, establishing a media channel between the second user and the third user specifically includes:
2071、第二用户恢复与第一用户的通话状态，并向应用服务器发送建立媒体通道的邀请信息，所述邀请信息中包括第二用户的属性参数。2071. The second user resumes the call state with the first user, and sends invitation information for establishing a media channel to the application server, where the invitation information includes attribute parameters of the second user.
2072、应用服务器判断所述邀请消息为多方通话内的呼叫请求，在不触发补充业务的前提下，将所述邀请信息转发至第三用户。2072. The application server determines that the invitation message is a call request in a multi-party call, and forwards the invitation message to the third user without triggering a supplementary service.
2073、第三用户根据已确定的“多方通话”的状态，并结合所述邀请消息为多方通话内的呼叫请求，向应用服务器发送第五消息，所述第五消息包括第三用户的属性参数。2073. The third user sends a fifth message to the application server according to the determined state of the "multi-party call" and combining the invitation message as a call request in the multi-party call, and the fifth message includes the attribute parameters of the third user .
2074、应用服务器将第五消息发送至第二用户，成功建立第二用户与第三用户的媒体通道。2074. The application server sends the fifth message to the second user, and successfully establishes a media channel between the second user and the third user.
由于第三用户已经与第一用户处于通话状态，因此，步骤2072中，应用服务器在向第三用户转发所述邀请信息前，需要对所述邀请信息的发送方进行判断。Since the third user is already in a conversation with the first user, in step 2072, before forwarding the invitation information to the third user, the application server needs to judge the sender of the invitation information.
在正常的通话方式下，由于第三用户已经处于多方通话的环境中，也就是处于通话状态，因此，再接收到来自于其他用户的信息后，默认是要向其他用户返回“您所拨打的用户正在通话中”等类似的占线信息；但是当前由于发送消息的第二用户是处于应用服务器已经获取到的多方通话成员中的，因此，对于像第二用户这样来自于多方通话成员中的用户发送的通话请求及通话信息，第三用户会采取会议内会话请求处理方式，即不弹出对话窗口或者不触发振铃模式，直接自动接收通话。In a normal call mode, since the third user is already in a multi-party call environment, that is, in a call state, after receiving information from other users, the default is to return "the number you dialed" to the other user. The user is in a call" and other similar busy information; but currently because the second user who sends the message is in the multi-party call members that the application server has acquired, therefore, for users from the multi-party call members like the second user For the call request and call information sent, the third user will adopt the in-conference conversation request processing method, that is, the conversation will be automatically received without popping up the dialogue window or triggering the ringing mode.
尤其是在第三用户已经将自身状态变更为“多方通话”时，采取上述“静默”处理的优点在于可以令第三用户更快的进入到已接入多方通话的状态，也就是仅仅是在第三用户接收到应用处理器发送的第三消息后，针对于由第二用户或其他用户发送的建立媒体通道的请求，不再进行多余的提示，这样有助于令第三用户尽快的进入多方通话的状态中，避免第三用户会接收到多次加入多方通话的提示，从而提升第三用户的体验。Especially when the third user has changed his status to "multi-party call", the advantage of adopting the above-mentioned "silent" process is that the third user can enter the state of having connected to the multi-party call faster, that is, only in the After the third user receives the third message sent by the application processor, no redundant prompts will be given for the request for establishing a media channel sent by the second user or other users, which helps the third user to enter the media channel as soon as possible. In the state of the multi-party call, the third user is prevented from receiving multiple prompts to join the multi-party call, thereby improving the experience of the third user.
在上述步骤中，各个用户与会议应用服务器及呼叫应用服务器之间的信息传输，均需要经过会话管理器Session Manager的转发，这样可以实现消息传输的准确性与及时性，以免由于会议应用服务器和呼叫服务器由于信息处理不及时导致的信息丢失或延迟。因为上述步骤众多，信息传输路径复杂，就没有将会话管理器在步骤中的信息转发过程进行描述，仅是将其功能在这里进行统一描述，但并不代表会话管理器没有参与上述步骤中的信息传输。In the above steps, the information transmission between each user and the conference application server and the call application server needs to be forwarded by the session manager Session Manager. Information loss or delay caused by call server due to untimely information processing. Because the above steps are many and the information transmission path is complicated, the information forwarding process of the session manager in the steps is not described, but its functions are described here in a unified manner, but it does not mean that the session manager does not participate in the above steps. Information transfer.
本发明实施例提供一种基于Web RTC多方通话建立的方法，通过第一用户向应用服务器发送建立多方通话的第一消息，应用服务器判断第一用户是否具有建立多方通话的权限，并在确定第一用户具有建立多方通话的权限后，向第一用户发送成功建立多方通话的第二消息，同时应用服务器向第三用户和第二用户分别发送第三、第四消息，以便第三用户和第二用户获取第一用户已经成功建立多方通话的消息并变更自身状态，在第二用户和第三用户建立媒体通道后，结合已经处于通话状态的第一用户和第三用户，最终实现多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。The embodiment of the present invention provides a method for establishing a multi-party call based on Web RTC. The first user sends the first message for establishing a multi-party call to the application server, and the application server judges whether the first user has the authority to establish a multi-party call. After a user has the authority to establish a multi-party call, the second message that the multi-party call is successfully established is sent to the first user, and at the same time, the application server sends the third and fourth messages to the third user and the second user respectively, so that the third user and the second user The second user obtains the message that the first user has successfully established a multi-party call and changes its own state. After the second user and the third user establish a media channel, combine the first user and the third user who are already in the call state to finally realize the multi-party call; It can reduce the high performance requirements of the equipment caused by local audio mixing, and also eliminates the need to contact the site server for cumbersome site resource applications, further saving the waste of site resources when a multi-party call returns to a two-party call, and ultimately reduces The step of establishing a multi-party call improves the utilization efficiency of communication resources.
本发明实施例提供一种基于Web RTC多方通话建立的设备1，如图6所示，该设备具体包括：The embodiment of the present invention provides a device 1 based on Web RTC multi-party call establishment, as shown in Figure 6, the device specifically includes:
第一接收单元11，用于接收正在通话的第一用户发送的多方通话建立请求，所述请求包括第一扩展消息，所述第一扩展消息中有待与所述第一用户建立多方通话的第二用户的信息；The first receiving unit 11 is configured to receive a multi-party call establishment request sent by the first user who is talking, the request includes a first extended message, and the first extended message is to establish a multi-party call with the first user. 2. User information;
权限判断单元12，用于判断所述第一用户建立所述多方通话的权限；An authority judgment unit 12, configured to determine the authority of the first user to establish the multi-party call;
第一消息发送单元13，用于当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，所述第三扩展消息中包括参加所述多方通话成员的列表信息；The first message sending unit 13 is configured to, when the first user has the authority to establish the multi-party call, send a second extended message to the first user confirming the establishment of the multi-party call, and communicate with the second user A third user who is talking with one user sends a third extended message, and the third extended message includes list information of members participating in the multi-party call;
第一请求发送单元14，用于向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息，并接收所述第二用户发送的确认加入的信息；The first request sending unit 14 is configured to send a request to the second user to join the multi-party call, the request includes list information of members participating in the multi-party call, and receive the joining confirmation sent by the second user Information;
第一通道建立单元15，用于分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道；The first channel establishment unit 15 is configured to establish media channels for multi-party calls between the first user and the second user, between the third user and the second user, respectively;
第一多方通话单元16，用于通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话。The first multi-party call unit 16 is configured to conduct the multi-party call through the established media channel for the multi-party call.
在设备1中，所述多方通话至少包括音频流和视频流的传输。In device 1, the multi-party call at least includes the transmission of audio streams and video streams.
进一步的，所述多方通话至少包括音频流和视频流的传输。Further, the multi-party call at least includes the transmission of audio streams and video streams.
当所述多方通话基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。When the multiparty call is based on Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second extended message, and the third extended message are extended messages based on the SIP.
所述第二用户为至少一个用户终端。The second user is at least one user terminal.
本发明实施例提供一种基于Web RTC多方通话建立的设备，该设备接收第一用户发送的多方通话建立请求，并判断第一用户建立多方通话的权限，当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息，并接收所述第二用户发送的确认加入的信息，分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道，通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。An embodiment of the present invention provides a device for establishing a multi-party call based on Web RTC. The device receives a request for establishing a multi-party call sent by a first user, and judges the authority of the first user to establish a multi-party call. When the multi-party call permission is granted, send a second extended message to the first user to confirm the establishment of the multi-party call, and send a third extended message to the third user who is talking with the first user, and send a third extended message to the second user The user sends a request to join the multi-party call, the request includes the list information of the members participating in the multi-party call, and receives the joining confirmation information sent by the second user, and the first user and the second Between the two users, the third user and the second user establish a media channel for multi-party conversation, and conduct the multi-party conversation through the established media channel for multi-party conversation; Due to the high performance requirements of the equipment caused by the sound mixing, there is no need to contact the site server to apply for cumbersome site resources, which further saves the waste of site resources when the multi-party call is restored to a two-party call, and finally reduces the establishment of the multi-party call. , improving the usage efficiency of communication resources.
本发明实施例还提供一种基于Web RTC多方通话建立的装置2，如图7所示，该装置2包括：总线21；以及连接到总线21上的存储器22、处理器23、接收器24和发射器25，其中存储器22用于存储相关指令，该处理器23执行该指令用于接收正在通话的第一用户发送的多方通话建立请求，所述请求包括第一扩展消息，所述第一扩展消息中有待与所述第一用户建立多方通话的第二用户的信息；该处理器23执行相关指令还用于判断所述第一用户建立所述多方通话的权限；该处理器23执行相关指令还用于当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，所述第三扩展消息中包括参加所述多方通话成员的列表信息；该处理器23执行相关指令还用于向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息；该处理器23执行相关指令还用于接收所述第二用户发送的确认加入的信息；该处理器23执行相关指令还用于分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道；该处理器23执行相关指令还用于通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话。The embodiment of the present invention also provides a device 2 based on Web RTC multi-party call establishment, as shown in Figure 7, the device 2 includes: a bus 21; and a memory 22, a processor 23, a receiver 24 and The transmitter 25, wherein the memory 22 is used to store related instructions, and the processor 23 executes the instructions to receive a multi-party call establishment request sent by the first user who is talking, the request includes a first extended message, and the first extended The information of the second user who is to establish a multi-party call with the first user in the message; the processor 23 executes related instructions and is also used to determine the authority of the first user to establish the multi-party call; the processor 23 executes related instructions It is also used to send a second extended message to the first user to confirm the establishment of the multi-party call when the first user has the authority to establish the multi-party call, and send a second extended message to the first user who is talking with the first user The third user sends a third extended message, the third extended message includes list information of members participating in the multi-party call; the processor 23 executes related instructions and is also used to send a request to the second user to join the multi-party call , the request includes the list information of the members participating in the multi-party call; the processor 23 executes related instructions and is also used to receive the information for confirming joining sent by the second user; the processor 23 executes related instructions and is also used to respectively Establish a media channel for a multi-party conversation between the first user and the second user, and between the third user and the second user; the processor 23 executes related instructions and is also used to pass the established The media channel used for the multi-party call is used to perform the multi-party call.
在本发明实施例中，可选的，该处理器23执行相关指令进行的多方通话至少包括音频流和视频流的传输。In the embodiment of the present invention, optionally, the multi-party call performed by the processor 23 executing related instructions includes at least the transmission of audio streams and video streams.
在本发明实施例中，可选的，该处理器23执行相关指令进行的多方通话基于会话发起协议（Session Initiation Protocol，SIP）时，所述第一扩展消息、所述第二扩展消息、所述第三扩展消息为基于所述SIP的扩展消息。In this embodiment of the present invention, optionally, when the processor 23 executes related instructions and the multi-party call is based on the Session Initiation Protocol (Session Initiation Protocol, SIP), the first extended message, the second extended message, the The third extended message is an extended message based on the SIP.
在本发明实施例中，可选的，该处理器23执行相关指令进行多方通话中的第二用户为至少一个用户终端。In the embodiment of the present invention, optionally, the processor 23 executes related instructions and the second user in the multi-party call is at least one user terminal.
本发明实施例提供一种基于Web RTC多方通话建立的装置，该设备接收第一用户发送的多方通话建立请求，并判断第一用户建立多方通话的权限，当所述第一用户具有建立所述多方通话的权限时，向所述第一用户发送确认建立所述多方通话的第二扩展消息，并向与所述第一用户正在通话的第三用户发送第三扩展消息，向所述第二用户发送加入所述多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息，并接收所述第二用户发送的确认加入的信息，分别在所述第一用户与所述第二用户间、所述第三用户与所述第二用户间建立用于多方通话的媒体通道，通过已经建立的所述用于多方通话的媒体通道，进行所述多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。An embodiment of the present invention provides a device for establishing a multi-party call based on Web RTC. The device receives a request for establishing a multi-party call sent by a first user, and judges the authority of the first user to establish a multi-party call. When the multi-party call permission is granted, send a second extended message to the first user to confirm the establishment of the multi-party call, and send a third extended message to the third user who is talking with the first user, and send a third extended message to the second user The user sends a request to join the multi-party call, the request includes the list information of the members participating in the multi-party call, and receives the joining confirmation information sent by the second user, and the first user and the second Between the two users, the third user and the second user establish a media channel for multi-party conversation, and conduct the multi-party conversation through the established media channel for multi-party conversation; Due to the high performance requirements of the equipment caused by the sound mixing, there is no need to contact the site server to apply for cumbersome site resources, which further saves the waste of site resources when the multi-party call is restored to a two-party call, and finally reduces the establishment of the multi-party call. , improving the usage efficiency of communication resources.
本发明实施例提供一种基于Web RTC多方通话建立的系统3，如图8所示，该系统3至少包括：The embodiment of the present invention provides a system 3 based on Web RTC multi-party call establishment, as shown in Figure 8, the system 3 at least includes:
如上述实施例中设备1所示的会议应用服务器，或如上述实施例装置2所述的会议应用服务器；A conference application server as shown in device 1 in the above embodiment, or a conference application server as described in device 2 in the above embodiment;
如上述实施例中所示的第一用户。The first user as shown in the above example.
本发明实施例提供一种基于Web RTC多方通话建立的系统，通过接收应用服务器发送的加入多方通话的请求，所述请求中包括参加所述多方通话成员的列表信息，在确认所述多方通话请求的正确性后，向所述应用服务器发送确认加入所述多方通话的信息，建立与第一用户的用于多方通话的媒体通道，最终通过所述媒体通道进行多方通话；能够降低由于进行本地混音造成的对设备性能较高的要求，还可以无需联系会场服务器进行繁琐的会场资源申请，进一步节省当多方通话恢复成双方通话时造成的会场资源的浪费，最终减少了多方通话的建立步骤，提高了通信资源的使用效率。The embodiment of the present invention provides a system for establishing a multi-party call based on Web RTC. By receiving a request for joining a multi-party call sent by an application server, the request includes list information of members participating in the multi-party call, and confirming the multi-party call request After confirming the correctness of the multi-party call, send to the application server the information confirming to join the multi-party call, establish a media channel for the multi-party call with the first user, and finally conduct a multi-party call through the media channel; The high performance requirements of the equipment caused by the voice can also avoid the need to contact the site server to apply for cumbersome site resources, further saving the waste of site resources when the multi-party call is restored to a two-party call, and finally reducing the establishment of the multi-party call. The utilization efficiency of communication resources is improved.
在本申请所提供的几个实施例中，应该理解到，所揭露的方法，装置，和系统，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed method, device, and system can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理包括，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机设备（可以是个人计算机，服务器，或者网络设备等）执行本发明各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器（Read-Only Memory，简称ROM）、随机存取存储器（Random Access Memory，简称RAM）、磁碟或者光盘等各种可以存储程序代码的介质。The above-mentioned integrated units implemented in the form of software functional units may be stored in a computer-readable storage medium. The above-mentioned software functional units are stored in a storage medium, and include several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM for short), random access memory (Random Access Memory, RAM for short), magnetic disk or optical disk, etc., which can store program codes. medium.
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以所述权利要求的保护范围为准。The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410081884.8ACN104902111B (en) | 2014-03-06 | 2014-03-06 | A kind of method, apparatus and system established based on Web RTC multi-party call |
| PCT/CN2015/072829WO2015131750A1 (en) | 2014-03-06 | 2015-02-12 | Method, device and system for establishing multi-party call based on web rtc |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410081884.8ACN104902111B (en) | 2014-03-06 | 2014-03-06 | A kind of method, apparatus and system established based on Web RTC multi-party call |
| Publication Number | Publication Date |
|---|---|
| CN104902111Atrue CN104902111A (en) | 2015-09-09 |
| CN104902111B CN104902111B (en) | 2019-02-01 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410081884.8AActiveCN104902111B (en) | 2014-03-06 | 2014-03-06 | A kind of method, apparatus and system established based on Web RTC multi-party call |
| Country | Link |
|---|---|
| CN (1) | CN104902111B (en) |
| WO (1) | WO2015131750A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105743889A (en)* | 2016-01-27 | 2016-07-06 | 福建星网智慧科技股份有限公司 | Method and system for realizing multi-party audio call based on webrtc (Web Real-Time Communication) |
| CN105915521A (en)* | 2016-04-18 | 2016-08-31 | 北京小米移动软件有限公司 | Multi-party communication management method, device and terminal |
| CN107682657A (en)* | 2017-09-13 | 2018-02-09 | 中山市华南理工大学现代产业技术研究院 | WebRTC-based multi-user voice video call method and system |
| CN108270584A (en)* | 2016-12-30 | 2018-07-10 | 展讯通信（上海）有限公司 | Realize the method, apparatus of conference telephone capabilities and mostly logical terminal |
| CN109660491A (en)* | 2017-10-10 | 2019-04-19 | 中国移动通信有限公司研究院 | A kind of point-to-point call method and device, equipment, storage medium in many ways |
| CN110401623A (en)* | 2018-04-25 | 2019-11-01 | 中国移动通信有限公司研究院 | A multi-party call method, platform, terminal, medium, device and system |
| CN112019791A (en)* | 2019-05-30 | 2020-12-01 | 广州云积软件技术有限公司 | Multi-party audio and video call method and system based on education examination |
| CN114710461A (en)* | 2022-03-31 | 2022-07-05 | 中煤科工集团重庆智慧城市科技研究院有限公司 | Multi-terminal audio and video instant messaging method and system |
| CN116094799A (en)* | 2023-01-06 | 2023-05-09 | 以萨技术股份有限公司 | Communication method and device based on multi-device fusion |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852081A (en)* | 2005-07-12 | 2006-10-25 | 华为技术有限公司 | Method for realizing muti-part meeting through uest generation network |
| CN1933482A (en)* | 2005-09-16 | 2007-03-21 | 腾讯科技（深圳）有限公司 | Method for initiating speech conversation |
| WO2007085525A1 (en)* | 2006-01-27 | 2007-08-02 | Nokia Siemens Networks Gmbh & Co. Kg | Method for communicating with several users, arrangement, communication management server, and communication terminal |
| CN101106536A (en)* | 2006-07-15 | 2008-01-16 | 华为技术有限公司 | A method of establishing a group conversation |
| CN101237336A (en)* | 2007-02-01 | 2008-08-06 | 华为技术有限公司 | Method, system and device for multi-party communication and method for publishing event status |
| CN101547107A (en)* | 2008-03-27 | 2009-09-30 | 天津德智科技有限公司 | Method and device for establishing multi-channel point-to-point connection |
| CN102571758A (en)* | 2011-12-16 | 2012-07-11 | 华为技术有限公司 | Method and device for realizing seamless transfer of two-party call transfer conference |
| CN102625080A (en)* | 2012-04-23 | 2012-08-01 | 广东大晋对接信息科技有限公司 | P2P-based WEB video conference system |
| CN103404132A (en)* | 2013-03-08 | 2013-11-20 | 华为终端有限公司 | Video communication method, home terminal, and home server |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852081A (en)* | 2005-07-12 | 2006-10-25 | 华为技术有限公司 | Method for realizing muti-part meeting through uest generation network |
| CN1933482A (en)* | 2005-09-16 | 2007-03-21 | 腾讯科技（深圳）有限公司 | Method for initiating speech conversation |
| WO2007085525A1 (en)* | 2006-01-27 | 2007-08-02 | Nokia Siemens Networks Gmbh & Co. Kg | Method for communicating with several users, arrangement, communication management server, and communication terminal |
| CN101106536A (en)* | 2006-07-15 | 2008-01-16 | 华为技术有限公司 | A method of establishing a group conversation |
| CN101237336A (en)* | 2007-02-01 | 2008-08-06 | 华为技术有限公司 | Method, system and device for multi-party communication and method for publishing event status |
| CN101547107A (en)* | 2008-03-27 | 2009-09-30 | 天津德智科技有限公司 | Method and device for establishing multi-channel point-to-point connection |
| CN102571758A (en)* | 2011-12-16 | 2012-07-11 | 华为技术有限公司 | Method and device for realizing seamless transfer of two-party call transfer conference |
| CN102625080A (en)* | 2012-04-23 | 2012-08-01 | 广东大晋对接信息科技有限公司 | P2P-based WEB video conference system |
| CN103404132A (en)* | 2013-03-08 | 2013-11-20 | 华为终端有限公司 | Video communication method, home terminal, and home server |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105743889A (en)* | 2016-01-27 | 2016-07-06 | 福建星网智慧科技股份有限公司 | Method and system for realizing multi-party audio call based on webrtc (Web Real-Time Communication) |
| CN105743889B (en)* | 2016-01-27 | 2019-05-17 | 福建星网智慧科技股份有限公司 | A kind of method and system for realizing multi-party audio call based on webrtc |
| CN105915521A (en)* | 2016-04-18 | 2016-08-31 | 北京小米移动软件有限公司 | Multi-party communication management method, device and terminal |
| CN108270584A (en)* | 2016-12-30 | 2018-07-10 | 展讯通信（上海）有限公司 | Realize the method, apparatus of conference telephone capabilities and mostly logical terminal |
| CN107682657A (en)* | 2017-09-13 | 2018-02-09 | 中山市华南理工大学现代产业技术研究院 | WebRTC-based multi-user voice video call method and system |
| CN107682657B (en)* | 2017-09-13 | 2020-11-10 | 中山市华南理工大学现代产业技术研究院 | A method and system for multi-person voice and video call based on WebRTC |
| CN109660491A (en)* | 2017-10-10 | 2019-04-19 | 中国移动通信有限公司研究院 | A kind of point-to-point call method and device, equipment, storage medium in many ways |
| CN110401623A (en)* | 2018-04-25 | 2019-11-01 | 中国移动通信有限公司研究院 | A multi-party call method, platform, terminal, medium, device and system |
| CN112019791A (en)* | 2019-05-30 | 2020-12-01 | 广州云积软件技术有限公司 | Multi-party audio and video call method and system based on education examination |
| CN114710461A (en)* | 2022-03-31 | 2022-07-05 | 中煤科工集团重庆智慧城市科技研究院有限公司 | Multi-terminal audio and video instant messaging method and system |
| CN114710461B (en)* | 2022-03-31 | 2024-03-12 | 中煤科工集团重庆智慧城市科技研究院有限公司 | Multi-terminal audio and video instant messaging method and system |
| CN116094799A (en)* | 2023-01-06 | 2023-05-09 | 以萨技术股份有限公司 | Communication method and device based on multi-device fusion |
| Publication number | Publication date |
|---|---|
| WO2015131750A1 (en) | 2015-09-11 |
| CN104902111B (en) | 2019-02-01 |
| Publication | Publication Date | Title |
|---|---|---|
| CN113746808B (en) | Converged communication method, gateway, electronic equipment and storage medium for online conference | |
| WO2015131750A1 (en) | Method, device and system for establishing multi-party call based on web rtc | |
| US12355830B2 (en) | Transferring a phone call into a video conferencing session | |
| CN107682657B (en) | A method and system for multi-person voice and video call based on WebRTC | |
| CN102137080B (en) | Method, device and system for cross-platform conference convergence | |
| KR101571925B1 (en) | Multipoint conference device and switching method from multipoint conference to point-to-point communication | |
| CN103188300B (en) | The methods, devices and systems of VOIP phone are realized in cloud computing environment | |
| WO2017129129A1 (en) | Instant call method, device, and system | |
| WO2016150213A1 (en) | Data processing method in webpage-based real-time communication media and device utilizing same | |
| WO2016082577A1 (en) | Video conference processing method and device | |
| WO2016019775A1 (en) | Conference migration method, device and system | |
| CN112887271A (en) | Method, system, electronic device and storage medium for realizing instant conference | |
| JP7463552B2 (en) | SESSION CREATION METHOD, ELECTRONIC DEVICE, AND READABLE STORAGE MEDIUM | |
| CN106549978B (en) | Session mode switching method and proxy server | |
| KR101589195B1 (en) | METHOD AND APPARATUS FOR SEAMLESSlY IMPLEMENTING TRNASFERRING DUAL-PARTY CALL INTO CONFERENCE | |
| CN110460603A (en) | Multimedia file transmission method, terminal, server, system and storage medium | |
| RU2573268C2 (en) | Method and apparatus for controlling multimedia conference | |
| CN107666396B (en) | Multi-terminal conference processing method and device | |
| US20100228832A1 (en) | Method, apparatus and system for creating and operating conferences | |
| CN102291366A (en) | Method for realizing instant messaging of multi-media conference and user equipment | |
| CN117880422A (en) | Audio telephone video capability expanding method, service system, equipment and storage medium | |
| CN114125362B (en) | Conference joining method, device, conference platform and computer readable storage medium | |
| CN102196106B (en) | Method and related equipment for realizing call between calling party and called party | |
| CN115604045A (en) | Online conference fusion method and device and computer storage medium | |
| CN110505070A (en) | Method and device for establishing a three-party session |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |