Summary of the invention
For above-mentioned defect, the object of the present invention is to provide the implementation method of a kind of mobile terminal and multi-party video calls, can amplify display client video information according to user's request, improve user and experience.
To achieve these goals, the invention provides a kind of implementation method of multi-party video calls, described method comprises:
The display frame of multi-party video is arranged to He Fu viewing area, main viewing area;
After multi-party video calls starts, receive the video information of service end transmission and be shown in described main viewing area and/or secondary viewing area, the video information of described main viewing area is the video information after amplifying processing of particular clients;
According to operational order, by a client hand-off process of the client of described main viewing area and described secondary viewing area.
According to the implementation method of multi-party video calls of the present invention, described method also comprises: receive user instruction, select the display mode of described multi-party video calls, described display mode comprises automatic preference pattern and the manual preference pattern of described main viewing area video information.
According to the implementation method of multi-party video calls of the present invention, the display mode of described multi-party video calls is manual preference pattern, described according to operational order, and a client hand-off process step of the client of described main viewing area and described secondary viewing area is comprised:
To service end, send the handover request instruction of the first client of described secondary viewing area; Receive the video information of service end to described the first client process, and be shown to described main viewing area, the video information of the client of described main viewing area is shown to described secondary viewing area.
According to the implementation method of multi-party video calls of the present invention, the display mode of described multi-party video calls is automatic preference pattern, described according to operational order, and a client hand-off process step of the client of described main viewing area and described secondary viewing area is comprised:
To service end, send and say speech phase, described service end detects whether only have the first client of described secondary viewing area in saying speech phase, if, receive the video information of service end to described the first client process, and be shown to described main viewing area, the video information of the client of described main viewing area is shown to described secondary viewing area.
According to the implementation method of multi-party video calls of the present invention, described reception user instruction, select the step of the display mode of described multi-party video calls to comprise:
The current display mode of described multi-party video calls is automatic preference pattern, and, after the client selection instruction of described secondary viewing area, the display mode of described multi-party video calls switches to manual preference pattern.
The present invention also provides a kind of mobile terminal, comprising:
Region division module, for arranging the display frame of multi-party video He Fu viewing area, main viewing area;
Graphics Processing module, after starting, receives the video information of service end transmission and is shown in He Fu viewing area, described main viewing area for multi-party video calls, and the video information of described main viewing area is the video information after amplifying processing of particular clients;
Video frequency switching module, for according to operational order, by a client hand-off process of the client of described main viewing area and described secondary viewing area.
According to mobile terminal of the present invention, described mobile terminal also comprises mode selection module, be used for receiving user instruction, select the display mode of described multi-party video calls, described display mode comprises automatic preference pattern and the manual preference pattern of described main viewing area video information.
According to mobile terminal of the present invention, the display mode of described multi-party video calls is manual preference pattern;
Described video frequency switching module comprises the first transmission submodule, for send the handover request instruction of the first client of described secondary viewing area to service end;
Described Graphics Processing module is further used for receiving the video information of described service end to described the first client process, and processes and displays is to described main viewing area, and the video information process of the client of described main viewing area is shown to described secondary viewing area.
According to mobile terminal of the present invention, the display mode of described multi-party video calls is automatic preference pattern;
Described video frequency switching module comprises:
Second sends submodule, for sending and say speech phase to service end;
Described Graphics Processing module is further used for described service end and detects and only have the first client of described secondary viewing area when saying speech phase, receive the video information of service end to described the first client process, and be shown to described main viewing area, the video information of the client of described main viewing area is shown to described secondary viewing area.
According to mobile terminal of the present invention, if the current display mode of described multi-party video calls is automatic preference pattern, the client selection instruction of described secondary viewing area receive that, after the response of service end, described mode selection module switches to manual preference pattern by the display mode of described multi-party video calls.
The present invention is by least arranging the display interface of mobile terminal multi-party video He Fu viewing area, main viewing area, mobile terminal starts after multi-party video calls, after being amplified to processing, the video information of given client end is shown in main viewing area, the video information of other client is shown in to secondary viewing area, if client is more, can slides and browse the client of participation in secondary viewing area.Further, mobile terminal can also make main viewing area amplify the client that shows that user pays close attention to one client hand-off process of the client of described main viewing area and described secondary viewing area whereby, improves user and experiences.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.
Fig. 2 is the mobile terminal structure schematic diagram of one embodiment of the invention.For ease of describing, this Figure only shows part related to the present invention.This mobile terminal 100 can be mobile phone, PDA(Personal Digital Assistant, personal digital assistant), palmtop PC etc., and mobile terminal 100 comprises region division module 10, Graphics Processing module 20 and video frequency switching module 30.Concrete:
Region division module 10 is for arranging the display frame of multi-party video He Fu viewing area, main viewing area.
In conjunction with Fig. 3, main viewing area only shows the video information of a client in multi-party video calls, and the video information of other client is all shown in secondary viewing area.If the client of multi-party video calls is more, user can browse by slip the client of secondary viewing area.
After Graphics Processing module 20 starts for multi-party video calls, receive the video information of service end transmission and be shown in described main viewing area and/or secondary viewing area, the video information of described main viewing area is the video information after amplifying processing of particular clients.
After multi-party video calls starts, the video information of other client that mobile terminal 100 reception service ends are sent.Because the demonstration size differences of He Fu viewing area, main viewing area is larger, therefore, the present invention processed setting by service end to the parameter of video information before showing.Such as, the video information (picture) that is shown in the client of main viewing area is amplified, and guarantee that picture is clear, also according to the display parameters of secondary viewing area, the video information of other client is done to adaptive processing simultaneously.
Video frequency switching module 30, for according to operational order, by a client hand-off process of the client of described main viewing area and described secondary viewing area.
In practical application, user can select the client of oneself paying close attention to, and is shown in main viewing area using it as key frame.If user need to be switched the video information of main viewing area, can directly click certain client in secondary viewing area, mobile terminal 100 carries out hand-off process in real time, meets consumers' demand whereby.
Concrete, then referring to Fig. 3, the video information of main viewing area can also carry label and be shown in a-quadrant, for indicating the name of key frame, such as cyber port, lake, Guia Hill and so on, makes user can understand this source video sequence.Preferably, the present invention also arranges a little display window C in the lower right corner of main viewing area, for showing the window of local preview, for user provides local video picture.Corresponding, the F in secondary viewing area is client video information, the corresponding label that shows each client in E district, and it is consistent with the effect of main viewing area label, shows the name information of the client of secondary viewing area, prompting user.
Fig. 4 is the mobile terminal structure schematic diagram of another embodiment of the present invention.In this embodiment, mobile terminal 100 also comprises mode selection module 40, for receiving user instruction, selects the display mode of described multi-party video calls, and described display mode comprises automatic preference pattern and the manual preference pattern of described main viewing area video information.Described automatic preference pattern is the client that mobile terminal 100 selects main viewing area to show automatically, and manually preference pattern is the client that user selects main viewing area.
Described video frequency switching module 30 comprises that the first transmission submodule 31 and second sends submodule 32.
Concrete, if the display mode of mobile terminal 100 current multi-party video calls is manual preference pattern, after the first client user's of secondary viewing area selection instruction, first sends submodule 31 sends the handover request instruction of the first client of described secondary viewing area to service end, service end is carried out hand-off process after receiving this instruction, and the video information after processing is mail to mobile terminal 100.Described Graphics Processing module 20 receives the video information of service end to described the first client process, and processes and displays is to described main viewing area, and the video information process of the client of described main viewing area is shown to described secondary viewing area.Need to illustrate, the first described client only refers to the client that user selects, and it can be arbitrary client of secondary viewing area.
If the display mode of mobile terminal 100 current multi-party video calls is automatic preference pattern, mobile terminal 100 and other client are sent and say speech phase to service end by the second transmission submodule 32, service end detects the client of current transmission voice messaging, if there is no voice messaging or a plurality of voice messaging, wouldn't deal with, if only have a client (such as the first client of secondary viewing area) in saying speech phase, service end is processed the video information of the first client, mail to mobile terminal 100, and by Graphics Processing module 20 processes and displays to main viewing area, the video information process of the former client of described main viewing area is shown to described secondary viewing area.Concrete, for client, the client of talking can send the order of talking to service end, and this order is whether client is talked and automatically sent according to user.Service end receives after the order of talking that client sends first check whether the speech sign of service end is empty, empty if, determines in current multi-party conversation and there is no other client in speech.This URI of URI(that service end just can be set to talker's (the first client) speech sign carries in the order of talking), the video flowing then the first client being sent is switched to automatic mode user's main video flowing, completes automatic switchover.If service end is received while talking order, check the speech identification discovery non-NULL of service end, illustrate and now have other users in speech, can not complete switching, make the demonstration of mobile terminal 100 keep original state.
In application of the present invention, if the current video mode of mobile terminal 100 is manual preference pattern, this mobile terminal can show a switching push button G, in conjunction with Fig. 3, user presses this button, the display mode of multi-party video can be switched to automatic preference pattern by manual preference pattern, under automatic preference pattern, picture no longer shows switching push button G.
Preferably, if the automatic preference pattern of display mode that described multi-party video calls is current, the client of secondary viewing area can receive user's selection instruction, and after the response of receiving service end, described mode selection module 40 switches to manual preference pattern by the display mode of described multi-party video calls automatically.When user clicks a window of secondary viewing area, obtain the source of media of this window and operation user's URI, and send switching command to service end, after handover success, present mode is set to manual mode, and more new window mark, recalls automatic mode switching push button G.Described window mark is used for the window that mark is play, consistent with the SIP URI of source of media.
In concrete application, mobile terminal 100 can be selected to send order to service end and sets up a new multi-party video calls or add already present multi-party video calls, when MPTY starts, other clients can be using first participant's video as key frame, first participant is using second participant's video as key frame, after switch as requested again.After adding two users, often add a new user, all can increase a display window in secondary viewing area, show the user's who newly adds video information and label information.
In the process of call, when user clicks certain secondary picture, client will send switching command to service end, and video flowing corresponding to this pair picture is switched to corresponding main video flowing, the video flowing of original main viewing area is switched to secondary viewing area and shows, referring to Fig. 5 A and Fig. 5 B.
In embodiments of the invention, if mobile terminal 100 is in automatic display mode, when certain user M speech, user M sends speech order, if now there is no other people in speech, service end mails to the video of M the main video flowing (being shown in main viewing area) of other client in automatic mode, if now also have other users in speech, do not switch, this user's video is still in secondary viewing area picture disply, until there is no other user's speeches, when user M speech finishes, send speech the finish command.Need explanation, when mobile terminal 100 is during in automatic display mode, send speech order and ring off order all by client according to whether talking and automatically send, do not need manual intervention.With Beijing Research Center speech, automatically switch to example, as shown in Fig. 5 C and Fig. 5 D.
When certain client (supposing user N) exits many people video calling or disconnects with service end, service end is closed the video flowing relevant with client N, comprises in other users and the related video flowing of client N.For other clients, if the video flowing of client N is the video flowing of secondary picture, client is closed a secondary picture disply window and its corresponding label, if main screen video stream, select a secondary picture video stream to show as key frame, then close this secondary picture video stream, client is also closed the corresponding display window of this pair viewing area.
The present invention is by arranging display frame, the picture that outstanding user need to pay close attention to, weaken non-concern picture, effectively avoid dividing equally the less problem seen clearly of being even difficult to of the picture bringing viewing area, and increased manually and automatic switchover user pays close attention to the function of picture, make user can switch manually or automatically the picture of paying close attention to, user-friendly, effectively improve user and experience.
In another embodiment of the present invention, when user opens application for the first time, user's name need to be set, this name will send to other users when video calling, and the source of reminding other these videos of user also can not arrange or arrange in follow-up call.Meanwhile, user user before initiating video calling can arrange own media parameter, and such as resolution, the information such as coded format, can not arrange yet, and Uses Defaults or the value of setting last time.After setting completed, user can initiatively create new many people video calling, also can join in already present video calling, start after video calling, mobile terminal 100 can receive the video and/or audio information of other client of service end forwarding, and shows corresponding client-side information according to being arranged on He Fu viewing area, main viewing area.Preferably, in the present invention the control information of client and service end alternatively based on SIP and SDP agreement, the transmission of media information is based on RTP/RTCP agreement.
Mobile terminal 100 of the present invention control to secondary viewing area in multi-party video calls is as follows: in multi-party video calls, have new client to add fashionable, mobile terminal 100 is a newly-built display window in secondary viewing area.Newly-built window and the label of this window have a unique mark, and because each participant's SIP URI is unique, so this mark serves as a mark with video source user's SIP URI, this URI carries with sip message in the time of can setting up Media Stream by service end.Meanwhile, the CNAME(Canonical NAME in RTCP Source Description (SDES) bag) be used for demarcating source of media, also require uniquely, therefore also use this SIP URI.Between SIP URI, secondary picture window, Media Stream, set up whereby a corresponding relation.This mark can be changed, when data flow come source user to change time, this mark should change user's SIP URI afterwards into.When closing because of Media Stream, window corresponding to convenient searching closed.If mobile terminal 100 is also received the label data of client, to its Graphics Processing.Described label data is to obtain in the NAME field Source Description (SDES) the RTCP bag from corresponding video flowing, can a name be intuitively set by user oneself, only plays suggesting effect.
The control of 100 pairs of call managements of mobile terminal of the present invention is as follows: first judge that user sets up a multi-party video calls or adds a multi-party video calls, if user successfully creates a multi-party video calls, can a corresponding multi-party video calls address, user calls out this address, by just can initiatively joining in multi-party video calls after authentication, thereby user is also by being associated with this address contact and service end.If user adds multi-party video calls, consult to set up Media Stream, and the decoder of Media Stream broadcast window is set according to negotiation result.Make whereby client obtain will to arrive the media parameter of Media Stream, guarantee that the Media Stream after switching can normal play.If there is client to exit multi-party video calls, mobile terminal 100 is closed corresponding window.
The processing procedure of detection voice messaging of the present invention is as follows: whether detect user in speech, if user is talking, user just sends to service end the order of talking, and to service end, shows that this participant talks.The frequency that the present invention preferably sends once according to every several seconds (2～3s) sends, every several seconds, detect once, if active user is speaking, and speech is designated and is not to say speech phase, just send speech order, if find not speech of user, and speech sign is to say speech phase, sends the order that rings off.Speech order includes talker's SIP URI, and service end determines with this who,, in speech, rings off order too.When user is in halted state, if this user is successfully stopped, no longer sends and cease and desist order.
In embodiments of the invention, when mobile terminal 100 receives voice data, play, judge whether that the voice data encoding needs to send simultaneously, if had, send voice data.If receive, user exits command, and closes the transmission of voice data.
Preferably, if the video calling user of mobile terminal of the present invention 100 only has two, secondary viewing area does not create display window, only by main viewing area and the preview window, can show.If there are three or three above users in MPTY, can set up in secondary viewing area corresponding client window and corresponding Media Stream.Mobile terminal 100 can read NAME field and obtain the name tags that user arranges from RTCP SDES bag corresponding to Media Stream, if got, name is mail to corresponding label area and show.
The present invention completes alternately based on client and service end, service end mainly completes and the function such as mutual, the Media Stream foundation of client-requested, media processing, forwarding, it is concrete, and to realize handling process corresponding with the processing of mobile terminal 100, therefore do not repeat them here.
Fig. 6 is the implementation method flow chart of the multi-party video calls of one embodiment of the invention, and it can be realized by mobile terminal 100 as shown in Figure 2, and the method comprises:
Step S601, arranges He Fu viewing area, main viewing area by the display frame of multi-party video.
In concrete application, main viewing area only shows the video information of a client in multi-party video calls, and the video information of other client is all shown in secondary viewing area.If the client of multi-party video calls is more, user can browse by slip the client of secondary viewing area.
Step S602, after multi-party video calls starts, receives the video information of service end transmission and is shown in described main viewing area and/or secondary viewing area, and the video information of described main viewing area is the video information after amplifying processing of particular clients.
After multi-party video calls starts, the video information of other client that mobile terminal 100 reception service ends are sent.Because the demonstration size differences of He Fu viewing area, main viewing area is larger, therefore, the present invention processed setting by service end to the parameter of video information before showing.Such as, the video information (picture) that is shown in the client of main viewing area is amplified, and guarantee that picture is clear, concrete adjustable picture frame, resolution etc., also do adaptive processing according to the display parameters of secondary viewing area to the video information of other client simultaneously.
Step S603, according to operational order, by a client hand-off process of the client of described main viewing area and described secondary viewing area.
In practical application, user can select the client of oneself paying close attention to, and is shown in main viewing area using it as key frame.If user need to be switched the video information of main viewing area, can directly click certain client in secondary viewing area, mobile terminal 100 carries out hand-off process in real time, meets consumers' demand whereby.
Concrete, then referring to Fig. 3, the video information of main viewing area can also carry label and be shown in a-quadrant, for indicating the name of key frame, such as cyber port, lake, Guia Hill and so on, makes user can understand this source video sequence.Preferably, the present invention also arranges a little display window C in the lower right corner of main viewing area, for showing the window of local preview, for user provides local video picture.Corresponding, the F in secondary viewing area is client video information, the corresponding label that shows each client in E district, and it is consistent with the effect of main viewing area label, shows the name information of the client of secondary viewing area, prompting user.
Fig. 7 is the implementation method flow chart of the multi-party video calls of another embodiment of the present invention, and it can be realized by mobile terminal 100 as shown in Figure 4, and the method comprises:
Step S701, arranges He Fu viewing area, main viewing area by the display frame of multi-party video.
Step S702, creates or adds a multi-party video calls.
Step S703, receives the video information of service end transmission and is shown in described main viewing area and/or secondary viewing area, and the video information of described main viewing area is the video information after amplifying processing of particular clients.It should be noted that, if video calling user is only two people, the video information of a client is shown in main viewing area, and another client-side information is shown in local the preview window, referring to Fig. 3.
Step S704, receives user instruction, selects the display mode of described multi-party video calls, and described display mode comprises automatic preference pattern and the manual preference pattern of described main viewing area video information.If select manual preference pattern, perform step S705, otherwise execution step S706.
Step S705, after the first client user's of secondary viewing area selection instruction, first sends submodule 31 sends the handover request instruction of the first client of described secondary viewing area to service end, service end is carried out hand-off process after receiving this instruction, and the video information after processing is mail to mobile terminal 100, execution step S707.。
Step S706, to service end, send and say speech phase, service end detects the client of current transmission voice messaging, if there is no voice messaging or a plurality of voice messaging, wouldn't deal with, if only there is a client (such as the first client of secondary viewing area), in saying speech phase, service end is processed the video information of the first client, mail to mobile terminal 100, and perform step S707.Concrete, for client, the client of talking can send the order of talking to service end, and this order is whether client is talked and automatically sent according to user.Service end receives after the order of talking that client sends first check whether the speech sign of service end is empty, empty if, determines in current multi-party conversation and there is no other client in speech.This URI of URI(that service end just can be set to talker's (the first client) speech sign carries in the order of talking), the video flowing then the first client being sent is switched to automatic mode user's main video flowing, completes automatic switchover.If service end is received while talking order, check the speech identification discovery non-NULL of service end, illustrate and now have other users in speech, can not complete switching, make the demonstration of mobile terminal 100 keep original state.
Step S707, is shown to described main viewing area by the video information process of described the first client, and the video information process of the former client of described main viewing area is shown to described secondary viewing area.
In application of the present invention, if the current video mode of mobile terminal 100 is manual preference pattern, this mobile terminal can show a switching push button G, in conjunction with Fig. 3, user presses this button, the display mode of multi-party video can be switched to automatic preference pattern by manual preference pattern, under automatic preference pattern, picture no longer shows switching push button G.
Preferably, if the automatic preference pattern of display mode that described multi-party video calls is current, the client of secondary viewing area can receive user's selection instruction, and after receiving the handoff response of service end, described mobile terminal 100 switches to manual preference pattern by the display mode of described multi-party video calls automatically.
In sum, the present invention is by least arranging the display interface of mobile terminal multi-party video He Fu viewing area, main viewing area, mobile terminal starts after multi-party video calls, after being amplified to processing, the video information of given client end is shown in main viewing area, the video information of other client is shown in to secondary viewing area, if client is more, can slides and browse the client of participation in secondary viewing area.Further, mobile terminal can also make main viewing area amplify the client that shows that user pays close attention to one client hand-off process of the client of described main viewing area and described secondary viewing area whereby, improves user and experiences.
Certainly; the present invention also can have other various embodiments; in the situation that not deviating from spirit of the present invention and essence thereof; those of ordinary skill in the art are when making according to the present invention various corresponding changes and distortion, but these corresponding changes and distortion all should belong to the protection range of the appended claim of the present invention.