Background technique
The application range of video conferencing system is very extensive, can be applicable to the network teleconference, synergetic office work, online trainingThe various aspects such as instruction, tele-medicine, long-distance education, can be widely used in government, army, enterprise, IT, telecommunications, electric power, education,The every field such as medical treatment, security, finance, manufacture.In video conference, most important mode of the audio as reception and registration conferencing information,It is the information of highest priority, sound effect and fluency are all extremely important, therefore, real-time acquisition and the playing function of audioOptimization just seems further important.
There are many methods that can carry out audio collection and play in cell phone system；In terms of audio collection, such as:AVAudioRecorder, AVCaptureDevice, Audio Unit, Audio Queues.Audio plays aspect, such as:AVAudioPlayer, AVPlayer, AudioQueue, OpenAL, Audio Unit.Their each have their own advantage and disadvantage, specificallyAre as follows:
AVAudioRecorder: although usage is fairly simple, streaming is not supported；This means that: playing audioBefore, it is necessary to after the completion of the load of entire audio, it could start to play audio；When recording, it is also necessary to after End of Tape,Recording data can be got.This causes significant limitation to application.AVCaptureDevice: its usage is also simplerSingle, when supporting streaming, but it is that audio-video acquires together, therefore initialize video parameter, audio frequency parameter is also simultaneouslyBe configured with, if it is desired to be separately configured it is more troublesome if audio collection parameter, and if audio is wanted to stop acquisition,Video acquisition also and then stopped, and compares limitation, Audio Unit: being the interface of the bottom, support streaming, its advantages are functionCan be powerful, postpone low；And the disadvantage is that learning cost is high, difficulty is big.Audio Queues: streaming is supported, advantage can be by audioQueue cooperates the interface of other Core Audio to use, along with the custom code of relatively small amount can be in the applicationSolution, the height but its recording is delayed are recorded or played to the digital audio for creating complete set.
AVAudioPlayer: describing for it is high-efficient, substantially supports all audio formats, the control to broadcasting, such asLoop play, sound size, pause etc. are more convenient.Disadvantage: can be some more to the consumption of memory.Do not support streaming, and can notPlay Online Music AVPlayer: it is similar with AVAudioPlayer, and different places is can to support streaming.AudioQueue: it supports streaming, its advantages to be responsible for carrying out memory management to its buffer area using audio queue, optimizes moneyThe use in source.OpenAL: advantage: cross-platform solution can use under multiple platforms, support file and streaming broadcasting,But if time delay is high in terms of being used in real-time voice.Audio Unit: streaming broadcasting, low delay, the bottom, but itBroadcasting is played by a playbackCallback call back function, and the number called for this function one second is probably20-30 times or so.If the size of the data flow of the backstage broadcasting per second transmitted and number is more will generate oneA little abnormal problems.
Summary of the invention
The present invention provides a kind of audio collection of video conference for the problem of background technique and plays optimization sideMethod, device and medium solve echo and the high disadvantage of CPU usage.
To achieve the goals above, the present invention proposes a kind of audio collection of video conference and plays optimization method, includingFollowing steps:
The first readjustment of S10, setting audio collection parameter and one for being handled the audio digital signals of acquisitionFunction；
S20, when each sampled audio is converted into digital signal by analog quantity, call the first call back function to digital audioSignal is handled；
The the second readjustment letter of S30, setting audio play parameter and one for changing audio play buffer occupied stateNumber；
S40, when audio plays out, call the second call back function judgement distribution is carried out to audio buffer.
Preferably, setting audio collection parameter described in step S10, comprising:
S101, the example for obtaining an AVAudioSession class, are arranged AudioSession audio categories；
S102, customized the first graph of structural body comprising AUGraph and AudioUnit, and create and be directed toward the knotThe 2nd graph of AUGraph in structure body；
S103, it adds in the graph of Node to the 2nd of a Remote I/O type；
S104, the underlying parameter that audio collection is set.
Preferably, first call back function, is arranged audio processing operation according to actual needs, including but not limited to:Balanced, mixing and format conversion.
Preferably, the underlying parameter, including but not limited to: audio collection sample rate, the frame number of each data packet, languageThe occupancy digit of sound sampled point, the bytes number of every frame, monophonic or multichannel are arranged, the bytes of each data packet is total, everyThe bytes number of frame × each data packet frame number and echo cancellor setting.
Preferably, setting audio play parameter described in step S30, comprising:
S301, creation audio play object；
The underlying parameter that S302, setting audio play；
S303, setting internal thread play, and create output.
Preferably, the underlying parameter that the setting audio plays, including but not limited to: sample rate, data package size, languageThe every sampled point of sound occupies digit.
Preferably, second call back function of calling carries out judgement distribution to audio buffer, comprising:
S304, the occupied state for judging multiple buffering area；
S305, it audio data is sent to unappropriated buffer area plays out.
Preferably, the audio data format is pcm data.
The present invention also proposes a kind of audio collection of video conference and plays optimization device, comprising:
Processor；
Memory is coupled to the processor and is stored with instruction, and the instruction is executing reality by the processorThe step of audio collection and broadcasting optimization method of the existing video conference.
The present invention also proposes that a kind of computer-readable storage medium, the computer-readable storage medium are stored with viewThe audio collection of frequency meeting and the application program for playing optimization method, the application program realize the sound of video conference as mentionedThe step of frequency acquisition is with optimization method is played.
The present invention uses Audio Unit to carry out audio collection first, and Audio Queues carries out audio broadcasting, realizes soundThe function that frequency acquisition plays.But the requirement for video conference for audio is very high, and this scheme is there are two defects, and firstReflective problem once in a while, second, discovery CPU is occupied very high when exactly playing audio；For the two defects present inventionFurther optimized, propose a kind of audio collection of video conference and plays optimization method, device and medium, due toAudio Unit can not complete independently echo cancellor work, need to cooperate AUGraph come using；AUGraph is manager,Different Unit is added to as Node to be gone to play a role in AUGraph；Echo cancellor is carried out using AUGraph；For broadcastingWhen playback frequency, the high problem of CPU usage is the discovery that the audio data frequency transmitted due to server-side is too fast caused, notIt influences to greatly reduce the occupancy of CPU with sleep () function under the effect that audio plays in real time.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, completeSite preparation description, it is clear that described embodiment is only a part of the embodiments of the present invention, instead of all the embodiments.BaseEmbodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts it is all itsHis embodiment, shall fall within the protection scope of the present invention.
It is to be appreciated that if relating to directionality instruction (such as up, down, left, right, before and after ...) in the embodiment of the present invention,Then directionality instruction be only used for explain under a certain particular pose (as shown in the picture) between each component relative positional relationship,Motion conditions etc., if the particular pose changes, directionality instruction is also correspondingly changed correspondingly.
In addition, being somebody's turn to do " first ", " second " etc. if relating to the description of " first ", " second " etc. in the embodiment of the present inventionDescription be used for description purposes only, be not understood to indicate or imply its relative importance or implicitly indicate indicated skillThe quantity of art feature." first " is defined as a result, the feature of " second " can explicitly or implicitly include at least one spySign.It in addition, the technical solution between each embodiment can be combined with each other, but must be with those of ordinary skill in the art's energyIt is enough realize based on, will be understood that the knot of this technical solution when conflicting or cannot achieve when occurs in the combination of technical solutionConjunction is not present, also not the present invention claims protection scope within.
The present invention proposes a kind of audio collection of video conference and plays optimization method；
In first preferred embodiment of the invention, as shown in Figure 1, including the following steps:
The first readjustment of S10, setting audio collection parameter and one for being handled the audio digital signals of acquisitionFunction；
As shown in Figures 2 and 3, comprising:
S101, the example for obtaining an AVAudioSession class, setting AudioSession audio categories arekAudioSessionCategory_PlayAndRecord；
S102, customized the first graph of structural body comprising AUGraph and AudioUnit, and create and be directed toward the knotThe 2nd graph of AUGraph in structure body；
S103, it adds in the graph of Node to the 2nd of a Remote I/O type；
In the embodiment of the present invention, the Node of a Remote I/O type is stated, and be added in the 2nd graph, finallyThis graph is opened, Remote I/OUnit belongs to one of Audio Unit, is one relevant to hardware deviceUnit, it is divided into input terminal and output end, such as loudspeaker, microphone and earphone；The present invention only needs to record, therefore to allow defeatedEntering to hold Intput and microphone to connect can；
S104, the underlying parameter that audio collection is set.
In the embodiment of the present invention, some audio frequency parameters of AudioUnitSetProperty () function setup, such as one are calledThe sample rate of a little audio collections, each data packet how many frame, the occupancy digit of speech sample point, the bytes number of every frame, monophonicOr multichannel, the bytes sum of each data packet, the frame number of each data packet of bytes number * of every frame, echo cancellor settingAlso it is arranged herein；
S20, when each sampled audio is converted into digital signal by analog quantity, call the first call back function to digital audioSignal is handled；
In the embodiment of the present invention, CallBack function is set, when every secondary audio program comes in be converted into digital signal from microphoneThis CallBack function will be called, digital signal is made into the processing that you want, audio can be done herein some balanced, mixedThe operations such as conjunction, format conversion, according to project demands, I is the pcm data directly taken；Call back function is the static state side of a C languageThen method associates it with AUGraph；
In the embodiment of the present invention, calls AUGraphInitialize () function to initialize graph, and then callAUGraphStart () function opens graph；AudioOutputUnitStart () function is finally called to begin to acquisition audio?；
The the second readjustment letter of S30, setting audio play parameter and one for changing audio play buffer occupied stateNumber；
As shown in Figure 4, comprising:
S301, creation audio play object；
The underlying parameter that S302, setting audio play；
In the embodiment of the present invention, some basis instruments are carried out to audio stream；Such as sample rate, data package size, voice are everySampled point occupies digit etc.；
S303, setting internal thread play, and create output；
In the embodiment of the present invention, AudioQueueNewOutput () function is called, is played, is created defeated using internal threadOut；
S304, the second call back function of calling judge the occupied state of multiple buffering area；
In the embodiment of the present invention, setting audio finishes playing call back function.This function is mainly used to following change bufferingThen area's occupied state initializes the buffer area of needs, be preferably provided with three buffer areas and facilitate management；Finally callAudioQueueStart () function opens audio and plays queue；
S305, it audio data is sent to unappropriated buffer area plays out.
S40, when audio plays out, call the second call back function judgement distribution is carried out to audio buffer.
In the embodiment of the present invention, after taking audio data from the background, because data are pcm data thus, so not having to carry outWhat decoding process, first judge before buffer area of throwing to three buffer areas which be it is unappropriated, then the data hand overThat unappropriated buffer area is given, in the case that CPU occupancy is excessively high when broadcasting for audio, is used cooperatively hereinSleep () function substantially reduces CPU occupancy.Finally AudioQueueEnqueueBuffer () function is called to play out.
The present invention also proposes a kind of audio collection of video conference and plays optimization device；
In second preferred embodiment of the invention, as shown in Figure 5, comprising:
Processor；
Memory is coupled to the processor and is stored with instruction, and the instruction is executing reality by the processorThe step of audio collection and broadcasting optimization method of the existing video conference, such as:
The first readjustment of S10, setting audio collection parameter and one for being handled the audio digital signals of acquisitionFunction；
S101, the example for obtaining an AVAudioSession class, setting AudioSession audio categories arekAudioSessionCategory_PlayAndRecord；
S102, customized the first graph of structural body comprising AUGraph and AudioUnit, and create and be directed toward the knotThe 2nd graph of AUGraph in structure body；
S103, it adds in the graph of Node to the 2nd of a Remote I/O type；
S104, the underlying parameter that audio collection is set.
S20, when each sampled audio is converted into digital signal by analog quantity, call the first call back function to digital audioSignal is handled；
The the second readjustment letter of S30, setting audio play parameter and one for changing audio play buffer occupied stateNumber；
S301, creation audio play object；
The underlying parameter that S302, setting audio play；
In the embodiment of the present invention, some basis instruments are carried out to audio stream；Such as sample rate, data package size, voice are everySampled point occupies digit etc.；
S303, setting internal thread play, and create output；
S304, the second call back function of calling judge the occupied state of multiple buffering area；
S305, it audio data is sent to unappropriated buffer area plays out.
S40, when audio plays out, call the second call back function judgement distribution is carried out to audio buffer.
Particular technique details herein has elaborated in first preferred embodiment above, no longer repeats herein；
In the embodiment of the present invention, the audio collection of the video conference and optimization device internal processor is played, it can be withIt is made of, such as can be made of the integrated circuit of single package integrated circuit, is also possible to by multiple identical functions or notCongenerous encapsulation integrated circuit formed, including one or more central processing unit, microprocessor, digital processing chip,The combination etc. of graphics processor and various control chips.Processor takes all parts using various interfaces and connection, passes throughOperation executes the program being stored in memory or unit, and calls the data being stored in memory, to execute viewThe audio collection of frequency meeting and the various functions and processing data for playing optimization.
Memory is mounted on the video conference of fusion emotion and semantic rules for storing program code and various dataAudio collection and the access for playing in optimization device, and realizing high speed in the process of running, being automatically completed program or data.InstituteStating memory includes read-only memory, random access memory, programmable read only memory, Erasable Programmable Read Only Memory EPROM, oneSecondary programmable read only memory, electronics erasing type can make carbon copies read-only memory, CD-ROM or other disc memories, disk and depositReservoir, magnetic tape storage or any other the computer-readable medium that can be used in carrying or storing data.
The present invention also proposes a kind of computer-readable storage medium；
In third preferred embodiment of the invention, as shown in fig. 6, the computer-readable storage medium is stored with video councilThe audio collection of view and the application program for playing optimization method, the application program realize that the audio of video conference as mentioned is adoptedThe step of collection is with optimization method is played, such as:
The first readjustment of S10, setting audio collection parameter and one for being handled the audio digital signals of acquisitionFunction；
S101, the example for obtaining an AVAudioSession class, setting AudioSession audio categories arekAudioSessionCategory_PlayAndRecord；
S102, customized the first graph of structural body comprising AUGraph and AudioUnit, and create and be directed toward the knotThe 2nd graph of AUGraph in structure body；
S103, it adds in the graph of Node to the 2nd of a Remote I/O type；
S104, the underlying parameter that audio collection is set.
S20, when each sampled audio is converted into digital signal by analog quantity, call the first call back function to digital audioSignal is handled；
The the second readjustment letter of S30, setting audio play parameter and one for changing audio play buffer occupied stateNumber；
S301, creation audio play object；
The underlying parameter that S302, setting audio play；
In the embodiment of the present invention, some basis instruments are carried out to audio stream；Such as sample rate, data package size, voice are everySampled point occupies digit etc.；
S303, setting internal thread play, and create output；
S304, the second call back function of calling judge the occupied state of multiple buffering area；
S305, it audio data is sent to unappropriated buffer area plays out.
S40, when audio plays out, call the second call back function judgement distribution is carried out to audio buffer.
Particular technique details herein has elaborated in first preferred embodiment above, no longer repeats herein；
In the description of embodiments of the present invention, it should be noted that in flow chart or described otherwise above hereinAny process or method description be construed as, indicate to include one or more for realizing specific logical function or mistakeModule, segment or the part of the code of the executable instruction of the step of journey, and the range packet of the preferred embodiment of the present inventionInclude other realization, wherein sequence shown or discussed can not be pressed, including according to related function by it is basic simultaneouslyMode or in the opposite order, Lai Zhihang function, this should be managed by the embodiment of the present invention person of ordinary skill in the fieldSolution.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered useIn the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, forInstruction execution system, device or equipment (such as computer based system, including the system of processing module or other can be from instructionExecute system, device or equipment instruction fetch and the system that executes instruction) use, or combine these instruction execution systems, device orEquipment and use.For the purpose of this specification, " computer-readable medium ", which can be, any may include, store, communicating, propagatingOr transfer program uses for instruction execution system, device or equipment or in conjunction with these instruction execution systems, device or equipmentDevice.The more specific example (non-exhaustive list) of computer-readable medium include the following: there are one or more wiringsElectrical connection section (electronic device), portable computer diskette box (magnetic device), random access memory, read-only memory is erasableExcept editable read-only memory, fiber device and portable optic disk read-only storage.In addition, computer-readable medium is veryTo can be the paper that can print described program on it or other suitable media, because can be for example by paper or other JieMatter carries out optical scanner, is then edited, interprets or is handled when necessary with other suitable methods electronically to obtainDescribed program is obtained, is then stored in computer storage.
The above description is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all at thisUnder the inventive concept of invention, using equivalent structure transformation made by description of the invention and accompanying drawing content, or directly/use indirectlyIt is included in other related technical areas in scope of patent protection of the invention.