CN101742218A - A method for implementing a videophone system between Internet TV set-top boxes - Google Patents

Abstract

Translated fromChinese

一种用于网络电视机顶盒之间的可视电话系统的实现方法，是一种在网络电视(IPTV)机顶盒上开发可视电话系统的方法，此方案主要是扩展网络电视机顶盒的增值业务功能，所依赖的硬件平台是TI公司提供的面向视频开发领域的达芬奇平台(DaVinci)TMS320DM6446。利用TI公司推出的DM6446DVEVM开发套件为硬件平台，对音频和视频数据分别进行采集和捕获，视频数据采用数字信号处理DSP自带的高性能的视频编解码技术H.264视频编解码，音频则采用软编解码方式，直接调用语音压缩标准G.711编解码算法，由微处理器ARM来处理；音视频数据的网络传输采用用户数据报协议作为传输层协议，而在应用层进行实时传送协议打包。

A method for realizing a videophone system between Internet TV set-top boxes is a method for developing a videophone system on an Internet TV (IPTV) set-top box. This solution is mainly to expand the value-added service functions of the Internet TV set-top box. The hardware platform it relies on is the DaVinci platform (DaVinci) TMS320DM6446 provided by TI for video development. Use the DM6446DVEVM development kit launched by TI as the hardware platform to collect and capture the audio and video data respectively. The soft codec method directly invokes the voice compression standard G.711 codec algorithm, which is processed by the microprocessor ARM; the network transmission of audio and video data uses the user datagram protocol as the transport layer protocol, and the real-time transmission protocol is packaged at the application layer .

Description

Translated fromChinese

一种用于网络电视机顶盒之间的可视电话系统的实现方法A method for implementing a videophone system between Internet TV set-top boxes

技术领域technical field

本发明是一种在网络电视(IPTV)机顶盒上开发可视电话系统的方法，此方案主要是扩展网络电视机顶盒的增值业务功能，所依赖的硬件平台是TI公司提供的面向视频开发领域的达芬奇平台(DaVinci)TMS320DM6446。属于嵌入式应用领域。The present invention is a method for developing a videophone system on an Internet TV (IPTV) set-top box. This solution is mainly to expand the value-added service function of the Internet TV set-top box. Vinci platform (DaVinci) TMS320DM6446. It belongs to the embedded application field.

背景技术Background technique

可视电话(Video Phone)业务是一种通过传统电话网、互联网或视讯专网的形式，集图像、话音于一体的多媒体通信业务，来实现人们面对面的实时沟通，即通话双方在通话过程中能够互相看到对方场景，近年来已在远程会议、远程教学、远程医疗等方面得到了快速的发展。目前的可视电话的实现基本上基于PC的软件，通过IP的网络服务提供视频电话。对于这种解决方案，用户需要电脑、摄像头、宽带上网和一定的电脑操作知识等基础条件，所以很大程度上局限了其拥有的用户群体。随着IPTV交互式网络电视的推广，普通家庭可以有两种方式享受IPTV服务：计算机或者IPTV终端。IPTV终端能够很好地适应当今网络飞速发展的趋势，充分地利用网络资源。IPTV既不同于传统的模拟式有线电视，也不同于数字电视。可视电话作为IPTV的一个增值业务提供给用户，对IPTV的运营起到了积极的作用。Video Phone (Video Phone) service is a multimedia communication service that integrates image and voice through the traditional telephone network, the Internet or a private video network to realize face-to-face real-time communication between people. Being able to see each other's scenes has developed rapidly in recent years in teleconferencing, distance teaching, and telemedicine. The current implementation of videophone is basically based on PC software, and provides videophone through IP network service. For this solution, users need basic conditions such as computers, cameras, broadband Internet access, and certain computer operation knowledge, so the user groups it has are largely limited. With the promotion of IPTV interactive Internet TV, ordinary families can enjoy IPTV services in two ways: computers or IPTV terminals. IPTV terminals can well adapt to the rapid development trend of today's network and make full use of network resources. IPTV is not only different from traditional analog cable TV, but also different from digital TV. As a value-added service of IPTV, videophone is provided to users and plays a positive role in the operation of IPTV.

TI公司提供的面向音、视频开发领域的达芬奇数字平台(DaVinci)TMS320DM6446，该平台是拥有ARM(Advanced RISC Machines)和DSP(DigitalSingnal Processor)双CPU(Central Processing Unit)内核的高端嵌入式开发平台，主频高达720MHZ。该平台上拥有丰富的硬件接口如USB(Universal SerialBus)、网卡、IDE硬盘接口(Integrated Drive Electronics)等等。此平台在音视频编解码的处理上采用了达芬奇(DaVinci)技术。达芬奇(DaVinci)技术是一种专门针对数字音视频应用、基于信号处理的解决方案，能为音视频设备制造商提供集成处理器、软件、工具等支持，以简化设计进程，加速产品创新。由于大量的音视频编解码工作需要一个强劲的DSP处理器作为支撑，此硬件平台自带的DVEVM开发套件，能通过DSP对音、视频进行编解码，音频支持G.711编解码算法，视频支持MPEG4和H.264两种编解码算法。The DaVinci digital platform (DaVinci) TMS320DM6446 provided by TI for the field of audio and video development is a high-end embedded development platform with ARM (Advanced RISC Machines) and DSP (Digital Signal Processor) dual CPU (Central Processing Unit) cores platform, the main frequency is up to 720MHZ. The platform has rich hardware interfaces such as USB (Universal Serial Bus), network card, IDE hard disk interface (Integrated Drive Electronics) and so on. This platform adopts DaVinci (DaVinci) technology in the processing of audio and video codec. DaVinci (DaVinci) technology is a signal-processing-based solution specifically for digital audio and video applications. It can provide audio and video equipment manufacturers with integrated processors, software, tools, etc. to simplify the design process and accelerate product innovation. . Since a large number of audio and video codecs need a powerful DSP processor as support, the DVEVM development kit that comes with this hardware platform can code and decode audio and video through DSP. The audio supports G.711 codec algorithm, and the video supports MPEG4 and H.264 two codec algorithms.

发明内容Contents of the invention

技术问题：本发明的目的是提供一种用于网络电视机顶盒之间的可视电话系统的实现方法，该平台是拥有ARM和DSP双核的高端嵌入式开发平台，考虑到机顶盒自带的只采用DSP进行音、视频编解码所产生的高负载问题，本发明通过使用ARM音频软编解码、DSP视频编解码，大大减轻了DSP的负荷，使得ARM的CPU占用率和DSP的CPU占用率能达到很好的平衡。Technical problem: the purpose of the present invention is to provide a kind of realization method that is used for the videophone system between the set-top box of Internet TV, this platform is to have the high-end embedded development platform of ARM and DSP dual-core, considering that the set-top box carrying only adopts DSP carries out the high load problem that sound, video coding and decoding produce, the present invention has alleviated the load of DSP greatly by using ARM audio frequency soft coding and decoding, DSP video coding and decoding, makes the CPU occupancy rate of ARM and the CPU occupancy rate of DSP can reach nice balance.

技术方案：本发明的一种用于网络电视机顶盒之间的可视电话系统的实现方法利用TI公司推出的DM6446 DVEVM开发套件为硬件平台，对音频和视频数据分别进行采集和捕获，视频数据采用DSP自带的H.264视频编解码，音频则采用软编解码方式，直接调用G.711编解码算法，由ARM来处理。音视频数据的网络传输采用UDP协议作为传输层协议，而在应用层进行RTP打包。Technical scheme: a kind of implementation method for the videophone system between Internet TV set-top boxes of the present invention utilizes the DM6446 DVEVM development kit that TI Company introduces as the hardware platform, collects and captures the audio and video data respectively, and the video data adopts The DSP comes with H.264 video codec, and the audio adopts soft codec method, which directly calls the G.711 codec algorithm and is processed by ARM. The network transmission of audio and video data adopts UDP protocol as the transport layer protocol, while RTP is packaged at the application layer.

用于网络电视机顶盒之间的可视电话系统的实现方法如下：The implementation method for the videophone system between the Internet TV set-top boxes is as follows:

用于网络电视机顶盒之间的可视电话系统的实现方法是利用TI公司推出的DM6446DVEVM开发套件为硬件平台，对音频和视频数据分别进行采集和捕获，视频数据采用数字信号处理DSP自带的高性能的视频编解码技术H.264视频编解码，音频则采用软编解码方式，直接调用语音压缩标准G.711编解码算法，由微处理器ARM来处理；音视频数据的网络传输采用用户数据报协议作为传输层协议，而在应用层进行实时传送协议打包，具体实现方法如下：The realization method of the videophone system between the Internet TV set-top boxes is to use the DM6446DVEVM development kit launched by TI as the hardware platform to collect and capture the audio and video data respectively. High-performance video codec technology H.264 video codec, audio adopts soft codec method, directly calls voice compression standard G.711 codec algorithm, and is processed by microprocessor ARM; network transmission of audio and video data adopts user data The packet protocol is used as the transport layer protocol, and the real-time transport protocol is packaged at the application layer. The specific implementation method is as follows:

步骤1).进行需求分析，对网络机顶盒之间的可视电话系统进行分析，并对划分的模块和功能的需求进行设计；Step 1). Carry out demand analysis, analyze the videophone system between network set-top boxes, and design the divided modules and functional requirements;

步骤2).按照步骤1设计的各功能模块，熟悉各模块之间的交互流程，对各个模块之间的逻辑关系和功能进行说明；Step 2). According to each functional module designed in step 1, be familiar with the interaction process between each module, and explain the logical relationship and function between each module;

步骤3).按照步骤2的功能说明，首先设计与实现人机交互的人性化界面，采用MiniGUI在Linux系统下界面编程，在可视电话系统运行之后，会弹出可视化界面，包括IP地址输入按钮、设置按钮和关闭按钮，点击以上按钮，会弹出相应的对话框，供用户简捷的操作，Step 3). According to the function description in step 2, first design and realize the humanized interface of human-computer interaction, and use MiniGUI to program the interface under the Linux system. After the videophone system is running, the visual interface will pop up, including the IP address input button , the setting button and the close button, click the above button, a corresponding dialog box will pop up for the user to operate simply,

步骤4).利用硬件平台自带的DVEVM开发套件，可以对音、视频数据进行采集和捕获，并且还能通过数字信号处理DSP对音、视频进行编解码，音频支持语音压缩标准G.711编解码算法，视频支持视频、音频和多媒体编码标准MPEG4和高性能的视频编解码技术H.264两种编解码算法；考虑到数字信号处理DSP的高负荷问题，此系统音频不用自带的编解码引擎，而是直接在程序中加入编解码算法代码，采用微处理器来处理，视频则采用自带的编解码引擎，由此微处理器和数字信号处理DSP资源占用均衡，Step 4). Using the DVEVM development kit that comes with the hardware platform, audio and video data can be collected and captured, and audio and video can be coded and decoded through digital signal processing DSP. The audio supports voice compression standard G.711 encoding Decoding algorithm, video supports video, audio and multimedia encoding standard MPEG4 and high-performance video encoding and decoding technology H.264 two encoding and decoding algorithms; considering the high load problem of digital signal processing DSP, this system audio does not use its own encoding and decoding engine, but directly add the codec algorithm code in the program, and use the microprocessor to process, and the video uses the built-in codec engine, so that the microprocessor and digital signal processing DSP resource occupation is balanced,

步骤5).各模块设计完成后，系统的运行主要是多个线程的交互和执行，可视电话系统运行后，首先运行网络监听回调函数和控制线程，控制线程主要负责用户界面，不停的查看遥控器是否有命令输入；用户输入对方IP地址请求对方应答，得到返回的接受消息后，运行音频线程、播放线程、视频线程、捕获线程、显示线程和网络传输线程，以此来进行双方音、视频的正常交互，Step 5). After the design of each module is completed, the operation of the system is mainly the interaction and execution of multiple threads. After the videophone system is running, it first runs the network monitoring callback function and the control thread. The control thread is mainly responsible for the user interface. Check whether the remote controller has a command input; the user inputs the other party's IP address to request the other party's response, and after receiving the returned acceptance message, run the audio thread, playback thread, video thread, capture thread, display thread and network transmission thread to carry out two-way audio , the normal interaction of the video,

步骤6).网络传输流媒体数据是可视电话系统中必不可少的一个重要环节，考虑到音、视频所要求的实时传输性，虽然采用传输控制协议传输具有高可靠性，但是由于三步握手带来的延迟和过多交互数据使其不适合发送大量的实时视频数据，在这种情况下，选用专为发送大量音、视频等多媒体数据的实时传送协议，实时传送协议由数据协议和控制协议两部分组成，实时传送协议通常使用用户数据报传送数据，控制协议用来支持其协议的功能，考虑使用一些开放源代码的实时传送协议库。Step 6). Network transmission of streaming media data is an essential part of the videophone system. Considering the real-time transmission required by audio and video, although the transmission control protocol is used to transmit with high reliability, due to the three-step The delay caused by the handshake and too much interactive data make it unsuitable for sending a large amount of real-time video data. In this case, a real-time transmission protocol designed for sending a large amount of multimedia data such as audio and video is selected. The real-time transmission protocol consists of data protocols and The control protocol consists of two parts. The real-time transport protocol usually uses user datagrams to transmit data, and the control protocol is used to support the functions of the protocol. Consider using some open-source real-time transport protocol libraries.

有益效果：本发明方法提出了在网络电视机顶盒上开发可视电话系统，并在具体实现中采用音频ARM软编解码、视频DSP编解码，主要用于解决了DSP高负载的问题，通过使用这种发明方法，不仅扩展了网络电视机顶盒的增值业务功能，还在一定程度上缓解了只采用DSP进行音、视频编解码的低效性和不稳定性。此发明方法还考虑人机交互问题，采用MiniGUI系统进行了人性化的界面设计。可视电话系统低耦合的模块化设计，具有良好的可扩展性。以下给出具体的说明：Beneficial effects: the method of the present invention proposes to develop a videophone system on the Internet TV set-top box, and adopts audio ARM soft codec and video DSP codec in the specific implementation, which is mainly used to solve the problem of high load of DSP. By using this This inventive method not only expands the value-added service functions of the Internet TV set-top box, but also alleviates the inefficiency and instability of only using DSP for audio and video encoding and decoding to a certain extent. This inventive method also considers the problem of human-computer interaction, and adopts the MiniGUI system to carry out the humanized interface design. The low-coupling modular design of the videophone system has good scalability. Specific instructions are given below:

高度的稳定性：本发明方法的可视电话系统功能完善，关键技术的运用合理，摆脱了只是运用DSP自带的音、视频编解码设计方式，采用了ARM和DSP共同协作来完成系统的设计，使得ARM和DSP之间的CPU占用率均衡稳定，达到了预期的设计目的。High stability: the videophone system of the inventive method has perfect functions, the use of key technologies is reasonable, and the design method of audio and video coding and decoding that only uses the DSP comes with it is eliminated, and ARM and DSP are used to cooperate together to complete the design of the system , so that the CPU occupancy rate between ARM and DSP is balanced and stable, and the expected design purpose is achieved.

高效的实时性：可视电话系统在音、视频数据流的实时传输方面，根据多媒体数据流要求实时性高、延迟小和可容忍适当的丢包率等特点，采用了专为发送大量实时音、视频等多媒体数据的RTP协议，其数据以UDP形式发送，增加了控制功能。相比TCP三次握手而产生的延迟，本发明方法选用RTP协议在实时性方面得到了很大的提高。Efficient real-time performance: In terms of real-time transmission of audio and video data streams, the videophone system adopts a special method for sending a large number of real-time audio , video and other multimedia data RTP protocol, the data is sent in the form of UDP, which increases the control function. Compared with the delay caused by the TCP three-way handshake, the method of the present invention uses the RTP protocol to greatly improve the real-time performance.

良好的扩展性：本发明方法采用低耦合的模块化设计，其功能模块大体由以下六个部分组成：音频采集播放模块、音频编解码模块、视频捕捉显示模块、视频编解码模块、系统控制模块和网络传输模块。系统各模块之间的层次化分明，提供了各模块通信的接口，因此也很容易升级各功能模块。如果更换音频或视频的编解码算法，只需在相应的音频编解码模块或是视频编解码模块上操作即可。也可很容易升级为PC对网络机顶盒的可视电话系统。Good expansibility: The method of the present invention adopts a low-coupling modular design, and its functional modules are generally composed of the following six parts: audio collection and playback module, audio codec module, video capture and display module, video codec module, system control module and network transmission module. The hierarchy among the various modules of the system is clear, and the communication interface of each module is provided, so it is easy to upgrade each functional module. If you change the codec algorithm of audio or video, you only need to operate on the corresponding audio codec module or video codec module. It can also be easily upgraded to a PC-to-Network set-top box videophone system.

人性化的界面：在人机界面的操作方面，引入了MiniGUI系统，为用户提供界面编程，如IP地址输入、呼叫和图像显示等对话框或窗口，使用户更易于操作。Humanized interface: In terms of the operation of the man-machine interface, the MiniGUI system is introduced to provide users with interface programming, such as IP address input, call and image display and other dialog boxes or windows, making it easier for users to operate.

附图说明Description of drawings

图1可视电话系统各模块交互的总体架构，Figure 1 The overall architecture of the interaction between modules of the videophone system,

图2音频所有相关的流程图，Figure 2 Audio All Related Flowcharts,

图3是可视电话系统用户交互的框图，Fig. 3 is a block diagram of user interaction of the videophone system,

图4是可视电话系统各个线程之间的交互图，Fig. 4 is an interaction diagram between each thread of the videophone system,

图5视频所有相关的流程图，Figure 5 Video All relevant flowcharts,

图6是运行此系统后用户通信的详细流程。Figure 6 is the detailed flow of user communication after running this system.

具体实施方式Detailed ways

一、体系结构：1. System structure:

本发明是在P2P IPTV终端上对可视电话进行的研究。其功能模块大体由以下六个部分组成：音频采集播放模块、音频编解码模块、视频捕捉显示模块、视频编解码模块、系统控制模块和网络传输模块。如图1所示。The present invention is a research on videophone on P2P IPTV terminal. Its functional modules are generally composed of the following six parts: audio collection and playback module, audio codec module, video capture and display module, video codec module, system control module and network transmission module. As shown in Figure 1.

下面我们对各模块进行具体的介绍：Below we introduce each module in detail:

音频采集播放模块：语音采集、播放模块为嵌入式视频系统的必备组件。该模块完成音频信号的采集、播放等功能。它主要由TI生产的低功耗立体声编解码芯片TLV320AIC33组成。该芯片有多个输入端口和多个可编程输出端口。基于寄存器的电源控制模块使其在48KHz DAC回路播放时系统功耗只有14mw。极低的功耗使其特别适合于嵌入式系统的应用。Audio collection and playback module: the voice collection and playback module is an essential component of the embedded video system. This module completes the audio signal acquisition, playback and other functions. It is mainly composed of low-power stereo codec chip TLV320AIC33 produced by TI. The chip has multiple input ports and multiple programmable output ports. The register-based power control module makes the system consume only 14mw when playing in the 48KHz DAC loop. Extremely low power consumption makes it especially suitable for embedded system applications.

AIC33输入端有数控立体声麦克风前置放大、自动增益控制和对多路输入混音处理等许多强大的功能。输出端有4路普通输出和3路差分输出。同时其DAC和ADC支持8KHz到96KHz之间的多种频率采样。AIC33使用多种电压，其中模拟电压2.7V-3.6V，数字内核电压1.525-1.95V，数字1/0电压1.IV-3.6V。AIC33与DSP的音频端接口(ASP)相连，工作模式为全双工串行通信。AIC33 input terminal has many powerful functions such as digital control stereo microphone preamplification, automatic gain control and multi-channel input mixing processing. There are 4 common outputs and 3 differential outputs at the output. At the same time, its DAC and ADC support multiple frequency sampling between 8KHz and 96KHz. AIC33 uses a variety of voltages, including analog voltage 2.7V-3.6V, digital core voltage 1.525-1.95V, and digital 1/0 voltage 1.IV-3.6V. AIC33 is connected with the audio frequency end interface (ASP) of DSP, and the working mode is full-duplex serial communication.

音频编解码模块：语音通信是可视电话最基本的功能。受网络条件的限制，可视电话通常工作在较低码率下。为了适应这种低码率语音应用，ITU_T推出了一系列音频和语音压缩标准。其中G.711、G.723.1、G.728、G.729和G.729A在可视电话中得到了广泛应用。Audio codec module: Voice communication is the most basic function of videophone. Limited by network conditions, videophones usually work at a lower bit rate. In order to adapt to this low-bit-rate voice application, ITU_T has introduced a series of audio and voice compression standards. Among them, G.711, G.723.1, G.728, G.729 and G.729A have been widely used in videophone.

G.711也称为PCM(脉冲编码调制)，是国际电信联盟订定出来的一套语音压缩标准，主要用于电话。它主要用脉冲编码调制对音频采样，采样率为8k每秒。它利用一个64Kbps未压缩通道传输语音讯号。起压缩率为1∶2，即把16位数据压缩成8位。G.711是主流的波形声音编解码器。G.711, also known as PCM (Pulse Code Modulation), is a set of voice compression standards formulated by the International Telecommunication Union, mainly used for telephones. It mainly uses pulse code modulation to sample audio, and the sampling rate is 8k per second. It uses a 64Kbps uncompressed channel to transmit voice signals. The starting compression ratio is 1:2, that is, 16-bit data is compressed into 8 bits. G.711 is the mainstream waveform sound codec.

视频捕捉显示模块：对于视频捕捉，在系统上电后，TMS320DM6446通过SPI接口对时序信号发生器(CXD2457R)进行初始化。初始化完成后，TMS320DM6446的CCD控制器产生行、场驱动信号送给时序信号发生器，时序信号发生器产生CCD时序控制信号和A/D转换芯片的采样时序信号。CCD采集的原始图像数据，送到A/D转换芯片，输出10bitBayer模板原始数据信号送给TMS320DM6446的CCD控制器进行处理。CCD控制器主要产生合适的行、场时序信号和对原始图像进行数字箝位和黑电平补偿等处理，处理后的图像送到DDR2存储器。DSP从DDR2存储器取到原始数据后，进行中值滤波和噪声滤波，CFA插值和RGB到YUV转换等算法处理，输出分辨率为1024×768的YUV(4:2:2)格式的数字视频信号。Video capture display module: For video capture, after the system is powered on, TMS320DM6446 initializes the timing signal generator (CXD2457R) through the SPI interface. After the initialization is completed, the CCD controller of TMS320DM6446 generates row and field drive signals to the timing signal generator, and the timing signal generator generates CCD timing control signals and sampling timing signals of the A/D conversion chip. The original image data collected by the CCD is sent to the A/D conversion chip, and the original data signal of the output 10bitBayer template is sent to the CCD controller of TMS320DM6446 for processing. The CCD controller mainly generates appropriate line and field timing signals and performs digital clamping and black level compensation on the original image, and the processed image is sent to the DDR2 memory. After the DSP fetches the original data from the DDR2 memory, it performs algorithm processing such as median filtering and noise filtering, CFA interpolation and RGB to YUV conversion, and outputs a digital video signal in YUV (4:2:2) format with a resolution of 1024×768 .

数字视频显示系统主要由DM6446的视频后端处理子系统、CPLD器件EPM570、LCD屏LQ057Q3DC12和LCD的背光源电路组成。其中，DM6446芯片采用ARM与DSP双核结构，ARM子系统搭载297MHz主频的ARM926核，DSP部分采用594MHz的C64X+DSP核，视频处理子系统(VPSS)具有丰富的视频前后处理功能，特色功能单元VICP专用的媒体协处理器，外围存储均支持DDR2、Flash、ATA、CF、SD等外设接口。由于DM6446的数字视频输出管脚电压是1.8V，须经EPM570转换为3.3V，再与LCD屏的3.3V相应管脚连接。The digital video display system is mainly composed of DM6446 video back-end processing subsystem, CPLD device EPM570, LCD screen LQ057Q3DC12 and LCD backlight circuit. Among them, the DM6446 chip adopts ARM and DSP dual-core structure, the ARM subsystem is equipped with an ARM926 core with a main frequency of 297MHz, and the DSP part adopts a 594MHz C64X+DSP core. VICP dedicated media coprocessor, peripheral storage supports DDR2, Flash, ATA, CF, SD and other peripheral interfaces. Since the digital video output pin voltage of DM6446 is 1.8V, it must be converted to 3.3V by EPM570, and then connected to the 3.3V corresponding pin of the LCD screen.

视频编解码模块：视频编码模块设计：DM6446片内的VPBE模块包含4个54MHz的D/A转换器，可在DM6446内部将数字视频信号直接转化为模拟视频信号，4路输出，并且支持CVBS、S-端子、YprPb 3种模拟视频格式。因此，视频编码模块设计较为简单，只需对4路模拟输出信号放大，就可直接与监视设备连接。选用TI公司的电压反馈CMOS运算放大器OPA357进行运算放大。Video codec module: Video codec module design: The VPBE module in DM6446 includes four 54MHz D/A converters, which can directly convert digital video signals into analog video signals inside DM6446, with 4 outputs, and supports CVBS, S-Video, YprPb 3 kinds of analog video formats. Therefore, the design of the video encoding module is relatively simple, and it can be directly connected to the monitoring equipment only by amplifying the 4-way analog output signals. Choose TI's voltage feedback CMOS operational amplifier OPA357 for operational amplification.

视频解码模块设计：这里选用专用的视频解码器ADV7189B，它支持12路模拟视频通道，包含3个具有防噪性能的12位54MHz的A/D转换器。支持CVBS、S-端子、YprPb 3种格式的模拟视频信号输入，能够自动侦测NTSL/PAL/SECAM制式，输出ITU-R BT.656标准的数字视频信号。选用12路模拟通道中的3路，复用的支持3种模拟视频格式。ADV7189B输出10位数字视频信号、独立的垂直同步信号VD、水平同步信号HD和像素同步时钟LLC1，电压均为3.3V电平，经过FPGA转换为DM6446要求的1.8V，然后从DM6446的VPFE模块专用数字视频信号接口送入DSP。压缩编码前，VPFE模块将ITU-R BT.656标准的视频数据转换为H.264兼容的YUV4:2:0格式，存入DDR2 SDRAM中。VPFE模块还支持对视频数据进行白平衡、缩放等预处理操作。ADG3301实现I2C总线的电平转换。Design of video decoding module: Here we use a dedicated video decoder ADV7189B, which supports 12 analog video channels, including 3 12-bit 54MHz A/D converters with anti-noise performance. Support CVBS, S-Video, YprPb 3 kinds of analog video signal input, can automatically detect NTSL/PAL/SECAM standard, output digital video signal of ITU-R BT.656 standard. Select 3 of the 12 analog channels, and multiplex supports 3 analog video formats. ADV7189B outputs 10-bit digital video signal, independent vertical synchronous signal VD, horizontal synchronous signal HD and pixel synchronous clock LLC1, the voltage is 3.3V level, after FPGA conversion to 1.8V required by DM6446, and then from the VPFE module dedicated to DM6446 The digital video signal interface is sent to DSP. Before compression encoding, the VPFE module converts ITU-R BT.656 standard video data into H.264 compatible YUV4:2:0 format and stores it in DDR2 SDRAM. The VPFE module also supports preprocessing operations such as white balance and scaling on video data. The ADG3301 implements the level translation of the I2C bus.

系统控制模块：主要是保证可视电话连接的正常建立、释放及提供可视电话会话过程中的信息控制，如终端间的主从决定、能力交换、逻辑信道的打开与关闭等。运用到实际环境中，WCDMA/TD-SCDMA电路域可视电话业务中采用ITU-T H.245作为控制协议。System control module: mainly to ensure the normal establishment and release of videophone connections and provide information control during videophone conversations, such as master-slave decision between terminals, capability exchange, opening and closing of logical channels, etc. When applied to the actual environment, ITU-T H.245 is used as the control protocol in the WCDMA/TD-SCDMA circuit domain videophone service.

网络传输模块：主要是音频和视频编码后的网络传输，一般采用RTP/UDP/IP协议。该音视频传输平台采用UDP协议作为传输层协议，而且在应用层进行RTP打包。在网络发送数据之前，视频通过DSP的H.264编码压缩，音频通过G.711编码进行压缩，以利于在网络中更好的进行音、视频的传输。Network transmission module: mainly for network transmission after audio and video encoding, generally using RTP/UDP/IP protocol. The audio and video transmission platform adopts UDP protocol as the transport layer protocol, and performs RTP packaging at the application layer. Before sending data over the network, the video is compressed by the H.264 encoding of the DSP, and the audio is compressed by the G.711 encoding, so as to facilitate better audio and video transmission in the network.

二、方法流程：2. Method flow:

基于网络机顶盒之间的可视电话系统的流程如图6所示。首先用户在用户界面内输入对方的IP地址，发起连接请求，对方同意后，两者建立连接。之后双方的操作步骤是相似的：初始化视频设备，通过摄像头捕获视频数据，并用DSP自带的H.264进行编码；初始化音频设备，通过音频采集设备，采集音频数据，并用G.711算法进行编码。然后通过网络，发送给对方音、视频数据包，对方接收数据后，再分别对音频和视频数据解码，音频由扬声器播放出来，视频通过电视机进行显示。The flow of the videophone system based on the network set-top boxes is shown in FIG. 6 . First, the user inputs the IP address of the other party in the user interface, initiates a connection request, and after the other party agrees, the two establish a connection. Afterwards, the operation steps of both parties are similar: initialize the video equipment, capture video data through the camera, and use the H.264 that comes with the DSP to encode; initialize the audio equipment, collect audio data through the audio collection equipment, and use the G.711 algorithm to encode . Then through the network, send audio and video data packets to the other party. After receiving the data, the other party decodes the audio and video data respectively. The audio is played by the speaker, and the video is displayed on the TV.

此发明的可视电话系统的实现方法，总体上把握可以归结为以下四块的实现：线程之间交互的实现、音频所有相关的实现，视频所有相关的实现和网络传输的实现。以下详细介绍各块的实现。The realization method of the videophone system of this invention can be summed up in the following four realizations on the whole: the realization of interaction between threads, the realization of all related audio, the realization of all related video and the realization of network transmission. The implementation of each block is described in detail below.

线程之间交互的实现：本发明的所依赖的是MontaVista Linux嵌入式操作系统，在此基础上所编写的程序由多个线程组成。首先主线程进行一些初始化的工作，用户输入对方IP地址，在请求对方连接并得到响应后，开始创建捕获线程、显示线程、视频线程、音频编码线程和音频解码线程，以上线程创建完毕，开始调用控制线程里的函数ctrlThrFxn()，此时主线程转化为控制线程。Realization of interaction between threads: what the present invention depends on is the MontaVista Linux embedded operating system, and the program written on this basis is made up of multiple threads. First, the main thread performs some initialization work. The user enters the IP address of the other party. After requesting the other party to connect and getting a response, the capture thread, display thread, video thread, audio encoding thread, and audio decoding thread are created. After the above threads are created, start calling The function ctrlThrFxn() in the control thread, at this time the main thread is transformed into a control thread.

为保证各线程稳定的执行，需要为各个线程设定优先权。除了主线程基于预定的(SCHED_FIFO)优先权之外，视频线程和显示线程享有最高优先权，其次是捕获线程，再次是音频编码线程和音频解码线程，最低优先权是控制线程。In order to ensure the stable execution of each thread, it is necessary to set the priority for each thread. In addition to the main thread based on predetermined (SCHED_FIFO) priority, the video thread and the display thread have the highest priority, followed by the capture thread, followed by the audio encoding thread and the audio decoding thread, and the lowest priority is the control thread.

程序运行后，各线程也相应的创建。控制线程主要负责用户界面，它使用msp430lib库去监测控制IR接口的msp430处理器，查看是否有IR命令输入。一旦接收到一个由遥控器键入的新IR命令，命令就能够被识别并且响应的动作会在keyAction中执行。在视频方面，捕获线程从video线程取得一个空的raw buffer。用已经移除重叠部分后的数据填充它，这个buffer随后被发送到视频线程使用VIDENC_process()调用DSP H.264进行编码，captured buffer随后被写入I/Obuffer，然后发送给对方。当接收到对方的视频编码数据后，写入raw buffer，视频线程使用VIDENC_process()调用DSP H.264进行解码，为了能把解码帧显示出来，调用函数FifoUtil_put()给显示线程传递一个指向raw buffer的指针，之后显示线程使用VPSS resizer模块和Rszcopy_execute()函数拷贝raw buffer到FBDev帧缓冲设备中显示出来。在音频方面，音频编码线程通过调用Read()函数获取音频数据，写入分配的raw buffer，直接调用G.711编码算法代码进行编码，并写入另一块分配的buffer，发送给对方。音频解码线程接收到对方发来的音频数据，写入raw buffer，直接调用G.711解码算法代码进行解码，并写入另一块分配的buffer，调用Write()函数进行音频播放。当用户需要终止通话时，只需通过遥控器键入终止键，则控制线程会响应并终止双方此次的通话。线程交互如图4所示。After the program runs, each thread is created accordingly. The control thread is mainly responsible for the user interface. It uses the msp430lib library to monitor the msp430 processor controlling the IR interface to see if there is an IR command input. Once a new IR command is received from the remote, the command is recognized and the corresponding action is executed in keyAction. On the video side, the capture thread gets an empty raw buffer from the video thread. Fill it with the data after the overlapping part has been removed, this buffer is then sent to the video thread to use VIDENC_process() to call DSP H.264 for encoding, the captured buffer is then written to the I/Obuffer, and then sent to the other party. After receiving the video encoding data from the other party, write it into the raw buffer, and the video thread uses VIDENC_process() to call DSP H.264 for decoding. In order to display the decoded frame, call the function FifoUtil_put() to pass a pointer to the raw buffer to the display thread. After that, the display thread uses the VPSS resizer module and the Rszcopy_execute() function to copy the raw buffer to the FBDev frame buffer device for display. In terms of audio, the audio encoding thread obtains audio data by calling the Read() function, writes it into the allocated raw buffer, directly calls the G.711 encoding algorithm code for encoding, writes it into another allocated buffer, and sends it to the other party. The audio decoding thread receives the audio data sent by the other party, writes it into the raw buffer, directly calls the G.711 decoding algorithm code to decode, and writes it into another allocated buffer, and calls the Write() function to play the audio. When the user needs to terminate the call, he only needs to key in the termination key through the remote control, and the control thread will respond and terminate the current call between the two parties. The thread interaction is shown in Figure 4.

音频所有相关的实现：网络电视机顶盒上需要连接用于音频采集的话筒。OSS为多种Unix(或Unix兼容的操作系统)提供声卡和其他声音设备的驱动，AIC33声音设备驱动是OSS其中一种设备驱动，用于音频数据的采集。本发明考虑音、视频都采用DSP编解码产生高负荷问题，采用了音频软编解码的方式，把G.711算法代码加入到音频线程中直接调用，避免了使用DSP的操作。其具体的方法流程如图2所示，步骤如下：All audio-related implementations: The Internet TV set-top box needs to be connected to a microphone for audio collection. OSS provides drivers for sound cards and other sound devices for a variety of Unix (or Unix-compatible operating systems). The AIC33 sound device driver is one of the OSS device drivers for audio data collection. The present invention considers the problem of high load caused by DSP encoding and decoding for both audio and video, adopts audio soft encoding and decoding, and adds G.711 algorithm codes to the audio thread for direct calling, avoiding the operation of using DSP. The specific method flow is shown in Figure 2, and the steps are as follows:

(1)首先用InitSoundDevice()函数，初始化AIC33设备驱动。(1) First use the InitSoundDevice() function to initialize the AIC33 device driver.

(2)为原始的立体声(stereo)采样数据分配缓冲区，因为这个缓冲区不会涉及到DSP(stereo-to-mono的转换是由ARM实现的)，所以此处分配的缓冲区不要求一定是连续的，使用malloc()函数。(2) Allocate a buffer for the original stereo (stereo) sampling data, because this buffer will not involve DSP (stereo-to-mono conversion is implemented by ARM), so the buffer allocated here does not require a certain Is continuous, use malloc () function.

(3)调用Read()函数来采集音频数据，又因为AIC33设备只支持立体声，所以要从两个信道上读取stereo采样数据放入缓冲区。(3) Call the Read() function to collect audio data, and because the AIC33 device only supports stereo, it is necessary to read the stereo sampling data from two channels and put them into the buffer.

(4)调用stereoToMono()函数，把立体双声道数据转换成单声道。(4) Call the stereoToMono() function to convert the stereo two-channel data into mono.

(5)在音频线程里调用G.711编码函数(g711a_Encode())进行音频数据的编码。(5) Call the G.711 encoding function (g711a_Encode()) in the audio thread to encode the audio data.

调用编码函数编码好音频数据后，通过网络传输发送给对方，对方也首先需要AIC33设备驱动的初始化，以及分配缓冲区等，然后解码音频数据，调用Write()函数写入缓冲区，通过AIC33设备播放出来。After calling the encoding function to encode the audio data, send it to the other party through network transmission. The other party first needs to initialize the AIC33 device driver and allocate buffers, etc., then decode the audio data, call the Write() function to write the buffer, and pass the AIC33 device Play it out.

视频所有相关的实现：网络电视机顶盒上首先需要连接用于视频捕获的摄像头。视频处理前端系统(VPFE)用于负责从外设(摄像头)接收并处理原始的视频流信号，视频处理前端中的CCD控制器(CCDC)将具体负责对视频数据的采集工作。Linux内核中关于管理视频采集设备的驱动接口是V4L2(Video for Linux Two)。获取采集到得数据后，DSP把视频的数据格式由RGB到YUV转换处理，输出分辨率为1024x768的YUV(4:2:2)格式的数字视频信号。利用mmap(map device memory intoapplication address space)将内核空间的设备内存地址空间映射到用户空间的地址空间的方式，方便进程访问数据。All video-related implementations: First, a camera for video capture needs to be connected to the Internet TV set-top box. The video processing front-end system (VPFE) is responsible for receiving and processing the original video stream signal from the peripheral (camera), and the CCD controller (CCDC) in the video processing front-end will be specifically responsible for the acquisition of video data. The driver interface for managing video capture devices in the Linux kernel is V4L2 (Video for Linux Two). After acquiring the collected data, the DSP converts the video data format from RGB to YUV, and outputs a digital video signal in YUV (4:2:2) format with a resolution of 1024x768. Using mmap (map device memory into application address space) to map the device memory address space of the kernel space to the address space of the user space, it is convenient for the process to access data.

视频程序首先通过FifoUtil_open()函数打开与捕获程序之间的通信缓冲，调用FifoUtil_get()函数和FifoUtil_put()函数作为视频线程与捕获线程之间的数据交流通道。视频线程对数据的编码步骤如下：The video program first opens the communication buffer with the capture program through the FifoUtil_open() function, and calls the FifoUtil_get() and FifoUtil_put() functions as the data exchange channel between the video thread and the capture thread. The video thread encodes data as follows:

(1)使用CodecEngine的Engine_open()来创建视频编码算法引擎，返回一个句柄，所有使用相同Engine的模块线程都需要单独的句柄，来确定线程的安全；(1) Use CodecEngine's Engine_open() to create a video encoding algorithm engine and return a handle. All module threads using the same Engine need separate handles to determine thread safety;

(2)使用videoEncodeAlgCreate()创建编码算法，使用VIDENC_create()里的静态参数来创建“H.264”视频编码器；(2) Use videoEncodeAlgCreate() to create an encoding algorithm, and use the static parameters in VIDENC_create() to create an "H.264" video encoder;

(3)使用Memory_contigAlloc()函数为编码缓存与原始视频数据缓存分配一段连续的内存空间；(3) Use the Memory_contigAlloc() function to allocate a continuous memory space for the encoding cache and the original video data cache;

(4)使用VIDENC_process()函数调用H.264算法对数据进行编码。(4) Use the VIDENC_process() function to call the H.264 algorithm to encode the data.

编码好的视频数据通过网络传输发送给对方，对方的视频线程对数据进行解码，步骤如下：The encoded video data is sent to the other party through network transmission, and the video thread of the other party decodes the data. The steps are as follows:

(1)使用CodecEngine的Engine_open()来创建视频解码算法引擎，返回一个句柄，所有使用相同Engine的模块线程都需要单独的句柄，来确定线程的安全；(1) Use CodecEngine's Engine_open() to create a video decoding algorithm engine and return a handle. All module threads using the same Engine need separate handles to determine thread safety;

(2)使用videoDecodeAlgCreate()创建解码算法，这包括：a.使用VIDDEC_create()里的静态参数来创建“H.264dec”视频解码器；b.使用VIDDEC_control()和XDM_GETSTATUS来设置动态的视频解码参数，查询编解码缓冲区大小；(2) Use videoDecodeAlgCreate() to create a decoding algorithm, which includes: a. Use static parameters in VIDDEC_create() to create a "H.264dec" video decoder; b. Use VIDDEC_control() and XDM_GETSTATUS to set dynamic video decoding parameters , query the codec buffer size;

(3)使用Memory_contigAlloc()函数为解码缓存分配一段连续的内存空间；(3) Use the Memory_contigAlloc() function to allocate a continuous memory space for the decoding cache;

(4)使用VIDDEC_process()函数调用H.264算法对数据进行解码。(4) Use the VIDDEC_process() function to call the H.264 algorithm to decode the data.

对于解码好的视频帧，通过FifoUtil_put()函数，发送解码好的视频帧给显示线程，显示线程通过FifoUtil_get()函数收到视频解码帧，然后用initDispaly-Device()函数来初始化FBDev(Frame Buffer Device)显示设备驱动，帧缓冲设备FBDev用来访问视频输入输出硬件，是视频硬件的一个抽象表示，这样应用程序不用了解任务低层次的接口。显示线程选择帧缓冲设备/dev/fh/3进行视频的显示播放。视频处理后端系统系统(VPBE)实现对视频流信号进行显示、输出等功能。流程如图5所示。For the decoded video frame, send the decoded video frame to the display thread through the FifoUtil_put() function, and the display thread receives the video decoded frame through the FifoUtil_get() function, and then uses the initDispaly-Device() function to initialize the FBDev(Frame Buffer Device) display device driver, the frame buffer device FBDev is used to access the video input and output hardware, which is an abstract representation of the video hardware, so that the application program does not need to understand the low-level interface of the task. The display thread selects the frame buffer device /dev/fh/3 to display and play the video. The video processing back-end system (VPBE) realizes functions such as displaying and outputting video stream signals. The process is shown in Figure 5.

网络传输的实现：根据可视电话自身的特点，需要传输实时流多媒体数据，本发明选用专为发送大量音视频等多媒体数据的RTP协议。对采集到得音视频数据，压缩后写入FIFO缓冲队列，进行RTP打包，形成RTP流发送到网络中(可选用JRTPLIB库中SendPacket()函数)，对方接收到数据包后，根据RTP包头里的信息进行排序处理，送入缓冲区。等待解码处理。Realization of network transmission: According to the characteristics of the videophone itself, it is necessary to transmit real-time stream multimedia data. The present invention selects the RTP protocol specially for sending multimedia data such as a large amount of audio and video. The collected audio and video data is compressed and written into the FIFO buffer queue, and RTP is packaged to form an RTP stream and sent to the network (the SendPacket() function in the JRTPLIB library can be selected). After the other party receives the data packet, it The information is sorted and sent to the buffer. Waiting for decoding processing.

在处理RTP数据包的时候，采用缓冲技术来处理数据。数据在网络传输中可以分解成若干个RTP数据包，由于网络的动态变化，每个包的传输路径和到达接收端的时间都可能不一样，因此采用缓冲技术来弥补延迟和变化的影响。接收端将接收到的RTP数据包解包后放入缓冲区，根据RTP包头中的序列号将数据重新排列，送入解码缓冲区进行实时解码。When processing RTP data packets, buffering technology is used to process data. Data can be decomposed into several RTP packets during network transmission. Due to the dynamic changes of the network, the transmission path of each packet and the time to reach the receiving end may be different, so buffering technology is used to compensate for the impact of delay and changes. The receiving end unpacks the received RTP data packet and puts it into the buffer, rearranges the data according to the sequence number in the RTP header, and sends it to the decoding buffer for real-time decoding.

Claims (1)

1. implementation method that is used for the video-phone system between the network television-set top-set-box, it is characterized in that the DM6446DVEVM development kit of utilizing TI company to release is hardware platform, the Voice ﹠ Video data are gathered respectively and caught, the high performance video coding and decoding technology that video data employing Digital Signal Processing DSP carries is coding and decoding video H.264, audio frequency then adopts soft code encoding/decoding mode, directly call G.711 code decode algorithm of voice compression, ARM handles by microprocessor; The Network Transmission of audio, video data adopts User Datagram Protoco (UDP) as transport layer protocol, and carries out the real time transport protocol packing in application layer, and the specific implementation method is as follows:

Step 1). carry out demand analysis, the video-phone system between the network machine top box is analyzed, and the module of division and the demand of function are designed;

Step 2). according to each functional module of step 1 design, be familiar with the interaction flow between each module, logical relation between each module and function are described;

Step 3). according to the function declaration of step 2, at first design and the user-centered interface of realizing man-machine interaction, adopt MiniGUI interface programming under linux system, after the video-phone system operation, can eject visualization interface, comprise IP address load button, button and X button are set, click above button, can eject corresponding dialog box, for the simple and direct operation of user

Step 4). the DVEVM development kit of utilizing hardware platform to carry, can gather and catch sound, video data, and can also carry out encoding and decoding to sound, video by Digital Signal Processing DSP, audio frequency support voice compression standard is code decode algorithm G.711, and video is supported H.264 two kinds of code decode algorithms of video, audio frequency and multimedia coding standard MPEG4 and high performance video coding and decoding technology; Consider the high load capacity problem of Digital Signal Processing DSP, the encoding and decoding engine that this system audio need not carry, but directly in program, add the code decode algorithm code, adopt microprocessor to handle, video then adopts the encoding and decoding engine that carries, microprocessor and the equilibrium of Digital Signal Processing DSP resource occupation thus

Step 5). after each modular design is finished, the operation of system mainly is the mutual of a plurality of threads and carries out that after the video-phone system operation, at first operational network is monitored call back function and control thread, the control thread mainly is responsible for user interface, and whether the remote controller of checking that does not stop has the order input; The user imports the other side IP Address requests the other side and replys, obtain returning accept message after, operation audio frequency thread, play thread, video thread, catch thread, show thread and Network Transmission thread, carry out the normal mutual of both sides' sound, video with this,

Step 6). the Network Transmission stream medium data is a requisite important step in the video-phone system, consider sound, the desired real-time Transmission of video, though adopt the transmission control protocol transmission to have high reliability, but delay and the too much interaction data that brings makes it be not suitable for sending a large amount of real time video datas because three steps shook hands, in this case, select for use to aim at and send a large amount of sounds, the real time transport protocol of multi-medium datas such as video, real time transport protocol is made up of data protocol and control protocol two parts, real time transport protocol uses user datagram to transmit data usually, control protocol is used for supporting the function of its agreement, considers to use the real time transport protocol storehouse of some open source codes.

Images