Disclosure of Invention
The technical problem to be solved by the invention is to provide a local area network video monitoring implementation method based on a domestic linux system, so as to support the function of video monitoring which is relatively popular all the time, and improve the service performance of an operating system.
In order to solve the problems, the technical scheme adopted by the invention is as follows:
a local area network video monitoring realization method based on a domestic linux system comprises the realization of a local area network video monitoring system, wherein the system comprises a signal acquisition device, an kylin operating system, a network transmission module, a local area network and a video receiving terminal, and the method comprises the following steps:
the signal acquisition equipment, namely an information source, is used for acquiring audio and video data;
the kylin operating system is used for realizing signal driving, signal capturing, coding compression, decoding processing and format conversion of the signal acquisition equipment;
the network transmission module is used for providing a network transmission function, converting data transmitted by the kylin operating system into data which can be received by an Ethernet physical layer and transmitting the data to a channel through a standard network interface; corresponding to the following channels, this step is used to complete the channel coding and modulation work;
a local area network, a channel for audio and video data transmission, that is, a channel;
and the video receiving terminal is used for displaying and processing the audio and video data received from the channel.
The technical scheme of the invention is further improved as follows: the signal acquisition equipment comprises a camera, an image sensor and an A/D converter which are connected in sequence, correspondingly, the transmission format of the signals is optical signals, electric signals, analog signals and digital signals in sequence, and the electric signals of the image sensor are finally in information connection with a kylin operating system through the A/D converter.
The technical scheme of the invention is further improved as follows: the kylin operating system comprises a signal processing module, an operating system main control module, an encoding and compressing module, a decoding module and a format transcoding module, wherein:
the signal processing module is used for realizing signal driving and capturing of the signal acquisition equipment so as to obtain an information source;
the operating system main control module is responsible for overall scheduling of the system;
the coding compression module is used for reducing the redundancy of the digital signal formed after A/D conversion and realizing the coding compression of the information source;
the decoding module is used for decoding the encoded and compressed data;
and the format transcoding module is used for transcoding the data after being encoded, compressed and decoded, namely the video stream and the picture photo of the network to be uploaded into a streaming media format, wherein the streaming media is a standard real-time video stream and a standard real-time picture photo.
The technical scheme of the invention is further improved as follows: the compression mode adopted by the coding compression module for the video image is a compression mode formulated by the Moving Picture Expert Group, such as MPEG-1, MPEG-2 or MPEG-4.
The technical scheme of the invention is further improved as follows: the streaming media format adopts an H.264 coding mode.
The technical scheme of the invention is further improved as follows: the network protocol adopted by the network transmission module is one or more of RTP/RTCP, UDP, HTTP and TCP/IP.
The technical scheme of the invention is further improved as follows: and transmitting audio and video data by using a UDP-based RTP/RTCP protocol, and transmitting a control command by using TCP/IP.
The technical scheme of the invention is further improved as follows: the video receiving terminal comprises a video decoder and an audio display and a video display which correspond to the video decoder, wherein the video decoder is positioned in the kylin operating system, and the synchronization of the audio display and the video display is realized by adding time stamps.
The technical scheme of the invention is further improved as follows: the operation flow of the local area network video monitoring is as follows:
a-1, starting a signal acquisition device: opening the signal acquisition equipment through an equipment interface of the kylin operating system, and starting to transmit audio and video data after the signal acquisition equipment receives a command; here RAW data.
A-2, capturing data: the kylin operating system carries out format image signal processing on the RAW format audio and video data captured by the built-in signal processing module and then outputs an audio and video pixel format, wherein the audio and video pixel format is imaging data, and the format is NV12 format.
A-3, encoding and decoding: the method comprises the following steps that a coding compression module of the kylin operating system carries out coding compression after receiving audio and video data, and then decoding processing is carried out through a decoding module for the next step;
a-4, format conversion of video stream/picture: the kylin operating system carries out format conversion of video stream/picture on the decoded data, so that the data can be accessed to a network at a place such as the web.
A-5, network transmission: after the format conversion of the video stream/picture is completed, the application program of the kylin operating system sends the data through the real-time streaming protocol, so that the pushing of the video stream is completed.
The technical scheme of the invention is further improved as follows: the real-time streaming protocol is one or more of RTP, RTSP, MJPEG and V4 l.
Due to the adoption of the technical scheme, the invention has the beneficial effects that:
1. the function of transmitting network video data on the kylin operating system is realized.
2. The function of decoding by using video format hardware on the kylin operating system is realized.
3. The function of processing signals of different cameras on the kylin operating system is realized.
4. And a network-based digital video transmission mode is adopted, so that the wiring problem of an analog video system is avoided, and the transmission quality is improved. Compared with an analog system, the system also has the advantages of multiple aspects such as storage, encryption, signal processing and the like.
5. Compared with an analog camera, the network video monitoring has fewer A/D (analog-to-digital) and D/A (digital-to-analog) conversion links, and can reduce the loss of image quality as little as possible. The used network camera adopts line-by-line scanning, and carries a linux operating system to realize the acquisition of higher resolution, thereby obtaining a video image with higher quality. The network video monitoring can integrate the function of bidirectional audio input and output, is realized by integrating corresponding software packages of an operating system, and is connected with a microphone and a loudspeaker through an audio input and output port arranged at the front end.
6. As a video monitoring function, the method can be applied to various scenes, for the kylin operating system, the usability of the operating system can be greatly enhanced by the method for supporting video monitoring, and the use scenes and the penetration of the operating system in various fields are enriched.
7. The local area network video monitoring scheme based on the linux operating system is realized, so that a client using the kylin operating system can support the local area network video monitoring function in advance.
Detailed Description
The present invention will be described in further detail with reference to examples.
The invention discloses a local area network video monitoring realization method based on a domestic linux system, which is realized by a local area network video monitoring system, wherein the system comprises signal acquisition equipment, an kylin operating system, a network transmission module, a local area network and a video receiving terminal, wherein the local area network video monitoring realization method comprises the following steps:
the signal acquisition equipment, namely an information source, is used for acquiring audio and video data;
the kylin operating system is used for realizing signal driving, signal capturing, coding compression, decoding processing and format conversion of the signal acquisition equipment;
the network transmission module is used for providing a network transmission function, converting data transmitted by the kylin operating system into data which can be received by an Ethernet physical layer and transmitting the data to a channel through a standard network interface; this step also completes the channel coding and modulation work;
a local area network, a channel for audio and video data transmission, that is, a channel;
and the video receiving terminal is used for displaying and processing the audio and video data received from the channel.
The following examples are given for illustrative purposes.
1. Schema framework
The main components of the network video monitoring are a signal acquisition device, an kylin operating system and a coding compression module. Such as the block diagram shown in fig. 1.
First, abbreviations and key term definitions are introduced
Kylin operating system platform: a home-made operating system flagship product comprises a server version and a desktop version, and respectively supports a Loongson processor, an ARM processor, a Shenwei processor and an X86 processor.
Linux: a system developed based on the open source kernel code warehouse can be called a Linux operating system.
And (4) DSP: and (5) digital signal processing.
Ethernet is Ethernet, which refers to the baseband lan specification created by Xerox corporation and jointly developed by Xerox, Intel, and DEC corporation, and is the most common communication protocol standard adopted by the existing lans today.
And then detailed below.
1.1 the signal acquisition equipment includes camera, image sensor, the AD converter that connects gradually, and the camera is the network camera. The transmission format of the signals is optical signal-electric signal-A/D conversion in sequence, and the signals of the camera and the image sensor are finally connected with information of the kylin operating system through the A/D converter.
Compared with an analog camera, the network video monitoring utilizes the network camera, so that the A/D, D/A conversion link of the network camera is less, and the loss of image quality can be reduced as little as possible. The network camera adopts progressive scanning, and carries a linux operating system to realize acquisition with higher resolution, so that a video image with higher quality can be obtained. The network video monitoring can integrate a bidirectional audio function, is realized by integrating corresponding software packages of an operating system, and is connected with a microphone and a loudspeaker through an audio input/output port arranged at the front end.
The network video monitoring hardware comprises: the camera is a front-end component for network video monitoring, and images are adjusted in modes of zooming and zooming; the image sensor is a device for converting optical signals into electric signals, and generally comprises a CCD (charge coupled device) and a CMOS (complementary metal oxide semiconductor), wherein the CCD is superior to the CMOS in the aspects of sensitivity, signal to noise ratio and the like, so that the image sensor is used more; A/D conversion is required to be carried out on the CCD-based image sensor to generate a digital signal; and then, coding and compressing are carried out, and a coding and compressing part carries out coding and compressing according to a certain standard.
1.2 kylin operating system, including signal processing module, operating system master control module, coding compression module, decoding module, format transcoding module, wherein:
the signal processing module is used for realizing signal driving and capturing of the signal acquisition equipment so as to obtain an information source;
the operating system main control module is responsible for overall scheduling of the system;
the coding compression module is used for reducing the redundancy of the digital signal formed after A/D conversion and realizing the coding compression of the information source;
the decoding module is used for decoding the encoded and compressed data;
and the format transcoding module is used for transcoding the data after being coded, compressed and decoded (namely the video stream and the picture photo of the network to be uploaded) into a streaming media format, wherein the streaming media is a standard real-time video stream and a standard real-time picture photo.
The signal processing module is completed by a chip or an kylin operating system kernel, a camera is connected to the chip in a pin mode, the chip can read the data of the camera and complete the step of processing the data signal of the camera, and then the operating system kernel finishes the signal capture of the camera on a mounting CPU (central processing unit) such as MIPI (mobile industry processor interface), USB (universal serial bus) and the like.
The main control module of the operating system is the core of network video monitoring and is responsible for the scheduling work of the whole system, the kylin operating system firstly captures the data of the camera, the data is directly sent to the coding compression module through the system layer, and after the coding compression and decoding processing are finished, the audio and video data which are subjected to coding compression are read and are sent to the network module for transmission.
The format transcoding module is mainly used in the process of uploading video streams/picture photos to a network, and due to the fact that video decoding needs a uniform format mode, all format transcoding needs to be transcoded into standard real-time video streams and picture photos through an kylin operating system.
The network transmission module provides network function, which converts the data transmitted from the main control module of the operation system into the data that can be received by the Ethernet physical layer, and transmits the data to the network through the standard network interface. Network video monitoring generally adopts network protocols such as RTP/RTCP, UDP, HTTP, TCP/IP and the like, and allows a user to remotely access the network video monitoring, modify parameters, browse and control real-time videos and the like.
2. Principle of code compression
A camera and an image sensor of the network video monitoring are equivalent to an analog signal source part in a communication system, and digital signals are formed after A/D conversion. The digital signal at this time contains great redundancy and needs to be compressed by a source. The compression method of the video image can be a compression method established by Moving Picture Expert Group such as MPEG-1, MPEG-2, MPEG-4, etc.
The streaming media format adopts an H.264 coding mode, the H.264 is the tenth part of the MPEG-4 standard and is provided by JVT formed by combining ITU-T Video Coding Experts Group (VCEG) and MPEG of ISO/IEC. H.264 adopts a mixed coding mode of predictive coding and transform coding, has a very high compression ratio which is about 1.5-2 times of MPEG-4.
H.264 adopts a hybrid coding scheme of intra and inter prediction. The main functions include prediction, transform, quantization and entropy coding, but one more loop filter function is added to enhance the image quality.
Such as the h.264 coding structure shown in fig. 2, such a coding algorithm may be implemented on a CPU or other chip of a network video surveillance or video encoder.
3. Network transmission
The retransmission mechanism of the TCP protocol has higher time delay, but ensures the integrity and accuracy of the data; the UDP protocol has poor data transmission quality. Based on the characteristics, in the network video transmission process, a real-time transport protocol (RTP) and a transport control protocol (RTCP) are usually adopted on UDP and are used in cooperation to meet the requirement of network video data stream transmission; and the transmission of control commands is performed by using TCP/IP.
The audio and video data are transmitted by adopting an RTP protocol based on UDP, and the RTP has a timestamp mark, a serial number and other methods which can ensure the real-time data transmission processing time, thereby providing time information for data streams and realizing stream synchronization. RTCP provides flow control and congestion control services along with RTP. Each participant periodically transmits RTCP packets during the RTP session, which contain statistics of the number of packets sent, the number of packets lost, etc., and the server changes the transmission rate by this.
RTP and RTCP are used in conjunction to optimize transmission efficiency with efficient feedback and minimal overhead.
As shown in fig. 3, the RTP video stream is encapsulated, and the audio and video data is transmitted by using a UDP-based RTP/RTCP protocol, and the two protocols are used in cooperation to meet the requirement of network video data stream transmission, so that the audio and video stream is encapsulated as an RTP packet for network transmission, and the RTP packet includes five parts, namely an Ethernet header, an IP header, a UDP header, an RTP header, and an RTP load.
Fig. 4 shows an RTP video streaming process, having in sequence: encoder, encoded video stream, RTP packets, RTP streams, IP streams, network
4. And the video receiving terminal and the audio and video are synchronous.
The receiving end is composed of a video decoder, an audio display and a video display corresponding to the video decoder.
The video decoder is located in the kylin operating system and is divided into a soft decoder and a hard decoder. One end of the hard decoder is connected with the network, and the other end of the hard decoder is connected with the monitor, converts the digital signals into analog signals and displays the analog signals through the monitor. The hard decoder is generally integrated in an kylin operating system based on a chip existing form, the soft decoder is generally used for calculating soft resolutions for a CPU (central processing unit) carried by the operating system, and the kylin operating system carries out decoding through a computer program. These programs are typically provided by the encoding vendor. The h.264 decoding algorithm is complex and requires a high CPU.
The audio and video synchronization is realized by adding a time stamp. SCR is understood to be the time when the decoder should start reading data from disk, each packet in the MPEG file has an SCR time stamp and this time stamp is the system time when this packet was read. The DTS timestamp determines that the decoder decodes when the SCR time equals the DTS time, and similarly the PTS timestamp. DTS (decoding time stamp) and PTS (presentation time stamp) are time stamps relative to SCR (system reference) when a decoder decodes and presents frames, respectively.
The decoder is a high-definition decoder AoKu Decx2, and the H.264 high-definition embedded codec is-BE/HVX 18100 and the like.
6. Communication system architecture
From the perspective of the communication system, the video surveillance system is presented in the form of a block diagram, as shown in fig. 5.
As can be seen from fig. 5, first of all signal acquisition devices, such as cameras, CMOS, sensors, etc., are the source.
Followed by source coding such as a/D, DSP, h.264 compression coding, etc.
The network transmission module is used for channel coding and modulation.
Then the local area network, which is a channel.
And then a decoding module, which is a demodulation part for channel decoding, source decoding.
And finally a video receiving terminal such as a monitor, a sound box, a computer, etc.
The key point in the design of the scheme is that a network-based digital video transmission mode is adopted, the wiring problem of an analog video system is avoided, and the transmission quality is improved. Compared with an analog system, the system also has the advantages of multiple aspects such as storage, encryption, signal processing and the like. The difficulty is that the digital video equipment is more complex than the analog equipment, the analog equipment uses a cable, a network system needs to occupy network bandwidth, and the digital video equipment is easily influenced by a network environment in practice.
7. Flowchart parsing
The running process of video monitoring on an kylin operating system is shown in fig. 6, and the process is as follows:
a-1: and (3) starting the camera, starting the camera through an equipment interface of the kylin operating system, and starting to transmit camera data after the camera receives a command, wherein the camera data is original data-RAW data.
A-2: and (3) capturing data, and carrying out format image signal processing on the captured RAW format camera data by the kylin operating system, and outputting an audio and video pixel format NV 12.
In step a-2, only format image signal processing of the video RAW is actually performed without decoding, the format image signal processing includes one or more of automatic exposure control, automatic gain control, automatic white balance, color correction, and dead pixel removal, and finally outputs a video pixel format such as NV12, and a format such as NV12 is output without encoding compression and decoding.
A-3: encoding and decoding of video pixel format: the imaged data are sequentially sent to a coding compression module and a decoding module of an kylin operating system for coding and decoding, the decoded data (in YUV/RGB/JPEG format) are sent to an operating system main control module after decoding, and the operating system main control module receives the coded and decoded audio/video data through a hardware interface (such as CSI/USB and the like) for subsequent scheduling work.
In the step a-3, mainly, encoding and decoding are performed on the data for image signal processing, and the processing of the imaged data is performed as encoding and decoding in a pixel format. In practice, data in the format of NV12 and the like in step a-2 is not generally common, and it is generally necessary to transcode the data into a relatively common data format such as YUV/RGB/JPEG.
A-4: the method comprises the steps of converting video stream (picture) formats, wherein an operating system main control module of an kylin operating system receives audio and video data input with a certain format, such as camera data with YUV/RGB/JPEG formats and the like, and then performs video stream/picture format conversion on the video data input, so that the video data input can be accessed to a network on a web and other places. Such as format conversion to streaming video H264.
In step a-4, the video data in YUV format that has been imaged is obtained through a certain real-time streaming protocol interface, for example, using V4l protocol, and then converted into streaming media video H264, and then sent to the opposite web browser for playing through a network sending method. The reason why the format transcoding is necessary is because video streams in the camera format that do not support YUV data generally are on the web, and in this case, it is necessary to convert the YUV format camera data into video data that can be played by a general-purpose player such as H264/H265 or the like, such as streaming media.
A-5: and (3) network transmission, after the format conversion of the video stream/picture is completed, the data of the data is transmitted by the application program of the kylin operating system through a real-time streaming protocol RTP, an RTSP, an MJPEG, a V4l and the like, so that the pushing of the video stream is completed.
The key point of the method is that the scheme of local area network video monitoring based on the kylin operating system is realized, so that a client using the kylin operating system can support the function of local area network video monitoring in advance, when the user uses the standard kylin operating system, the user can control the own low-power-consumption Bluetooth intelligent equipment, and the use evaluation of the user on the standard kylin operating system can be improved. Furthermore, the image visualization effect is carried out on the data of various low-power-consumption Bluetooth devices, so that a user can know the state of information collected by the intelligent device more clearly, the user can operate the low-power-consumption Bluetooth device more conveniently based on the Kangma operating system, and the user can not miss any alarm message from the intelligent device of the user due to the message notification of the low-power-consumption Bluetooth device.