TWI855158B

Movatterモバイル変換

Info

Publication number: TWI855158B
Application number: TW109131857A
Authority: TW
Inventors: 施清德
Original assignee: 大陸商深圳釩佳科技有限公司
Priority date: 2020-09-16
Filing date: 2020-09-16
Publication date: 2024-09-11
Also published as: TW202213990A

Abstract

A live broadcasting system for real time 3D)image display is disclosed and includes an image capturing module, a 3D live broadcast algorithm module, a live broadcast platform server, an AI learning module and a 3D decoder. The image capturing module is to acquire a 3D image of an object. The 3D live broadcast algorithm module is used to receive and optimizing the 3D image. The live broadcast platform server is used to receive data signal with the 3D image and broadcast the data signal to the device of at least one user terminal. The AI learning includes a learning mode to record a video choice of the at least one user terminal and generate a video deployment setup according to the video choice. The 3D decoder is connected to the live broadcast platform server and the AI learning module and used to encode and decode the 3D image.

Description

Translated fromChinese

即時三維影像顯示的直播系統Live broadcast system for real-time 3D image display

本發明涉及一種直播系統，特別是涉及一種即時三維影像顯示的直播系統。The present invention relates to a live broadcast system, in particular to a live broadcast system for real-time three-dimensional image display.

自從即時通訊軟體發明以來，經過許多年的發展及技術進步，已經從一開始的純文字雙方溝通、聊天，到現在已經進入到視訊加文字的溝通聊天方式，漸漸成為現代人們生活的一部分，除了傳統的即時通訊功能以外，經過一段時間的發展，已經產生了許多的應用，如直播購物、網路即時會議、直播娛樂等等。即時通訊的聊天方式在表情、自拍、濾鏡的表達方式進化到“將一切都圖片化、視頻化”的社交趨勢後，直播這個自我表達的新領域便順勢誕生了。Since the invention of instant messaging software, after many years of development and technological progress, it has evolved from the initial pure text-based two-way communication and chat to the current video and text communication and chat mode, gradually becoming a part of modern people's lives. In addition to the traditional instant messaging function, after a period of development, many applications have been produced, such as live shopping, online instant meetings, live entertainment, etc. After the chat mode of instant messaging evolved from expressions, selfies, and filters to the social trend of "picturing and videoizing everything", the new field of self-expression, live broadcasting, was born.

網路直播的購物方式特點在於現場直播，並經過後台很短的時間差距即刻將產品介紹內容廣播出去到網路上，所以用戶終端或是觀看者可即時的看到產品介紹內容，並可以與直播主進行即時的互動。如圖1所示，其顯示通過直播主12透過行動裝置的軟體(APP)在網路上向用戶終端11直播販售貨品的示意圖，這種交易與交流的方式，不同於傳統的電視購物及網電購物，電視購物採用的預錄製再擇時播放的方式，所以購物並不是即時的；而傳統網店則是採用網上置放產品介紹圖片的掛網，再配合網店經營者的即時服務完成產品交易。The characteristic of online live shopping is that the live broadcast is on-site, and the product introduction content is broadcast to the Internet in a short time gap in the background, so the user terminal or the viewer can see the product introduction content in real time and can interact with the live broadcaster in real time. As shown in Figure 1, it shows a schematic diagram of the live broadcaster 12 selling goods to the user terminal 11 through the software (APP) of the mobile device on the Internet. This transaction and communication method is different from traditional TV shopping and online shopping. TV shopping adopts the method of pre-recording and then playing at a selected time, so shopping is not real-time; while traditional online stores adopt the method of placing product introduction pictures on the Internet, and then cooperate with the online store operator's real-timeservice to complete product transactions.

直播的可視性及即時的互動方式，漸漸受到現代人們追捧，因此，當這種方式應用在網路的購物方式上，可以大量的節省人們在外採購物品所耗費的時間，並且在購物的過程中，充滿了娛樂的樂趣，也提高了購物的即時性。但是這種網路即時觀看即時購物的方式，雖然有主持人或是專業人員在直播現場介紹產品，購物者在遠端的室內環境或是移動的情境中觀看顯示螢幕上的2D產品顯像而決定是否購物，由於不是在產品現場觀看，透過用戶終端的手機或是智能型電視機所看到的2D產品介紹影像，會產生與實際的物品有落差，等到消費終端收到實物產品時候，發現與心理的期待有不小的差異。The visibility and real-time interaction of live broadcasts are gradually being sought after by modern people. Therefore, when this method is applied to online shopping, it can save a lot of time that people spend on purchasing items outside. In addition, the shopping process is full of entertainment and the immediacy of shopping is improved. However, in this online real-time viewing and real-time shopping method, although there are hosts or professionals introducing products on the live broadcast site, shoppers watch the 2D product images on the display screen in a remote indoor environment or mobile situation to decide whether to buy. Since they are not watching the products on site, the 2D product introduction images seen on the user's mobile phone or smart TV will be different from the actual products. When the consumer receives the physical product, he will find that it is quite different from his psychological expectations.

直播聊天、娛樂、購物的方式之所以會受到人們的喜歡及流行，除了即時、生動畫面的可看性，直播主的氣氛帶動之外，觀賞者更可以與直播主進行互動或是留言，顯示螢幕上可以直接顯示出觀賞者及直播主的文字互動結果，這種互動式的行為模式，可以很大的縮短雙方的溝通成本，在直播購物的應用例子中，就可以縮短購物者及銷售方的距離，並且這種方式也適合於手持行動裝置，例如手機、平板電腦等等。所以直播可應用的平台範圍相較於過去的各種即時通訊軟體、電視購物、平面式的網路購物更廣泛也更直接，也打破了過去的平台之間的界線。The reason why live chat, entertainment, and shopping are popular and popular is that in addition to the real-time and vivid images and the atmosphere of the live broadcaster, viewers can interact with the live broadcaster or leave messages. The text interaction results between the viewers and the live broadcaster can be directly displayed on the display screen. This interactive behavior model can greatly shorten the communication cost of both parties. In the application example of live shopping, the distance between shoppers and sellers can be shortened, and this method is also suitable for handheld mobile devices, such as mobile phones, tablets, etc. Therefore, the platform range of live broadcasting is wider and more direct than various instant messaging software, TV shopping, and flat-screen online shopping in the past, and it also breaks the boundaries between the platforms in the past.

但是目前的這種直播平台10，如圖1所示，仍然是依靠在傳統的二維(2D)視頻及影像顯示技術基礎上，因此如觀眾或購物的用戶終端11對於主播12的認識、直播購物依然存在與真實的世界有很大的理解差異。舉例來說，直播方或是銷售方在進行化妝品的說明場景，如圖2所示，由於傳統2D視頻上所廣播的影像20不具備深度的立體訊息，銷售方在直播現場已經塗抹了化妝品的顏色及形狀，在遠端的觀看者卻因為影像形成的限制，很難感受到直播現場光亮立體的色彩及光澤感，這對於產品的展示效果大大的打了折扣。這是因為這類的產品在展示時，沒有透過具備深度訊息的三維立體光影、及三維影像，無法表現出產品的完整特色。類似這樣的應用例子，在目前的二維影像顯示上，已經產生了很多限制。However, the current live broadcast platform 10, as shown in FIG1, still relies on the traditional two-dimensional (2D) video and image display technology. Therefore, the understanding of the anchor 12 and the live shopping of the audience or the shopping user terminal 11 is still very different from the real world. For example, the live broadcaster or the seller is explaining the cosmetics, as shown in FIG2. Since the image 20 broadcasted on the traditional 2D video does not have a deep three-dimensional message, the seller has already applied the color and shape of the cosmetics at the live broadcast site. However, due to the limitation of the image formation, the remote viewers find it difficult to feel the bright three-dimensional color and gloss of the live broadcast site, which greatly reduces the display effect of the product. This is because when displaying such products, they cannot show the full features of the products without three-dimensional lighting and three-dimensional images that have depth information. Application examples like this have already caused many limitations in the current two-dimensional image display.

故，如何通過設計的改良，來解決直播平台的平面顯示限制，並可以提高交流以及溝通的效率，已成為該項事業所欲解決的重要課題之一。Therefore, how to solve the flat display limitation of the live broadcast platform through design improvement and improve the efficiency of communication has become one of the important issues that this business wants to solve.

本發明所要解決的技術問題在於，針對現有技術的不足提供一種可以顯示三維影像的直播方法，且這種直播方法不會造成影像品質的低落或使用者觀看時的延遲問題產生。The technical problem to be solved by the present invention is to provide a live broadcast method that can display three-dimensional images in response to the shortcomings of the existing technology, and this live broadcast method will not cause a decrease in image quality or delay problems when users watch.

為了解決上述的技術問題，本發明所採用的其中一技術方案是提供一種即時三維影像顯示的直播系統，其包括一影像擷取模組、一三維直播演算模組、一直播系統伺服器、一人工智慧學習模組以及一三維解碼器。影像擷取模組用於取得一實體的一三維影像，三維直播演算模組連接影像擷取模組，用於接收三維影像，並對三維影像進行優化，直播系統伺服器連接三維直播演算模組，用於接收具有三維影像的資料訊號，並將資料訊號推播至至少一用戶終端的裝置上，人工智慧學習模組連接直播系統伺服器，具有一學習模式，用於記錄至少一用戶終端的視頻選擇條件，並根據視頻選擇條件，產生一視頻調配組合，三維解碼器連接直播系統伺服器與人工智慧學習模組，用於編碼或解碼三維影像。In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a live broadcast system for real-time three-dimensional image display, which includes an image capture module, a three-dimensional live broadcast calculation module, a live broadcast system server, an artificial intelligence learning module and a three-dimensional decoder. The image capture module is used to obtain a three-dimensional image of an entity. The three-dimensional live broadcast calculation module is connected to the image capture module and is used to receive and optimize the three-dimensional image. The live broadcast system server is connected to the three-dimensional live broadcast calculation module and is used to receive a data signal with a three-dimensional image and push the data signal to at least one user terminal device. The artificial intelligence learning module is connected to the live broadcast system server and has a learning mode for recording the video selection conditions of at least one user terminal and generating a video allocation combination according to the video selection conditions. The three-dimensional decoder is connected to the live broadcast system server and the artificial intelligence learning module and is used to encode or decode the three-dimensional image.

本發明的其中一有益效果在於，本發明所提供的直播系統與直播方法，為減少資料的即時流量，採用影像的二維與三維並存，混合編輯的方法，這樣觀看者可以把焦點放置在需要關注的產品或人物身上，以觀賞到最佳及最需要的三維影像，並且這種二維與三維影像並存的方式，可以在使用者介面上進行選擇設置二維影像觀賞，或是三維影像觀賞，亦即表示使用者可以隨時變更需要觀看的三維影像位置，可以設置在區域性，或是全圖三維顯示。二維影像可以選擇三維左右式影像中的左影像或是右影像作為二維影像的顯示圖。另外，除了用戶終端裝置的立體顯示方式外，主播端或是雲端後台的控制端，也可以放置立體顯示裝置，以提供主播端的立體顯示預覽，及與用戶終端的互動立體顯示。另外，更可以讓用戶終端或直播主端通過三維直播影像的橫向縱向控制裝置可以容易選擇橫放或縱放的方式來進行三維直播顯示。One of the beneficial effects of the present invention is that the live broadcast system and live broadcast method provided by the present invention adopts the method of coexistence of two-dimensional and three-dimensional images and mixed editing to reduce the real-time flow of data, so that viewers can focus on the products or people that need to be paid attention to, so as to watch the best and most needed three-dimensional images, and this coexistence of two-dimensional and three-dimensional images can be selected and set on the user interface to watch two-dimensional images or three-dimensional images, that is, the user can change the position of the three-dimensional image to be watched at any time, and can be set to regional or full-image three-dimensional display. The two-dimensional image can select the left image or the right image in the three-dimensional left-right image as the display image of the two-dimensional image. In addition to the stereoscopic display mode of the user terminal device, the anchor terminal or the cloud backend control terminal can also be placed with a stereoscopic display device to provide a stereoscopic display preview for the anchor terminal and interactive stereoscopic display with the user terminal. In addition, the user terminal or the live broadcast host can easily select the horizontal or vertical mode for the three-dimensional live broadcast display through the horizontal and vertical control device of the three-dimensional live broadcast image.

為使能更進一步瞭解本發明的特徵及技術內容，請參閱以下有關本發明的詳細說明與圖式，然而所提供的圖式僅用於提供參考與說明，並非用來對本發明加以限制。To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not used to limit the present invention.

10:直播平台10: Live broadcast platform

11:用戶終端11: User terminal

12:直播主12: Live broadcast host

APP:軟體APP: Software

20:影像20: Image

S301-S305:步驟S301-S305: Steps

40:三維影像40: Three-dimensional image

50:直播系統50: Live broadcast system

51:影像擷取模組51: Image capture module

52:三維直播演算模組52: 3D live broadcast calculation module

521:影像校正單元521: Image correction unit

522:影像分離單元522: Image separation unit

523:影像合成單元523: Image synthesis unit

524:影像修正單元524: Image correction unit

525:影像調配單元525: Image adjustment unit

526:影像管理單元526: Image management unit

527:視訊編碼單元527: Video coding unit

53:直播系統伺服器53: Live broadcast system server

54:人工智慧學習模組54: Artificial Intelligence Learning Module

55:三維解碼器55: 3D decoder

56:三維顯示器56: 3D display

71:直播主71: Live Broadcaster

72:行動裝置72: Mobile devices

73:禮物73: Gift

74:顯示螢幕74: Display screen

75:用戶終端75: User terminal

80:三維直播影像的橫向縱向控制裝置80: Horizontal and vertical control device for three-dimensional live image

81:方向偵測器81: Direction detector

82:控制介面82: Control interface

83:直播影像調整器83: Live video adjuster

S901-S905:步驟S901-S905: Steps

圖1為現有直播平台的示意圖。Figure 1 is a schematic diagram of the existing live broadcast platform.

圖2為現有直播軟體執行的示意圖。Figure 2 is a schematic diagram of the execution of existing live broadcast software.

圖3為本發明的即時三維影像顯示功能的直播方法的步驟流程圖。Figure 3 is a flow chart of the steps of the live broadcast method of the instant three-dimensional image display function of the present invention.

圖4A為本發明的三維影像的影像調整的示意圖。FIG4A is a schematic diagram of the image adjustment of the three-dimensional image of the present invention.

圖4B為本發明的人工智慧學習模組學習用戶終端的使用習慣的示意圖。Figure 4B is a schematic diagram of the artificial intelligence learning module of the present invention learning the usage habits of the user terminal.

圖5為本發明的即時三維影像顯示功能的直播系統的系統方塊圖。Figure 5 is a system block diagram of the live broadcast system with real-time three-dimensional image display function of the present invention.

圖6為本發明的三維直播演算模組的影像處理的示意圖。Figure 6 is a schematic diagram of the image processing of the three-dimensional live broadcast calculation module of the present invention.

圖7A為應用本發明的直播系統顯示三維影像的示意圖。Figure 7A is a schematic diagram of a live broadcast system using the present invention to display a three-dimensional image.

圖7B為應用本發明的直播系統使直播主與用戶終端互動的示意圖。Figure 7B is a schematic diagram of the live broadcast system of the present invention being used to enable the live broadcast host to interact with the user terminal.

圖8為本發明三維直播影像控制裝置的方塊圖。Figure 8 is a block diagram of the three-dimensional live image control device of the present invention.

圖9為本發明三維直播影像控制裝置的影像二維與三維直播轉換的判斷步驟流程圖。Figure 9 is a flowchart of the judgment steps for converting two-dimensional and three-dimensional live broadcast images of the three-dimensional live broadcast image control device of the present invention.

以下是通過特定的具體實施例來說明本發明所公開有關“即時三維影像顯示的直播系統”的實施方式，本領域技術人員可由本說明書所公開的內容瞭解本發明的優點與效果。本發明可通過其他不同的具體實施例加以施行或應用，本說明書中的各項細節也可基於不同觀點與應用，在不背離本發明的構思下進行各種修改與變更。另外，本發明的附圖僅為簡單示意說明，並非依實際尺寸的描繪，事先聲明。以下的實施方式將進一步詳細說明本發明的相關技術內容，但所公開的內容並非用以限制本發明的保護範圍。另外，本文中所使用的術語“或”，應視實際情況可能包括相關聯的列出項目中的任一個或者多個的組合。The following is an explanation of the implementation of the "live broadcast system for real-time three-dimensional image display" disclosed in the present invention through specific concrete embodiments. Technical personnel in this field can understand the advantages and effects of the present invention from the contents disclosed in this manual. The present invention can be implemented or applied through other different specific embodiments, and the details in this manual can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. Please note in advance. The following implementation will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used in this document may include any one or more combinations of the associated listed items as the case may be.

[本發明直播方法實施例][Implementation example of the live broadcast method of the present invention]

圖3顯示本發明之即時三維影像顯示的直播方法的步驟流程圖，圖4A為本發明的三維影像的影像調整的示意圖，圖4B為本發明的人工智慧學習模組學習用戶終端的使用習慣的示意圖，圖5為本發明的即時三維影像顯示功能的直播系統的系統方塊圖。FIG3 shows a flow chart of the steps of the live broadcast method of the instant three-dimensional image display of the present invention, FIG4A is a schematic diagram of the image adjustment of the three-dimensional image of the present invention, FIG4B is a schematic diagram of the artificial intelligence learning module of the present invention learning the usage habits of the user terminal, and FIG5 is a system block diagram of the live broadcast system of the instant three-dimensional image display function of the present invention.

如圖3所示，並參考圖5的元件標示，本發明實施例提供一種即時三維影像顯示的直播方法包括下列步驟。在步驟S301中，對一實體進行取像，並獲取實體的一三維影像，應用一影像擷取模組51對直播主進行攝影，影像擷取模組51可以是三維影像機或二維影像機等，任何可以攝影的電子裝置都可以本發明的影像擷取模組51。As shown in FIG3 and referring to the component markings in FIG5, the embodiment of the present invention provides a live broadcast method for real-time three-dimensional image display, which includes the following steps. In step S301, an entity is imaged and a three-dimensional image of the entity is obtained. An image capture module 51 is used to shoot the live broadcaster. The image capture module 51 can be a three-dimensional image machine or a two-dimensional image machine, etc. Any electronic device that can shoot can be the image capture module 51 of the present invention.

在本發明的較佳實施例中，影像擷取模組51為三維攝影機，要解決三維影像顯示訊息的不足，在本發明的較佳實施例中利用了三維影像的拍攝及編制，在此所述的三維影像拍攝，是採用即時的內嵌式三維的雙鏡頭攝影機擷取三維影像。或者，在不同實施例中，可以應用二維攝影機，先取得直播主的二維影像，再透過二維轉三維的影像轉換技術，將二維影像轉換為三維影像，如何擷取三維影像或如何將二維影像轉換為三維影像的技術為本領域具有通常知識者所熟知，在此不再贅述。為減少資料的即時流量，採用影像的二維與三維並存，混合編輯的方法，這樣用戶終端在觀看時可以把焦點放置在需要關注的產品或人物身上，以觀賞到最佳及最需要的三維影像，並且這種二維與三維影像並存的方式，可以在使用者介面上進行選擇設置二維影像觀賞，或是三維影像觀賞，亦即表示用戶終端可以隨時變更需要觀看的三維影像位置，可以設置在區域性，或是全圖三維顯示。另外，在此需要說明的是，三維影像所需傳輸的資料量大於二維影像所需傳輸的資料量，本發明通過在影像中同時包括三維的圖像與二維的圖像的方式，可以在影像撥放時，傳輸相較於全圖三維顯示時較少的資料量，因此，可以維持直播時撥放的順暢度以及降低資料傳輸延遲(lag)的產生。In the preferred embodiment of the present invention, the image capture module 51 is a three-dimensional camera. To solve the insufficiency of the three-dimensional image display information, the preferred embodiment of the present invention utilizes the shooting and editing of three-dimensional images. The three-dimensional image shooting described herein is to use a real-time embedded three-dimensional dual-lens camera to capture three-dimensional images. Alternatively, in different embodiments, a two-dimensional camera can be used to first obtain the two-dimensional image of the live broadcaster, and then convert the two-dimensional image into a three-dimensional image through a two-dimensional to three-dimensional image conversion technology. How to capture a three-dimensional image or how to convert a two-dimensional image into a three-dimensional image is well known to those with ordinary knowledge in this field, and will not be repeated here. In order to reduce the real-time data flow, the two-dimensional and three-dimensional images coexist and are mixed and edited. In this way, the user terminal can focus on the products or people that need to be paid attention to when viewing, so as to view the best and most needed three-dimensional images. In addition, this two-dimensional and three-dimensional image coexistence method can be selected on the user interface to set the two-dimensional image viewing or three-dimensional image viewing, which means that the user terminal can change the position of the three-dimensional image to be viewed at any time, and can be set to regional or full-image three-dimensional display. In addition, it should be noted that the amount of data required to transmit a three-dimensional image is greater than that required to transmit a two-dimensional image. The present invention includes both three-dimensional and two-dimensional images in the image, so that when the image is played, less data can be transmitted compared to the full three-dimensional display. Therefore, the smoothness of the live broadcast can be maintained and the data transmission delay (lag) can be reduced.

在步驟S302中，通過一三維直播演算模組52進行三維影像的影像處理。當本發明的影像擷取模組51接收到三維影像40後，會進行影像處理，因為用戶終端所使用的行動裝置並非相同，且用戶終端所在的位置或環境也不相同，不同的行動裝置會有不同的資料傳輸速度，且所在位置不同，其資料傳輸的速度也不同，因此在三維影像進行優化的步驟中，對三維影像進行切割調整，並根據視頻調配組合，通過人工智慧學習模組54，對三維影像進行優化。In step S302, a three-dimensional live broadcast calculation module 52 performs image processing of the three-dimensional image. When the image capture module 51 of the present invention receives the three-dimensional image 40, it will perform image processing. Because the mobile devices used by the user terminals are not the same, and the locations or environments of the user terminals are also different, different mobile devices will have different data transmission speeds, and the data transmission speeds are also different at different locations. Therefore, in the step of optimizing the three-dimensional image, the three-dimensional image is cut and adjusted, and the three-dimensional image is optimized through the artificialintelligence learning module 54 according to the video matching combination.

本發明的即時三維影像顯示功能的直播方法會根據不同的資料傳輸速度調整影像。在現有的立體視覺技術上，都是將一整幅影像進行處理，因此往往產生至少二倍於二維影像的資料量，這樣的資料傳輸量對於即時傳輸的直播系統產生了相當大的障礙，最常碰到的情況便是影像掉禎(frame)或是延遲嚴重，降低了使用的樂趣及耐心。因此，本發明對於影像進行了自動偵測，並做前景、後景以及影像分割做適配效果外，如圖4A所示，更進一步將三維影像40進行影像容錯及校正，並偵測與分離三維影像40中的多個物體。若物體為人，對物體進行影像柔和濾光、背景虛化、濾鏡效果或美顏美肌等影像處理；若物體為物，對物體進行背景剝離、邊緣強化、增強對比或影像放大等影像處理；若物體同為人與物，對影像進行顏色、美顏、圖形邊緣最適化的計算。背景剝離即是將三維影像40中的背景從影像中分離出來，因此可以在後續的步驟中根據用戶終端的使用習慣替換不同的背景。最後，輸出最佳化、經過影像處理後的三維影像40。The live broadcast method of the instant three-dimensional image display function of the present invention adjusts the image according to different data transmission speeds. In the existing stereoscopic vision technology, an entire image is processed, so the amount of data generated is often at least twice that of a two-dimensional image. Such a data transmission amount creates a considerable obstacle for the real-time live broadcast system. The most common situation is that the image is dropped (frame) or delayed seriously, which reduces the fun and patience of use. Therefore, the present invention automatically detects the image, and performs foreground, background and image segmentation for adaptation effects. As shown in Figure 4A, the three-dimensional image 40 is further subjected to image error tolerance and correction, and multiple objects in the three-dimensional image 40 are detected and separated. If the object is a person, the object is subjected to image processing such as soft light filtering, background blurring, filter effect or beauty and skin beautification; if the object is an object, the object is subjected to image processing such as background stripping, edge enhancement, contrast enhancement or image magnification; if the object is both a person and an object, the image is subjected to color, beauty and graphic edge optimization calculation. Background stripping is to separate the background in the three-dimensional image 40 from the image, so that different backgrounds can be replaced in subsequent steps according to the user's terminal usage habits. Finally, the optimized and processed three-dimensional image 40 is output.

另外，在本發明的直播方法更包括對至少一用戶終端進行取像以獲取用戶終端的三維影像，進而執行多人互動的一立體顯示模式。本發明的直播方法除了可以三維顯示外，更可以使用直播主廣播的一對多的顯示模式，除了一般的一對一模式，還可以多人同時同步互動的立體顯示模式，因此本發明的直播方法除了可以提供直觀性，更可以提高娛樂的樂趣，對於避免購物的錯誤以外，可以增進人類生活的愉悅感。In addition, the live broadcast method of the present invention further includes capturing images of at least one user terminal to obtain a three-dimensional image of the user terminal, thereby executing a stereoscopic display mode for multi-person interaction. In addition to three-dimensional display, the live broadcast method of the present invention can also use a one-to-many display mode of the live broadcast host. In addition to the general one-to-one mode, it can also use a stereoscopic display mode for multiple people to interact synchronously at the same time. Therefore, the live broadcast method of the present invention can not only provide intuitiveness, but also improve the fun of entertainment. In addition to avoiding shopping errors, it can enhance the pleasure of human life.

透過本發明的三維影像的影像處理，除了可以給用戶終端一眼即可見到目標顯示區域，同時對於網路頻寬的資料裝載量可以有效的減少，本發明的直播方法對於即時的直播系統更是可以提高用戶終端的視訊體驗感，才不致產生視頻卡頓現象。在本發明中，三維影像可通過一三維直播演算模組52來達到影像處理的目的。本發明三維影像的影像處理，對網路頻寬的資料裝載量可以有效減少的原因在於，局部的三維影像資料相較於傳統的全圖三維資料量小很多，加上資料壓縮的技術，僅會比一般全圖二維影像的資料多些位元組(bytes)~幾千位元組的資料。Through the image processing of the three-dimensional image of the present invention, in addition to allowing the user terminal to see the target display area at a glance, the data load of the network bandwidth can be effectively reduced. The live broadcast method of the present invention can also improve the video experience of the user terminal for the real-time live broadcast system, so as to avoid the video jamming phenomenon. In the present invention, the three-dimensional image can achieve the purpose of image processing through a three-dimensional live broadcast calculation module 52. The reason why the image processing of the three-dimensional image of the present invention can effectively reduce the data load of the network bandwidth is that the local three-dimensional image data is much smaller than the traditional full-map three-dimensional data. In addition, with the data compression technology, it will only have a few bytes to a few thousand bytes more than the general full-map two-dimensional image data.

在步驟S303中，根據一用戶終端的至少一視頻選擇條件，通過一人工智慧學習模組54，產生一視頻調配組合。用戶終端的至少一視頻選擇條件為用戶終端對一直播系統的使用慣性，每個用戶終端的使用習慣都不相同，本發明的直播方法通過一人工智慧學習模組54，根據每個用戶終端在使用本發明的直播軟體時的使用慣性，顯示用戶終端最喜歡的撥放模式。隨著用戶終端的習慣，亦被人工智慧學習模組54的學習模式所記錄，下次用戶終端在開啟相同或是類似畫面，可以自動的顯示用戶終端習慣的區域或是以全圖的立體畫面顯示。In step S303, a video allocation combination is generated according to at least one video selection condition of a user terminal through an artificialintelligence learning module 54. The at least one video selection condition of the user terminal is the user terminal's habit of using a live broadcast system. The usage habits of each user terminal are different. The live broadcast method of the present invention displays the user terminal's favorite playback mode through an artificialintelligence learning module 54 according to the usage habits of each user terminal when using the live broadcast software of the present invention. As the user terminal develops habits, they are also recorded by the learning mode of the artificialintelligence learning module 54. Next time the user terminal opens the same or similar screen, the area that the user terminal is accustomed to can be automatically displayed or displayed in a three-dimensional screen of the entire image.

進一步來說，每個用戶終端都會有個人的使用習慣，因此，當用戶終端觀看直播時，本發明的人工智慧學習模組54會記錄每個用戶終端在觀看直播時的使用習慣。舉例來說，如圖4B所示，當用戶終端在觀看直播時，人工智慧學習模組54會記錄用戶終端的使用模式，例如，該用戶終端喜歡發送哪種禮物，用戶終端會針對那個特定的人、物或人與物進行二維影像與三維影像之間的切換。本發明的直播方法會記錄每個用戶終端在觀看直播時的使用習慣，並在人工智慧學習模組54中根據所記錄的每個用戶終端的觀看直播的使用習慣，當該用戶終端在下一次觀看直播時，人工智慧學習模組54會根據所記錄的使用習慣，顯示特定的直播畫面給該用戶終端，讓該用戶終端有較佳的直播觀看的使用經驗。Furthermore, each user terminal has individual usage habits. Therefore, when the user terminal watches the live broadcast, the artificialintelligence learning module 54 of the present invention will record the usage habits of each user terminal when watching the live broadcast. For example, as shown in FIG4B , when the user terminal watches the live broadcast, the artificialintelligence learning module 54 will record the usage pattern of the user terminal, for example, what kind of gift the user terminal likes to send, and the user terminal will switch between two-dimensional images and three-dimensional images for that specific person, object, or person and object. The live broadcast method of the present invention will record the usage habits of each user terminal when watching live broadcasts, and in the artificialintelligence learning module 54, based on the recorded usage habits of each user terminal when watching live broadcasts, when the user terminal watches live broadcasts next time, the artificialintelligence learning module 54 will display a specific live broadcast screen to the user terminal based on the recorded usage habits, so that the user terminal has a better live broadcast viewing experience.

另外，在步驟S304中，根據視頻調配組合與直播系統的使用環境，對三維影像進行優化。對三維影像進行優化可以是根據直播系統的使用環境，例如觀眾數量與網路速度等，對三維影像進行優化。另外，在本發明的直播方法中，在拍攝的人物或是產品時，除了即時拍攝，也可以透過預先錄製的後台背景即時加入，增加虛擬現實的使用場景，方便用戶終端藉由使用場景變更，來提高環境使用愉悅體感。在本發明的直播方法中，除了針對每個用戶終端提供不同的視頻調配組合，產生優化後的三維影像外，還可以根據不同的用戶終端選擇不同的背景圖案。而且，本發明的直播方法為避免錯誤或是不恰當的影像或是視頻流放到終端，因此不論用戶終端是與直播主進行聊天或交流，甚至是欣賞直播主的節目表演，或是直播主銷售產品，在影像以相機即時攝入人體或物體以後，除了即時的影像三維編輯壓縮外，都會在直播雲端的後台上，經過人工智慧的自動檢查，或是人工檢查，檢查完成後，才會根據用戶終端的設定或是需求，與後台系統的正確對應後，發送出相對應的碼流(Data Streaming)到用戶終端的手機或是終端交互裝置上，用戶便可以看到該終端所設置及需求相對應的顯示結果。In addition, in step S304, the three-dimensional image is optimized according to the video allocation combination and the use environment of the live broadcast system. The optimization of the three-dimensional image can be based on the use environment of the live broadcast system, such as the number of viewers and the network speed. In addition, in the live broadcast method of the present invention, when shooting people or products, in addition to real-time shooting, the pre-recorded background can be added in real time to increase the use scene of virtual reality, so that the user terminal can improve the pleasant feeling of the environment by changing the use scene. In the live broadcast method of the present invention, in addition to providing different video allocation combinations for each user terminal to generate optimized three-dimensional images, different background patterns can also be selected according to different user terminals. Moreover, the live broadcast method of the present invention is to avoid erroneous or inappropriate images or videos being streamed to the terminal. Therefore, no matter whether the user terminal is chatting or communicating with the live broadcast host, or even enjoying the live broadcast host's performance, or the live broadcast host is selling products, after the image is captured by the camera in real time on the human body or object, in addition to the real-time three-dimensional image editing and compression, it will be automatically checked by artificial intelligence or manually checked in the background of the live broadcast cloud. After the inspection is completed, the corresponding code stream (Data Streaming) will be sent to the user terminal's mobile phone or terminal interactive device after the correct correspondence with the background system according to the user terminal's settings or needs, and the user can see the display results corresponding to the terminal settings and needs.

在步驟S305中，優化後的三維影像，並配合用戶終端的至少一視頻選擇條件，將對應的三維影像顯示在用戶終端中。本發明的直播方法除了根據用戶終端的設置，在後台端提供用戶終端所需要的前述前景、後景的切割及顯示設置以外，亦會根據用戶終端的設置及需求，將相對應的三維影像發送到用戶終端的裝置上。舉例來說，用戶終端進行了禮物的購買支付，直播系統便會根據用戶終端購買的對應禮物，將該禮物的立體顯示影像或視頻，發送到直播主的裝置上，這樣直播主便可以在其裝置的顯示螢幕上，觀看到對應的立體禮物的視頻或影像。In step S305, the optimized three-dimensional image is matched with at least one video selection condition of the user terminal, and the corresponding three-dimensional image is displayed in the user terminal. In addition to providing the aforementioned foreground and background cutting and display settings required by the user terminal at the backend according to the settings of the user terminal, the live broadcast method of the present invention also sends the corresponding three-dimensional image to the device of the user terminal according to the settings and requirements of the user terminal. For example, when a user terminal pays for a gift, the live broadcast system will send the 3D display image or video of the corresponding gift purchased by the user terminal to the live broadcast host's device, so that the live broadcast host can watch the corresponding 3D gift video or image on the display screen of his device.

通過本發明即時三維影像顯示功能的直播方法，除了可以根據不同的資料傳輸速度可以調整影像的輸出品質，還可以根據不同的用戶終端提供不同的體驗感受，讓每個用戶終端都獲得較佳的觀看經驗或購物經驗。Through the live broadcast method of the present invention with real-time three-dimensional image display function, in addition to being able to adjust the image output quality according to different data transmission speeds, it can also provide different user experience feelings according to different user terminals, so that each user terminal can obtain a better viewing experience or shopping experience.

[本發明直播系統實施例][Implementation example of the live broadcast system of the present invention]

圖5顯示本發明即時三維影像顯示的直播系統的系統方塊圖。如圖5所示，本發明之即時三維影像顯示的直播系統50包括一影像擷取模組51、一三維直播演算模組52、一直播系統伺服器53、一人工智慧學習模組54、一三維解碼器55與一三維顯示器56。FIG5 shows a system block diagram of the live broadcast system for real-time three-dimensional image display of the present invention. As shown in FIG5, the live broadcast system 50 for real-time three-dimensional image display of the present invention includes an image capture module 51, a three-dimensional live broadcast calculation module 52, a live broadcast system server 53, an artificialintelligence learning module 54, a three-dimensional decoder 55 and a three-dimensional display 56.

影像擷取模組51可以是三維影像機或二維影像機等，任何可以攝影的電子裝置都可以本發明的影像擷取模組51。在本發明的較佳實施例中，影像擷取模組51為三維攝影機，要解決三維影像顯示訊息的不足，在本發明的較佳實施例中利用了三維影像的拍攝及編制，在此所述的三維影像拍攝，是採用即時的內嵌式三維雙鏡頭攝影機擷取三維影像，這種內嵌於主播主所使用裝置內或是用戶終端的裝置(例如智能手機、平板電腦等)內的攝影機，由於是內嵌式，所以相機是內置在電子裝置上，兩個三維攝影機透過介面，如移動工業處理器介面(Mobile Industry Process Interface，MIPI)、串列介面等，與手持裝置端直接連接，再透過本發明的三維直播演算模組52及直播系統伺服器53連接，即時發送三維影像及視頻至用戶終端。對於個人電腦端或手持裝置，假如沒有裝設內嵌三維攝影機，可以採用外接式的單眼攝影機，或是雙眼立體相機，透過通用序列匯流排(USB)介面，或是無線(WIFI)方式，與主機連接，當連接到本發明的直播系統後，便根據二維轉三維或是立體相機的處理方式或流程進行資料處理。The image capture module 51 can be a three-dimensional imager or a two-dimensional imager, etc. Any electronic device that can take pictures can be the image capture module 51 of the present invention. In the preferred embodiment of the present invention, the image capture module 51 is a 3D camera. To solve the problem of the lack of 3D image display information, the preferred embodiment of the present invention utilizes the shooting and editing of 3D images. The 3D image shooting described herein uses a real-time embedded 3D dual-lens camera to capture 3D images. This camera is embedded in the device used by the anchor or the user's terminal device (such as a smart phone, tablet computer, etc.). Since it is embedded, the camera is built into the electronic device. The two 3D cameras are connected through an interface, such as a mobile industry processor interface (MIPI). Interface, MIPI), serial interface, etc., directly connected to the handheld device end, and then connected through the 3D live broadcast calculation module 52 and the live broadcast system server 53 of the present invention, and the 3D image and video are sent to the user terminal in real time. For personal computers or handheld devices, if there is no built-in 3D camera, an external SLR camera or binocular stereo camera can be used to connect to the host through a universal serial bus (USB) interface or wireless (WIFI). When connected to the live broadcast system of the present invention, data processing is performed according to the processing method or process of 2D to 3D or stereo camera.

或者，在不同實施例中，可以應用二維攝影機，先取得直播主的二維影像，再透過二維轉三維的影像轉換技術，將二維影像轉換為三維影像，如何擷取三維影像或如何將二維影像轉換為三維影像的技術為本領域具有通常知識者所熟知，在此不再贅述。影像擷取模組51可以安裝在直播主以及用戶終端的裝置上，可以同時取得直播主以及用戶終端的三維影像，以便於進行後續的直播主與至少一用戶終端的一對一模式或一對多的立體顯示模式。Alternatively, in different embodiments, a two-dimensional camera may be used to first obtain a two-dimensional image of the live broadcaster, and then convert the two-dimensional image into a three-dimensional image through a two-dimensional to three-dimensional image conversion technology. How to capture a three-dimensional image or how to convert a two-dimensional image into a three-dimensional image is well known to those with ordinary knowledge in this field and will not be repeated here. The image capture module 51 may be installed on the device of the live broadcaster and the user terminal, and may simultaneously obtain the three-dimensional images of the live broadcaster and the user terminal, so as to facilitate the subsequent one-to-one mode or one-to-many stereoscopic display mode between the live broadcaster and at least one user terminal.

三維直播演算模組52連接影像擷取模組51，其用於接收三維影像，並對三維影像進行優化，三維直播演算模組52可以是設置直播主或用戶終端的直播軟體中，或者三維直播演算模組52也可以安裝在直播系統伺服器53中，在此並不侷限。進一步來說，如圖6所示，並參考圖5，三維直播演算模組52包括一影像校正單元521、一影像分離單元522、一影像合成單元523、一影像修正單元524、一影像調配單元525、一影像管理單元526以及一視訊編碼單元527。影像校正單元521連接影像擷取模組51，接收三維影像，用於將所接收的三維影像做影像的校正，在擷取三維影像或影像在進行二維轉三維的過程中，都會產生些許的影像誤差或影音不同步的問題，通過影像校正單元521修正三維影像在擷取時或轉換時所產生的錯誤。影像分離單元522連接影像校正單元521，針對校正後的三維影像中的前景、後景、人或物進行分割，因此可以對於三維影像中的任何人或物進行特定的影像處理，舉例來說，將三維影像中的後景切割，因此可以在後續的影像處理中，在背景中加入特定的背景圖案，或者，在特定的直播動作中，將特定的物或影像進行三維顯示。The 3D live broadcast calculation module 52 is connected to the image capture module 51, and is used to receive and optimize the 3D image. The 3D live broadcast calculation module 52 can be set in the live broadcast software of the live broadcast host or the user terminal, or the 3D live broadcast calculation module 52 can also be installed in the live broadcast system server 53, which is not limited here. Further, as shown in FIG6 and referring to FIG5, the 3D live broadcast calculation module 52 includes an image correction unit 521, an image separation unit 522, an image synthesis unit 523, an image correction unit 524, an image adjustment unit 525, an image management unit 526 and a video encoding unit 527. The image correction unit 521 is connected to the image capture module 51, receives the three-dimensional image, and is used to correct the received three-dimensional image. In the process of capturing the three-dimensional image or converting the image from two dimensions to three dimensions, there will be some image errors or audio and video synchronization problems. The image correction unit 521 corrects the errors generated when capturing or converting the three-dimensional image. The image separation unit 522 is connected to the image correction unit 521, and is used to segment the foreground, background, people or objects in the corrected three-dimensional image. Therefore, any person or object in the three-dimensional image can be processed specifically. For example, the background in the three-dimensional image can be cut off, so that in the subsequent image processing, a specific background pattern can be added to the background, or, in a specific live action, a specific object or image can be displayed in three dimensions.

影像合成單元523連接影像分離單元522，分離後的三維影像，可以分別對於特定的圖案、人或物進行影像處理，處理後的特定圖案、人或物可以通過影像合成單元523進行結合，影像合成單元523並非只是將原本的三維影像還原，而是可以將個別立體化的圖案、人或物與其他的二維影像結合，更可以通過影像合成單元523在背景中加入特定的背景圖案。影像修正單元524連接影像調配單元525，影像修正單元524會根據不同圖案的取像選擇調整影像的視角，或者，影像修正單元524可以將三維影像中的禮物影像(即特定影像)進行影像特效渲染效果，影像修正單元524也包括一般的影像邊緣強化、影像轉向、影像背景虛化、濾鏡效果或影像區域放大等功能。The image synthesis unit 523 is connected to the image separation unit 522. The separated three-dimensional image can be processed for specific patterns, people or objects respectively. The processed specific patterns, people or objects can be combined through the image synthesis unit 523. The image synthesis unit 523 does not just restore the original three-dimensional image, but can combine individual three-dimensional patterns, people or objects with other two-dimensional images. The image synthesis unit 523 can also add specific background patterns to the background. The image correction unit 524 is connected to the image adjustment unit 525. The image correction unit 524 will adjust the image viewing angle according to the image selection of different patterns, or the image correction unit 524 can render the gift image (i.e., the specific image) in the three-dimensional image with image special effects. The image correction unit 524 also includes general image edge enhancement, image rotation, image background blurring, filter effect, or image area magnification and other functions.

影像調配單元525連接影像修正單元524，影像調配單元525會根據不同的網速或移動裝置的性能，調整三維影像的影像輸出。舉例來說，當網速較快，三維影像可以完整的輸出，當網速較慢時，可以局部顯示三維影像，其餘的影像以二維顯示。影像管理單元526連接影像調配單元525，將經過影像處理後的三維影像整合並輸出，視訊編碼單元527連接影像管理單元526，通過視訊編碼單元527將三維影像轉換為資料訊號，進而可將具有三維影像的資料訊號以有線或無線的方式傳送至直播系統伺服器53。The image adjustment unit 525 is connected to the image correction unit 524. The image adjustment unit 525 adjusts the image output of the three-dimensional image according to different network speeds or the performance of the mobile device. For example, when the network speed is fast, the three-dimensional image can be fully output. When the network speed is slow, the three-dimensional image can be partially displayed, and the rest of the image is displayed in two dimensions. The image management unit 526 is connected to the image adjustment unit 525 to integrate and output the three-dimensional image after image processing. The video encoding unit 527 is connected to the image management unit 526. The three-dimensional image is converted into a data signal through the video encoding unit 527, and the data signal with the three-dimensional image can be transmitted to the live broadcast system server 53 in a wired or wireless manner.

直播系統伺服器53無線連接三維直播演算模組52，其也可稱之為直播系統雲端，直播系統伺服器53用於接收具有三維影像的資料訊號，換句話說，每個直播主所直播的內容都會傳送至直播系統伺服器53，然後再透過直播系統伺服器53推播至用戶終端的裝置上。人工智慧學習模組54連接直播系統伺服器53，在本發明的較佳實施例中，人工智慧學習模組54可以設置在直播系統伺服器53，或者在不同實施例中，人工智慧學習模組54可以設置在不同的伺服器或電腦主機上，然後再以無線或有線連接的方式連接直播系統伺服器53，在此並不侷限。每個用戶終端的視屏選擇條件或稱使用習慣都會被人工智慧學習模組54的學習模式所記錄，並輸出一視頻調配組合。當下次用戶終端開啟相同或是類似畫面，人工智慧學習模組54可根據視頻選擇條件自動輸出視頻調配組合，換句話說，顯示用戶終端所習慣的顯示區域或是以全圖的立體畫面顯示。The live broadcast system server 53 is wirelessly connected to the three-dimensional live broadcast calculation module 52, which can also be called the live broadcast system cloud. The live broadcast system server 53 is used to receive data signals with three-dimensional images. In other words, the content broadcast by each live broadcaster will be transmitted to the live broadcast system server 53, and then pushed to the user's terminal device through the live broadcast system server 53. The artificialintelligence learning module 54 is connected to the live broadcast system server 53. In the preferred embodiment of the present invention, the artificialintelligence learning module 54 can be set in the live broadcast system server 53, or in different embodiments, the artificialintelligence learning module 54 can be set on different servers or computer hosts, and then connected to the live broadcast system server 53 by wireless or wired connection, which is not limited here. The video selection conditions or usage habits of each user terminal will be recorded by the learning mode of the artificialintelligence learning module 54, and a video matching combination will be output. The next time the user terminal opens the same or similar screen, the artificialintelligence learning module 54 can automatically output the video matching combination according to the video selection conditions. In other words, the display area that the user terminal is accustomed to or the full-image stereoscopic display is displayed.

因此，不論是用戶終端與直播主進行聊天、交流，或是欣賞直播主的節目表演，或是觀看直播主銷售產品，在影像以相機即時攝入人體或物體以後，除了即時的影像三維編輯、壓縮以外，都會在直播系統伺服器53的後台上，經過人工智慧學習模組54的自動檢查，檢查完成後，才會根據用戶終端的設定或是需求，與後台系統的正確對應後，發送出相對應的碼流(Data Streaming)到用戶終端的手機或是終端交互裝置上，用戶終端便可以看到該終端所設置及需求相對應的顯示結果。Therefore, whether the user terminal is chatting and communicating with the live broadcast host, or enjoying the live broadcast host's performance, or watching the live broadcast host sell products, after the image is captured by the camera in real time on the human body or object, in addition to real-time three-dimensional image editing and compression, it will be automatically checked by the artificialintelligence learning module 54 in the background of the live broadcast system server 53. After the inspection is completed, it will be sent to the user terminal's mobile phone or terminal interactive device according to the user terminal's settings or needs. After the correct correspondence with the background system, the corresponding code stream (Data Streaming) will be sent, and the user terminal can see the display results corresponding to the terminal settings and needs.

三維解碼器55連接直播系統伺服器53，或者三維解碼器55也可以安裝在直播系統伺服器53上，且連接人工智慧學習模組54，透過三維解碼器55可將三維影像進行編碼以及解碼，在本發明的直播系統50中，可以應用三維解碼器55讓直播主可以跟多個用戶終端進行互動的立體顯示模式。直播系統50便會根據用戶終端購買的對應禮物，如圖7A所示，在直播主71的行動裝置72上，將禮物73的立體顯示影像或視頻顯示出來，這樣直播主71便可以在其行動裝置72的顯示螢幕74上，觀看到對應的立體禮物73的視頻或影像。三維顯示器56設置在直播主以及用戶終端的裝置上，直播主以及用戶終端的裝置以無線傳輸的方式接收從直播系統伺服器53所推播的三維影像，並透過三維顯示器56顯示在直播主以及用戶終端的裝置上。透過三維解碼器55讓本發明的直播系統50具有三維立體雙向編解碼技術，如圖7B所示，在行動裝置72上，除了直播主71可以發送立體視頻、圖像，給用戶終端75接收觀看立體視頻、圖像以外，用戶終端75也可以透過雙向立體顯像技術，對直播主71發送立體圖像或是視頻，使直播主71也可以即時收到用戶終端75的立體圖像訊息，可以快速的判斷出用戶終端75的需求，達成即時互動的目的。The 3D decoder 55 is connected to the live broadcast system server 53, or the 3D decoder 55 can also be installed on the live broadcast system server 53 and connected to the artificialintelligence learning module 54. The 3D decoder 55 can encode and decode the 3D image. In the live broadcast system 50 of the present invention, the 3D decoder 55 can be applied to allow the live broadcast host to interact with multiple user terminals in a three-dimensional display mode. The live broadcast system 50 will display the three-dimensional display image or video of the gift 73 on the mobile device 72 of the live broadcast host 71 according to the corresponding gift purchased by the user terminal, as shown in FIG. 7A, so that the live broadcast host 71 can watch the corresponding video or image of the three-dimensional gift 73 on the display screen 74 of its mobile device 72. The three-dimensional display 56 is set on the device of the live broadcast host and the user terminal. The device of the live broadcast host and the user terminal receives the three-dimensional image pushed from the live broadcast system server 53 by wireless transmission, and displays it on the device of the live broadcast host and the user terminal through the three-dimensional display 56. Through the three-dimensional decoder 55, the live broadcast system 50 of the present invention has three-dimensional stereoscopic two-way encoding and decoding technology. As shown in FIG7B, on the mobile device 72, in addition to the live broadcast host 71 being able to send three-dimensional videos and images to the user terminal 75 for viewing, the user terminal 75 can also send three-dimensional images or videos to the live broadcast host 71 through two-way stereoscopic display technology, so that the live broadcast host 71 can also receive the three-dimensional image information of the user terminal 75 in real time, and can quickly determine the needs of the user terminal 75, achieving the purpose of real-time interaction.

另外，請參閱圖6，在用戶終端上，同樣包括在直播主端的三維影像的顯示功能，在用戶終端的裝置上，同樣包括本發明之直播系統50的功能，因為在觀看直播時，用戶終端也可以直播自己的影像給直播主看，或者，用戶終端也可以在接收到直播主端的視頻時，自動或手動切換二維與三維之間的轉換。因此，當用戶終端接收到二維或三維影像時，若接收到二維影像可轉換為三維影像，用戶終端的三維影像同樣會通過影像校正單元521、影像分離單元522、影像合成單元523、影像修正單元524、影像調配單元525以及影像管理單元526等元件，將三維影像進行影像處理，讓在用戶終端的三維影像同樣可以做全景三維影像的顯示，或者可以針對特定的人、物或人與物做三維影像的顯示，二維或三維影像的顯示在客戶終端上可以主動或手動的方式進行切換。如何通過三維直播演算模組52進行影像處理已於前面章節介紹過，因此，相關的影像處理細節在此不再贅述。In addition, please refer to Figure 6. The user terminal also includes the display function of the three-dimensional image of the live broadcast host. The device of the user terminal also includes the function of the live broadcast system 50 of the present invention. Because when watching the live broadcast, the user terminal can also broadcast its own image to the live broadcast host, or the user terminal can automatically or manually switch between two-dimensional and three-dimensional when receiving the video from the live broadcast host. Therefore, when the user terminal receives a two-dimensional or three-dimensional image, if the received two-dimensional image can be converted into a three-dimensional image, the three-dimensional image of the user terminal will also be processed by components such as the image correction unit 521, the image separation unit522, the image synthesis unit 523, the image correction unit 524, the image allocation unit 525 and the image management unit 526, so that the three-dimensional image at the user terminal can also be displayed as a panoramic three-dimensional image, or can be displayed as a three-dimensional image for a specific person, object or person and object. The display of two-dimensional or three-dimensional images can be switched actively or manually on the client terminal. How to perform image processing through the three-dimensional live broadcast calculation module 52 has been introduced in the previous chapter, so the relevant image processing details are not repeated here.

舉例來說，當用戶終端的資料傳輸速度不快，三維直播演算模組52會自動只顯示部分的三維影像在用戶終端的三維顯示器56上，而不會顯示全景的三維影像，或者，當網速過低時，三維直播演算模組52甚至會自動將三維影像切換至二維影像。另外，用戶終端的三維影像除了可以自動切換外，三維影像也可以手動切換，用戶終端可以手動指定特定的人、物或人與物做三維影像的顯示。換句話說，本發明的三維影像並不局限於全圖的三維影像，本發明的三維影像也可以是局部的三維影像，所謂局部的三維影像就是在影像中，特定的物件(人、物或人與物)是以三維顯示的方式呈現，其餘的圖像則是以二維顯示的方式呈現。通過這樣的影像呈現方式，可以降低在直播時資料的傳輸量，並可以降低直播顯示延遲的產生機率。For example, when the data transmission speed of the user terminal is not fast, the 3D live broadcast calculation module 52 will automatically display only part of the 3D image on the 3D display 56 of the user terminal, and will not display the full 3D image. Or, when the network speed is too low, the 3D live broadcast calculation module 52 will even automatically switch the 3D image to a 2D image. In addition, in addition to the automatic switching of the 3D image of the user terminal, the 3D image can also be manually switched. The user terminal can manually specify a specific person, object, or person and object for the 3D image display. In other words, the three-dimensional image of the present invention is not limited to the three-dimensional image of the whole picture. The three-dimensional image of the present invention can also be a local three-dimensional image. The so-called local three-dimensional image is that in the image, specific objects (people, objects, or people and objects) are presented in a three-dimensional display mode, and the rest of the image is presented in a two-dimensional display mode. Through such an image presentation method, the amount of data transmission during live broadcast can be reduced, and the probability of live broadcast display delay can be reduced.

[本發明三維直播影像的橫向或者縱向顯示實施例][Horizontal or vertical display implementation example of the three-dimensional live broadcast image of the present invention]

另外，在本發明的較佳實施例中，更可以在直播主以及用戶終端的裝置上設置一三維直播影像的橫向縱向控制裝置80。因應用戶終端使用的裝置可能有習慣性的使用縱向顯示觀賞或是橫向直播顯示觀賞，本發明也提供三維直播影像的橫向縱向控制裝置80，對於移動裝置的顯示方向設定，這個直播顯示方向的設定可以在用戶終端的APP軟體上自動偵測或是人工設定，也適用在直播主的顯示終端及雲端的監測顯示上。因此，本發明的三維直播影像的橫向縱向控制裝置80包括一方向偵測器81與一控制介面82。In addition, in the preferred embodiment of the present invention, a horizontal andvertical control device 80 for three-dimensional live broadcast images can be set on the devices of the live broadcast host and the user terminal. In response to the fact that the device used by the user terminal may have the habit of using vertical display viewing or horizontal live broadcast display viewing, the present invention also provides a horizontal andvertical control device 80 for three-dimensional live broadcast images. For the display direction setting of the mobile device, this live broadcast display direction setting can be automatically detected or manually set on the APP software of the user terminal, and is also applicable to the display terminal of the live broadcast host and the monitoring display of the cloud. Therefore, the horizontal andvertical control device 80 for three-dimensional live broadcast images of the present invention includes adirection detector 81 and acontrol interface 82.

在自動偵測直播顯示方向上，方向偵測器81具備可以感測行動裝置或手持裝置內部的陀螺儀或是方向感測器的訊號，在直播主或用戶終端的裝置便會根據終端裝置回饋的方向訊號，對於直播顯示的方向進行調變。這個方向的調變會配合三維直播影像或視頻經過轉向後，進行三維直播影像的顯示轉換，這個顯示轉換包括螢幕顯示的長寬比或交織的配比(Interlace)的調整，以及軟體按鍵與功能顯示位置的調整，諸如此類跟3D立體直播顯示相關的轉向設置。前述的設定也包括了對於三維顯示器56的直播顯示功能設定。控制介面82設置在直播主與用戶終端的行動裝置或手持裝置上，進一步來說，控制介面82可以是控制器如一按鍵等，或者控制介面82也可以是一軟體使用介面，其包括控制器的功能。直播主或用戶終端可以透過方向偵測器81自動翻轉螢幕直播顯示，或者也可以透過控制介面82自行翻轉螢幕直播顯示，達到縱向直播顯示觀賞或是橫向直播顯示觀賞。In the automatic detection of the live broadcast display direction, thedirection detector 81 is capable of sensing the signal of the gyroscope or direction sensor inside the mobile device or handheld device, and the device at the live broadcast host or user terminal will adjust the direction of the live broadcast display according to the direction signal fed back by the terminal device. This direction adjustment will be coordinated with the three-dimensional live broadcast image or video after turning, and the display conversion of the three-dimensional live broadcast image will be performed. This display conversion includes the adjustment of the aspect ratio or interlace ratio of the screen display, as well as the adjustment of the software buttons and function display positions, and other such steering settings related to 3D stereoscopic live broadcast display. The aforementioned settings also include the live broadcast display function settings for the three-dimensional display 56. Thecontrol interface 82 is set on the mobile device or handheld device of the live broadcast host and the user terminal. Specifically, thecontrol interface 82 can be a controller such as a button, or thecontrol interface 82 can also be a software user interface, which includes the function of the controller. The live broadcast host or the user terminal can automatically flip the screen live display through thedirection detector 81, or can also automatically flip the screen live display through thecontrol interface 82 to achieve vertical live display viewing or horizontal live display viewing.

本發明的三維直播影像的橫向縱向控制裝置80具備有橫向與縱向兩方向皆可以顯示三維直播影像的功能，且具備自動或是人工轉向偵測及顯示調適。對於只有支持單方向的立體顯示終端，本發明在三維直播影像的橫向縱向控制裝置80更包括一直播影像調整器83，直播影像調整器83連接方向偵測器81，直播影像調整器83的判斷是否調整三維直播顯示的步驟可以如圖9所示，在步驟S901中，方向偵測器81偵測到行動裝置的擺設方向改變，或者，在步驟S902中，使用者通過控制介面82進行直播影像旋轉，在步驟S903中，直播影像調整器83判斷行動裝置是否橫向或縱向支持三維直播顯示，若否，在步驟S904中，直播影像調整器83可以提醒用戶終端，告知不支持轉向後的立體直播顯示，並透過直播影像調整器83改以二維的平面直播顯示方式在用戶終端的顯示器上，直到用戶終端再次轉向到可以顯示立體直播顯示，用戶終端便可以看到立體直播顯示。反之，在步驟S905中，直播影像調整器83調整轉向後的最佳化的三維直播影像。The lateral andlongitudinal control device 80 of the three-dimensional live broadcast image of the present invention has the function of displaying the three-dimensional live broadcast image in both the lateral and longitudinal directions, and has automatic or manual turning detection and display adjustment. For a stereo display terminal that only supports one direction, the lateral andlongitudinal control device 80 of the three-dimensional live broadcast image of the present invention further includes a live broadcast image adjuster 83. The live broadcast image adjuster 83 is connected to thedirection detector 81. The steps of the live broadcast image adjuster 83 to determine whether to adjust the three-dimensional live broadcast display can be shown in Figure 9. In step S901, thedirection detector 81 detects that the placement direction of the mobile device has changed, or, in step S902, the user controls the live broadcast image through thecontrol interface 82. In step S903, the live image adjuster 83 determines whether the mobile device supports three-dimensional live display in horizontal or vertical orientation. If not, in step S904, the live image adjuster 83 can remind the user terminal that the stereoscopic live display after the rotation is not supported, and the live image adjuster 83 changes to a two-dimensional plane live display on the user terminal display until the user terminal rotates again to display the stereoscopic live display, and the user terminal can see the stereoscopic live display. On the contrary, in step S905, the live image adjuster 83 adjusts the optimized three-dimensional live image after the rotation.

[實施例的有益效果][Beneficial effects of the embodiment]

本發明的其中一有益效果在於，本發明所提供的直播系統與直播方法，為減少資料的即時流量，採用影像的二維與三維並存，混合編輯的方法，這樣觀看者可以把焦點放置在需要關注的產品或人物身上，以觀賞到最佳及最需要的三維影像，並且這種二維與三維影像並存的方式，可以在使用者介面上進行選擇設置二維影像觀賞，或是三維影像觀賞，亦即表示使用者可以隨時變更需要觀看的三維影像位置，可以設置在區域性，或是全圖三維顯示。二維影像可以選擇三維左右式影像中的左影像或是右影像作為二維影像的顯示圖。另外，除了用戶終端裝置的立體顯示方式外，主播端或是雲端後台的控制端，也可以放置立體顯示裝置，以提供主播端的立體顯示預覽，及與用戶終端的互動立體顯示。另外，更可以讓用戶終端或直播主端通過三維直播影像的橫向縱向控制裝置可以容易選擇橫放或縱放的方式來進行三維直播顯示。One of the beneficial effects of the present invention is that the live broadcast system and live broadcast method provided by the present invention adopts a method of coexistence of two-dimensional and three-dimensional images and mixed editing to reduce the real-time flow of data, so that viewers can focus on the products or people that need to be paid attention to, so as to watch the best and most needed three-dimensional images, and this coexistence of two-dimensional and three-dimensional images can be selected and set on the user interface to watch two-dimensional images or three-dimensional images, that is, the user can change the position of the three-dimensional image to be watched at any time, and can be set to regional or full-image three-dimensional display. The two-dimensional image can select the left image or the right image in the three-dimensional left-right image as the display image of the two-dimensional image. In addition to the stereoscopic display mode of the user terminal device, the anchor terminal or the cloud backend control terminal can also be placed with a stereoscopic display device to provide a stereoscopic display preview for the anchor terminal and interactive stereoscopic display with the user terminal. In addition, the user terminal or the live broadcast host can easily select horizontal or vertical mode for three-dimensional live broadcast display through the horizontal and vertical control device of the three-dimensional live broadcast image.

以上所公開的內容僅為本發明的優選可行實施例，並非因此侷限本發明的申請專利範圍，所以凡是運用本發明說明書及圖式內容所做的等效技術變化，均包含於本發明的申請專利範圍內。The above disclosed contents are only the preferred feasible embodiments of the present invention, and do not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made by using the contents of the description and drawings of the present invention are included in the scope of the patent application of the present invention.