SYSTEM AND METHOD TO CONSUME WEB CONTENT USING TELEVISION SET
TECHNICAL FIELD
The invention relates to the Digital Video Broadcasting (DVB) television ecosystem More particularly, it relates to an interactive television infrastructure that enables TV users to ask, search, choose and consume internet content using a TV set, a remote Control or a voice recognition system, and a decoder Or set top box The invention relates to a client server infrastructure (like in Figure 1) comprising - as client - an hybrid decoder/set-top box receiver allowing broadcasting receiving (e g cable, satellite, terrestrial or IP) as well as accessing to the Internet network Through a broadband IP connection, and — as server - a xypicai Internet application server (web server and J2EE or equivalent application server) managing the access to the multitude of web-based applications, databases and services. In the presented architecture the set top boxes could be substituted by a mini-pc.
The invention also relates xo a user interface for a consumer eiecxronic product such as a television. The presented user interface allows a bimodal coexistence of the broadcasting ecosystem with an IP interactive environment (like in Figure 3). An additional interactive window enables to serach web-content displaying retrieved information and functions in images or objects inside a virtual environment making the navigation compatible with the television (image based and fluent) and zapping (scrolling) paradigms, and making the presented web content consumables using a normal remote control. In particular the invention relates to a system to retrieve a list of data and related services or events on this data from the internet making this content interactive and consumables by TV users Data can be common TV channels available within the broadcasting environment or any other content available within any web-based database. Interactivity with this data goes from the search and choose of the right information to the scrolling of the selected items and its consumption. BACKGROUND ART
Television viewing is a popular form of entertainment. Developments in television and digital video technologies allow viewers to watch a wide variety of high- resolution content and to record programs for later viewing.
While television programming provides a variety of information and content, it is largely bound by program scheduling. The Internet offers users an alternative source for many types of information, such as movies, music, commercial products, weather, financial information, etc. Moreover, users can access any type of web-based content at " any time. Nonetheless, users must often interrupt their television viewing in order to access the Internet via a computing device. Additionally, accessing a preferred type of web-based content regularly can require users to spend time repeating navigation steps.
The increasing complexity of consumer electronic products such as televisions and of the systems in which these products are incorporated (e.g., cable and satellite television systems with hundreds of channels) make it more difficult for users to use the products and take full advantage of the functionality that these products provide. Operations of these products that in the past were relatively straightforward and simple have become more difficult. Q For example, it can be difficult to channel "surf or "browse" to find programs of interest when a television receives hundreds of channels. In addition, with the advent of digital channels, even the task of simply tuning to a channel can involve entering a channel and can take up to five or more key presses on a keypad of a remote control or a front panel. Thus, surfing from one channel to another by entering different5 channel numbers is time-consuming and prone to error since so many numbers must be entered to surf to a series of different channels. When there are hundreds of channels and each channel may involve both channel numbers, even remembering which channels to surf to can be a difficult task.
This problems are more urgent if, additionally to existing TV channels, new0 kind of content is made available to TV users, particularly if web-content is made available within the television ecosystem. There is a constantly growth of Internet content that users would like to consume using TV set. TV and radio networks, already started putting more content on the Web — audio, video, etc. — and this allows people to choose what they wanted to watch using PCs and Web browsers. TV networks are doing what millions of other Web sites already do: offering content for people to choose from. There is conventional content today distributed over the Internet under the form of text, which interaction do not fit well with television interaction paradigms.
There's a good chance you have two connections coming into your home: One for television and one for the Internet. Your TV set is hooked up to one connection, and your cable/satellite provider gives you a list of channels to watch. A limited list. Your computer is hooked up to the other connection; your provider (maybe even the same company that provides your TV signal) lets you access an unlimited "list" of Web sites.
Video on the Internet is likely in the same format as what you get over a satellite or digital cable connection. It's called MPEG-2, and it's the same one used on DVDs as well. In other words, that same Internet connection you're using to view this Web page could be used to bring you full-size, DVD-quality video. In fact, some phone companies have acquired cable TV licenses and are delivering "IPTV" — digital television carried over phone lines, using Internet technology. It's indistinguishable from cable or satellite, except that the line coming into your living room looks like a phone cord, not a TV cable.
There's IPTV; so the technology exists to use the Internet infrastructure to carry television. There are faster and faster data pipes coming into your home. There's incredibly cheap storage; there are services that will let you download movies to watch on your PC. There are Media Center PCs, that let you watch and record television shows on your computer. There are legacy systems that let you interact with selected content in a predefined way.
Those are small steps to the on-demand finish line. A larger one is Microsoft's Media Center Extender Set-top Box. It connects your television to your PC, so you can not only watch TV networks, channels and broadcasting shows; you can also access the music, photos, and video that are on your computer.
Concerning the usability, in today television environment the capacity to perform search operation over general content is almost unavailable, while the user ^ satisfaction related to paradigms for moving between channels diminishes when the number of available channels augments. TV usage today is still a frustrating experience: TV as it own timing, Program Guides difficult to find, fragmented search features, difficult video recording programming, broadcaster video on demand offer difficult to find and buy.
10
Distributing multimedia content over Internet networks is based on the streaming technology. Streaming media technology market is still a chaos today. If an Internet user clicks on a video to watch it, there is no guarantee that he can watch it as he can easily watch a TV show after turning on a TV. Mostly due to the lack of local resources allocations, Video streaming is still a comparatively new technology. A video encoded in one technology usually cannot be played by another player, due to lack of content adaptation and uses awareness techniques. Technical architecture and low cost requirements of the set top box do not allow to integrate inside a unique Set Top Box different streaming players/viewers.
20
SUMMARY OF THE INVENTION
Accordingly, the invention seeks to provide an improved system for the consumption of web content using a TV set, and preferably seeks to mitigate, alleviate 25 or eliminate one or more of the above disadvantages singly or in any combination.
According to a first aspect of the invention, there is an interactive television system allowing TV users to search, choose and consume internet content using a TV set, a remote control and a set top box, where the user does not need to read text to completely understand the data retrieved and to interact with, but rather need only look 3Q at a three-dimensional animated landscape containing the complete answer in graphical form. Presented, therefore is a computer system and method for transforming data, technical environments and paradigms enabling the coexistence of the broadcasting television consumption with the interaction with internet services, within a unique user interface possible. Content can be classical TV content: channels, passed programmes, shows, movies, news, live broadcasting programs or future broadcasting offers. This content is made available searching real-time inside internet based electronic program guides (EPG). Passed shows and productions produced by TV networks are made available though the Internet network and consumed though a streaming flow. Present programs are the traditional broadcasting offers, distributed and consumed through a broadcasting medium, while future programs could be linked to services or events to preset recording commands. Searching on channel type enable to implement advanced favourite channel lists based for example on the language or the geographic location of the TV network. Searching on live programs' content enable to implement other favourite channel lists based on the content of the single programmes presenting for example only channels passing tennis games, thriller movies, news, etc. Interacting and selecting a single live program tune the TV set to that channel without entering any predetermined channel number.
Content can also be classical internet content (movies, music clips, any kind of products, auctions, additional content to single broadcasting events, electronic program guides) or any kind of user generated content (movies, music clips, texts or images). This content is made available searching real-time inside internet-based databases or applications.
The interaction with the presented content, is made compatible with the TV watching paradigms by transforming it in images (2D or 3D objects) inside 2D or 3D virtual environments, allowing the user an easy and intuitive navigation (scrolling) and consumption (selection). From the user's data query, a virtual world or environment is created and presented to the user that appropriately simulates the requested data in virtual objects in simulated realistic form. By performing these actions, the user is effectively interacting with the internet data returned by the query.
In some embodiments the three-dimensional environment is presented to a user on the TV set screen. In other embodiments, the user can interact with the presented scene by a single click using a remote control. In other embodiments, interaction with the presented scene brings additional results from the original data query into the three- dimensional view.
According to a second aspect of the invention, the system comprises the translation of the interaction with a graphical environments in the correct API or web services calls according to the predetermined parameters stored in the system. The graphical user interfaces acts as a logic interfaces between the user cognitive understanding of the presented content and the complexity of the URLs knowledge necessary to interacts with the presented objects. According to a third aspect of the invention, the presentation and the interaction with web content occurs in a bimodal user interface allowing to perform tasks while continuing watching conventional broadcasted content. The search and interaction process is implemented inside a bimodal user interface allowing the coexistence of the broadcasting technology with the on demand internet paradigms. The present invention describes an example interface that facilitates channel surfing and browsing while enables web-content delivery and consumption within the same user interface.
According to another aspect of the invention, the system comprises a transcoding component enabling a cross translation of multimedia formats: from text queries result set to graphical environments, from text to sounds, from a video streaming format to another, although the used video trans-coding method is not part of the presented invention.
By way of example, a channel list is generated, for example, based on a realtime database search operation (i.e. thriller movies, tennis games, news, ..). An interface displays at least a portion of the channel list and allows for user-selection of a channel from the channel list so that the selected channel may be tuned.
By way of another example, the presented list of objects could be generated based on a real-time search result of movies to be consumed in streaming, or music clips, or books to be bought, or any other web-based available content to be consumed. These and other features and advantages will be better and more completely understood by referring to the following detailed description of example embodiments in conjunction with the drawings. BRIEF DESCRIPTION OF DRAWINGS
Figure 1 is a diagram illustrating the technical architecture of the presented system Figure 2 illustrates the trans-coding features of the invented system
Figure 3 illustrates a typical broadcasting television user interface.
Figure 4 illustrates a bimodal (IP + broadcasting) user interface allowing the interact with web content while consuming broadcasting television.
Figure 5 is a flowchart of a method for translating an Internet search query and its related result set in graphical virtual environment according to an exemplary embodiment of the present invention.
Figure 6 is a flowchart of a method for transparently translating the interaction with a graphical user interface with internet web services calls for the consumption of the presented content. Figure 7 illustrates an example for the presentation of some menu items in a graphical form.
Figure 8 is a flowchart of a method for translating of a scrolling operation in a navigation process according to an exemplary embodiment of the present invention.
Figure 9 illustrated the graphical representation of a result set of movies responding to a real-time search request.
Figure 10 illustrated the graphical representation of a result set of books responding to a real-time search request inside a virtual library.
LIST OF PHYSICAL COMPONENTS REFERENCES
010 Internet public APIs, web resources or services
020 Web Server / Application Server layer
025 Broadband IP connection 030 TV set + Set Top Box + client program
035 Broadcasting medium
040 TV networks 045 Remote control
210 Algorithms for the translation of a user request in an API address
230 Algorithms for the translation of a query result set in a graphical environment
235 Algorithms for text to sound conversion 240 Video trans-coding algorithms
350 Passive broadcasting window
455 Active interactive IP window
LIST OF FORMULAE
F 1 API-URL - F 1 (SCENE-ID, FUNCTION-ID)
F2 RESULT-SET = F2(API-URL, SEARCH CRITERIA)
F3 SCENE-ID = F2( USER-ID, FUNCTION-ID)
F4 OBJECT-ID = F3 (SCENE-ID, DATAKIND-ID)
DETAIL DESCRIPTION OF THE INVENTION
The invention concerns a method for searching, browsing and consuming web-based content within a broadcasting environment (Figure 1). The system includes a set-top box device (Figure 1 - 030) comprising at least a processor and a memory accessible to the processor. The system includes a client computer program embedded within the memory of the set top box and executable by its processor. The client computer program comprising instructions to input search criteria, to display, to browse (surf or navigate) and to interact with displayed data in a graphical form (vectors). The computer program comprising as well a streaming player able to decode at least single streaming format. The set-top box device is connected to the IP Internet network through a broadband connection (Figure 1 - 025). The set-top box device is also connected to at least a broadcasting medium 035 (cable, satellite, terrestrial or IP) (Figure 1 - 035) allowing conventional broadcasting consumption of live television programs. The system also includes a server hardware and software platform (Figure 1 - 020) interfacing the invented architecture to the Internet (Figure 1 - 010). A software platform and a relational database runs inside the server managing all the necessary parameters, allowing graphical environments parameters to be translated in APIs or web services calls and allowing result set of database retrieval operations to be translated in 2D or 3D virtual environments. The server, with its predetermined parameters, acts as unique interface towards the multitude of web services, applications and databases hiding this complexity to client application and therefore to the TV user.
The server platform also includes a trans-coding component (smart edge) enabling the selected content to be transformed in a multimedia format in order to be consumables by TV users (Figure 2 - steps 030,035, 040) : from a list of items to a graphical environment, from text to sounds, from video streaming in one format to a video streaming in another format.
Start flow is a common broadcasting TV screen 350 on the screen with a common show received through a broadcasting medium 035.
The user starts the client program running inside a set top box opening an interactive window 455 beside the broadcasting window 350, implementing the physical coexistence of the passive broadcasting television consumption with the active interaction with IP-based internet content.
Start flow could also be a predefined user interface already initialized with the two presented windows 350 and 455: one dedicated to the broadcasting environment (Figure 4 - 350) , the other dedicated to the presentation of a real-time retrieve operation of web-based content (Figure 4 - 455). Focusing the attention to the interactive window 455, the user can input a search criteria starting a retrieval operation of web-content (Figure 5) and can navigate (surf) inside the displayed content interacting with single displayed items (Figure 6). Search and presentation flow
The logical process of the software functions necessaries to search, retrieve and present of web-content is described in the main functional block diagram of Figure
5.
The process begins at step 510 where the user inputs search criteria using the remote control 045. Search criteria are given to the system using the T9 protocol (numbers from 0 to 9) in a similar way as SMS messages are written using mobile phones. Search criteria could also be given to the system by future available voice recognition systems.
10
The program running inside the set top box detects the search criteria and sends in step 520 a request to the Internet server 020 through the IP connection 025. The content the user is searching for depends from the user location inside the navigational virtual environments. To search about books the user must enter a books' store, to search about TV channels the user must enter a virtual EPG an so on. The user physical location inside the virtual environments is translated in a SCENE-ID, then used by the server together with the search criteria to determine what the user is searching for.
Each request is characterized by a function internal descriptor (SCENE-ID), the interaction identifier (FUNCTION-ID) and the search criteria. In step 530 a server program uses this information for the determination of the kind of request and therefore the determination of name or the address of the web service to be called using the formulae Fl. The name of the program for each different combination is automatically retrieved interpreting predetermined parameters stored inside an internal database, avoiding programming activities necessary to extend the system to different external J1. data sources, thus making the system extendible to any kind of available data. The parameters allowing the correct execution of the above described formula by internal algorithms are manually stored and maintained during the configuration phase.
In step 540 the server platform redirects therefore dynamically a specific request to an external open existing web APIs (software system designed to support 30 interoperable Machine to Machine interaction over a network) using standard Web Services or proprietary protocols. In step 550 the called service 010 runs the database query and returns the information to the server, normally in XML format or any other readable format. A typical result set of a search operation comprises a list of items, where for each selected item the following attributes are defined: an external item-ID, a name or a description, one or more URLs resources to be called while interacting with the item, and eventually one or more images URL associated with the selected item. In step 560 the result set is received by the server 020. Step 570, based on the parameters stored into the server, gives a 2D or 3D aspect to each selected item, placing each object in an appropriate virtual environment and assigning to each object the defined services or events available to the users to interact with. The algorithms that transform received result sets in virtual environments are based on two different formulae, one for the determination of the landscape (F3) and a second one for the determination of the objects shape to assign to each returned item (F4). F3 uses as parameter the USER-ID enabling the definition of user preferences so that the system allows different users to interact with the same content through different graphical representations. Formulae F3 uses the same FUNTION-ID parameter of formulae Fl to allow the physical representation of different data sources inside different graphical landscapes (SCENE-ED). Formulae F4 gives each selected item a shape assigning a SHAPE-ED as a result of a combination of the SCENE-ED and a new D AT AKEND-ED parameter. The DAT AKIND-ED is any kind of data descriptor returned by the called web service inside the result set. Examples of D AT AKIND-ED could be the sex of a persons in case of a people database, the product type (book, disc, ..) of a shop, or any other external available item descriptor. The OBJECT-ED results of formulae F4 corresponds with a 3D model that is displayed as 3D object inside the landscape.
Received data (item-ED, descriptor, detail URLs, images URLs) are temporarily cached inside internal cache databases just for the time necessary to allow the user a later interaction. Any other information is permanently stored in the server databases.
Step 580 sends the generated graphical information to the set top box 030 in in a streaming format through the EP broadband connection 025. A typical graphical scene descriptor comprises: scene-ED, scene-parameters, vector of 3D-objects, vector of available function-EDs. The typical 3D-object comprises points and coordinates, image
URLs, and the external item-ED to recognize items to interact with. In step 590, the software running inside the set top box traduces graphical information (vectors and descriptors) in a visible virtual environment through generic 3D engine calls drawing a virtual environment inside the interactive IP window 455 of the TV screen. If a single displayed object is associated with an image file, the client program 030 running inside the set top box reads the image directly from the internet using the received original URL and places the image as a texture over the 2D/3D model using typical computer graphics warp algorithms. An example of a graphic generated list of some selected movies is given in Figure 9, where each presented object represent a single movie. Another examples is illustrated in Figure 10 where a te result set of a books search is depicted as galleries, which can be explored following a physical metaphor.
Using the remote control 045 the user can navigate or browse the single data items in the same way he normally performs zapping or channel surfing (for example pushing the P+ button to advance.
The logical process of the software functions necessaries to navigate inside the displayed 2D or 3D virtual environments is described in the flow main diagram of Figure 8.
The logical process of the software functions necessaries to interact with a retrieved list of objects is described in the flow main diagram of Figure 6.
Interaction Flow
Interacting with a presented item means the effectively interaction with an item returned by the search query, thus with internet data. In step 610 the client program automatically selects and highlights single presented items while navigating inside the presented virtual environments. The user can start the associated service or event with a single click of a confirmation button of the remote control 045 (for example the OK button). If more services or events are available for the selected item/object, these options are displayed and can be highlighted and chosen (for example using the + or - buttons of the remote control). In step 620 the client program running inside the set top box sends the selected service request to the Internet server 020.
Selected objects could be TV channels. The selection of a TV channel starts the selected broadcasting transmission in the main window without any call to the internet server, but just redirecting selected broadcasting flow, received through the broadcasting medium 035, to the screen. This method implements an innovative way to implement channel tuning based on a search retrieval operation without the need to enter fixed numbers with the remote control 045
Selected objects could be menu items. The interaction with menu items is automatically translated in new search queries as described by the flowchart at Figure 5 without the need to manually enter search criteria, as each menu item contains implicit a predefined search criteria string. Consequently the selection of a menu item has as result the 2D/3D presentation of some objects resulting from a real-time inquiry of a specific database. This method allows to transparently generate search requests and therefore new items presentation by simply on-click interactions with displayed objects. An example of a graphical presentation some menu items inside a virtual environment is illustrated in figure 7. Interacting the "TV channel' cube, will generate the presentation of all available TV channels, while the interaction of the "Movie" cube will dynamically display to the windows 455 the subset of the TV channels passing movies as illustrated in the figure 9. Menu items are stored inside internal database at system 020. Menu items could be automatically generated by user preferences and passed search queries, or by system managers, allowing the system to dynamically grow and change following user behaviours and preferences.
Selected objects could be consumable objects physically available somewhere over the Internet network.
In step 630 a server programs receives the request and traduces it in a program call or redirection to an item associated URL previously cached in step 570. The determination of the correct URL to call is done using predetermined server parameters following the formulae F4, where the parameters item-ID (selected item) and function-ID (requested service or event) are sent to the server inside the interaction request. Step 660 redirects, thus the invokes the selected URL, generating the consumption of some content related to the selected item: the streaming a movie/music clip (step 660), the real-time translation in sound of some text (step 675), the text display to the screen (step 670) or the intrinsic in the API operation included in the 5 called web resource or service (buy, movie play, related detail text..) (step 650).
The streaming flow could be displayed inside the main broadcasting window 350, or inside the interactive window 455. This method implements an innovative way to implement a television on demand environment using internet available multimedia content and transforming the consumption of non multimedia content in a TV 10 compatible experience (Figure 2 - steps 230, 235, 240).
Navigation and surfing flow
The logical process of the software functions necessaries to navigate (data , r scrolling) inside the displayed 2D or 3D virtual environments is described in the flow main diagram of Figure 8. While navigating inside the presented virtual environments the user is effectively scrolling the data returned by the search query.
The user controls the navigation, thus the item's scroll, by single-click commands using the remote control 045. These could be the common P+, P- burtons 20 normally used while zapping television channels (Figure 9 step 810).
To overcomes traditional 3D navigation problems, the navigation inside the displayed virtual environments is not free, but is imposed and limited by the parameters stored in the scene. In step 810, depending on an environment parameter the system decides, whether to rotate the scene (step 820) or whether to simulate a user's 25 advancement in the scene (step 830).
Depending on a second parameters, the system decides whether to start an automatic and passive endless movement (rotation or advancement) or to make a single object's scroll, thus rotating and highlighting the next displayed object or advancing and highlighting to the next displayed object. Movements and rotations simulate a movie- like animation on the screen compatible with the watching paradigms of the television. Therefore, because the search results are depicted in an environment with which the user is immediately comfortable, this presentation allows a useful dialog between the user and the presented content, by using natural movement and exploration functions of a virtual three-dimensional world.
5 The cited scene parameters are send from the server to the client as described in Figure 5 step 580.
Trans-coding features
-π In order to allow the end user to transparently consume different multimedia contents' formats the server platform 020 comprises also a multimedia trans-coding format block (Figure 2). This is a smart platform enabling processing and inserting of personalized content to end users. In step 570 / 230 search result sets are real-time traduced in graphical environments, in step 235 texts are traduced in sounds, and a
1 _ video streaming trans-coding step 240 changes the coding of a selected stream in order to be decoded and played by a unique player included in the client program. In this scenario, the server logic of the systems 020 acts as a trans-coding platform, dealing with on-the-fly different Trans-rating and trans-coding techniques and different physical encapsulation to fit different EP networks. This features, allows to implement inside the STB a single player/viewer allowing the platform to be integrated to different content encoded content types.
BESTMODEFORCARRYINGOUTTHEINVENTION
25 The following description is presented solely for the purpose of disclosing how the present invention may be made and used. The scope of the invention is defined by the claims.
The best way to carry out the invention is to define an end-to-end open infrastructure and to develop any form of software able to freely deliver and consume 30 web-based content to TV, making it consumable in an on demand way by TV users.
Figures 9 and 10 illustrate the final result obtained from an application prototype applying the invented method for the interaction with a movie database and a book-shop database. Figure 10 also empathizes the bimodality of the proposed television user interface, where the interaction with a web based virtual library occurs over a conventional broadcasting consumption window in a picture-in-picture modality. One architectural advantage of the proposed invention is to enable the delivery of web-content to TV users without the need of any PC. This is done transforming the set top box in a networked hybrid device, enabling the connection to the IP Internet network while still receiving and decoding DVB digital broadcasting signals.
An advantage of the proposed application is to simplify the consumption of web-content within an open infrastructure and platform where the content is not imposed chosen an predefined by a legacy system, operator content provider or TV operator, but is freely chosen from a multitude of internet applications, databases and services.
An advantage of the proposed application is to simplify the consumption of TV content (channels) implementing search features over an internet based electronic program guide. Figure 9 shows an example of a graphic television channels surfing and tuning process, while figure 10 shows how this process occurs keeping the broadcasting windows active over the TV screen, thus while watching television.
An advantage of the proposed user interface is to integrate in a unique and bimodal interface the passive consumption of TV content (broadcasting) with the active interaction with web-based content within an immersive and multimedia experience. The software form, the programming language, the operating system, the standard user interface, the 3D engine used, the development technique and tools of this software infrastructure aren't important because the invented method concerns only the form of dialog between a TV user and any accessible web-based database. The user formulates an inquiry and the software (developed applying the invention) redirects the query to a specific database and translates the list of data resulted from the inquiry in an animated and controllable 2D or 3D image of a simulated virtual space. The user needs any form of scroll in the data resulted from the inquiry and the software performs an appropriate movement in the 3D space simulating a navigation or a scroll operation within the result set. INDUSTRIAL APPLICABILITY
The primary commercial application of this method is the development of a software platform able to delivery and distribute TV content and web-content in an on demand way to TV users. TV content could be an internet based electronic program guide. TV content could be past aired TV shows and programmes stored by TV networks inside internet based databases. Web content could be all internet based content that users could have interest to consume using the TV set: movies and multimedia content today not distributed by broadcasting TV networks, auctions, books, music clips, any kind of product of internet offer, additional content to single broadcasting events, electronic program guides (EPG) or any kind of user generated content (movies, clips, texts or images).
Other commercial application of this method is to boost the development of advanced networked set top boxes enabling TV users to buy a retail product outside legacy realities enabling the consumption of on demand content within a television environment.
Under the strict application of the invented method, the TV user experience could be transformed in an on demand environment making enhanced TV a reality.