PRIORITY CLAIM AND REFERENCE TO RELATED APPLICATIONThe application claims priority under 35 U.S.C. §119 from prior provisional application Ser. No. 61/066,188, which was filed Feb. 19, 2008.
STATEMENT OF GOVERNMENT INTERESTThis invention was made with government support under Contract No. CNS 0420477 awarded by National Science Foundation. The government has certain rights in the invention.
TECHNICAL FIELDFields of the invention include interactive data display, exploration and collaboration.
BACKGROUND ARTStandard computer displays greatly limit the ability of a user to explore, interact and collaborate with others. Relatively small amounts of data are presented on a standard computer display. Use of multiple displays is common, but multiple displays do little to solve the difficulties encountered when attempting to view and explore complex data.
Scientists, designers and engineers increasingly focus on complex phenomena, rely on instruments that produce greater volumes of data, and collaborate with geographically distributed teams. General purpose computing, gaming and other applications also can present complex and highly detailed environments and interactions. A central challenge for researchers using scientific systems and other users of gaming and general purpose systems is the ability to manage the increased scale and complexity of the information and environment presented by a display. Greater scale and complexity places a heavy strain on computational systems and infrastructure. Additionally, the usability of such systems is also limited by human factors, such as their cognition and/or attention-span.
Large interactive displays have been developed, primarily for the field of scientific research and collaboration. One example is the LambdaVision 100-Megapixel wall-sized LCD tiled display introduced by Electronic Visualization Laboratory (“EVL”), which quickly resulted in over a dozen research laboratories constructing compatible instruments, called OptIPortals. EVL also developed the Scalable Adaptive Graphics Environment (“SAGE”) operating system software to enable domain scientists to work and collaborate using these displays The massive resolution afforded by these displays enabled users to view large collections of high-resolution visualizations generated in real-time from compute clusters housed at supercomputing facilities around the world.
These displays however, include limitations, which can reduce usability when being used with certain applications. One such limitation is the position and/or orientation of the display. Since electronic data often replaces the physical presentation of data, there is a concern that users will have difficulty adapting to the electronic presentation of data. Therefore, one design criteria of such displays is to provide users with the feeling of working in a traditional work environment, thereby increasing usability.
Another display device developed by EVL is the LambdaTable 24-Megapixel table-oriented LCD display. This device employs a horizontal display and presents a more natural working environment that encourages visualizations and collaborations as it replicates common human practices of working with whiteboards, printouts, blue prints, etc. where multiple people gather around a table to view data and/or documents.
Users of the LambdaTable interact with “pucks”, which are used to control the display. Special purpose pucks, for example, permit moving, shrinking, selecting, and magnifying a portion of data being displayed. Users can select and manipulate data with the pucks, and the table-sized display allows multiple users to view and interact with the data simultaneously. Displays using pucks however, have several limitations. An example limitation is that the number of users interacting with a display is limited by the number of available pucks. Pucks are also costly and subject to loss or damage, thereby requiring replacement.
One display device that avoids use of such pucks is the projector-based Microsoft® Surface display. The Surface employs a multi-touch interface, which allows a user to interact with the display by touching it with one or more fingers, thereby forgoing the need for pucks.
To further enhance usability and more closely resemble a user's natural working environment, some developers have introduced displays capable of producing three-dimensional (“3D” or “stereoscopic”) images, instead of the traditional two-dimensional (“2D” or monoscopic) images as provided in the examples above. Examples of stereoscopic displays include the Philips® MultiSync non-interactive LCD display product line. However, since these displays are configured specifically to display stereoscopic images, they are greatly limited in their ability to display monoscopic images. While such monoscopic images can be displayed, the quality/resolution is considerably poor when compared to traditional monoscopic displays.
DISCLOSURE OF INVENTIONAn interactive display device of a preferred embodiment includes a display surface and a touch interface associated with the display surface. A first LCD layer generates a dynamic parallax barrier and a second LCD layer generates stereoscopic images in cooperation with the first LCD layer. A light source backlights the first and second LCD layers.
A preferred method for controlling an interactive stereoscopic display device provides two-dimensional content to be displayed by the second LCD layer, three-dimensional content to be displayed by the second LCD layer, and generates a description that overlays two-dimensional content to be rendered monoscopically onto three-dimensional content to be rendered stereoscopically. Parallax barrier content is displayed on the first LCD layer.
A preferred method for displaying images overlays monoscopic and stereoscopic image data into combined image data and displays the combined image data to the second LCD layer. Dynamic parallax barriers are displayed on the first LCD layer in cooperation with the second LCD layer. User interaction with the display device is sensed.
BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1A is a schematic side view of a preferred embodiment of an interactive display device of the invention;
FIG. 1B is a schematic side view of the device ofFIG. 1A;
FIG. 2 is an exploded perspective view of a dynamic parallax barrier;
FIG. 3 illustrates a preferred software and control system for controlling an interactive display device of the invention;
FIG. 4 is a schematic perspective view of a parallax barrier;
FIG. 5 is a schematic perspective view of an alternate embodiment of a parallax barrier;
FIG. 6 is a schematic perspective view of an LCD screen and an alternate embodiment of a parallax barrier;
FIG. 7 is a schematic perspective view of a preferred embodiment of an interactive display device of the invention; and
FIG. 8 is a schematic perspective view of another a preferred embodiment of an interactive display device of the invention.
PREFERRED MODES FOR CARRYING OUT THE INVENTIONPreferred embodiments of the invention provide an interactive display device, which is capable of displaying both monoscopic and stereoscopic images. However, unlike other stereoscopic displays known in the art (e.g., the Philips® Multi-sync), the device displays monoscopic images as resolutions comparable to traditional monoscopic displays (i.e., full native resolution). In other words, monoscopic image resolution is not compromised by the device's ability to display stereoscopic images. The display device is also capable of displaying both monoscopic images and stereoscopic images simultaneously. That is, users can view both monoscopic windows and stereoscopic windows side by side without having to wear specialized 3D glasses or having to switch between modes. This minimizes physical encumbrances associated with the current commercial instruments. The device also provides the user with either touch or gesture-based interaction.
The usability of a display system is limited by human factors, such as cognition and/or attention-span. Usability is enhanced when a user is provided with the feeling of working in a traditional work environment. However, depending on the application being run, users will have different expectations regarding the working environment. The ability to display both monoscopic and stereoscopic images allows for greater flexibility in representing a traditional work environment for a given application.
For example, consider a group of individuals using the device to collaborate and research geographical information. A traditional working environment may include information likes maps, pictures, statistical data, etc. While some of this information lends itself to being displayed as a stereoscopic image (e.g., maps), other information is traditionally presented as a monoscopic image (e.g., statistical data). Consider for example, a spreadsheet of data concerning a particular region on a map. In a traditional working environment, geologists would likely use a map or a globe to view this particular region along with a standard spreadsheet of data. Using devices of the invention, users can view a stereoscopic representation of the map and simultaneously view a monoscopic representation of the statistical data alongside the map. Similarly, general purpose computing, gaming, communication and many other systems can benefit from the simultaneous clear display of monoscopic data along with stereoscopic data presented by a display of the invention.
An interactive display device of a preferred embodiment includes a display surface and a touch interface associated with the display surface. A first LCD layer generates a dynamic parallax barrier and a second LCD layer generates stereoscopic images in cooperation with the first LCD layer. A light source backlights the first and second LCD layers.
A preferred method for controlling an interactive stereoscopic display device provides two-dimensional content to be displayed by the second LCD layer, three-dimensional content to be displayed by the second LCD layer, and generates a description that overlays two-dimensional content to be rendered monoscopically onto three-dimensional content to be rendered stereoscopically. Parallax barrier content is displayed on the first LCD layer.
A preferred method for displaying images overlays monoscopic and stereoscopic image data into combined image data and displays the combined image data to the second LCD layer. Dynamic parallax barriers are displayed on the first LCD layer in cooperation with the second LCD layer. User interaction with the display device is sensed
Preferred embodiments of the invention will now be discussed with respect to the drawings. The drawings may include schematic representations, which will be understood by artisans in view of the general knowledge in the art and the description that follows. Features may be exaggerated in the drawings for emphasis, and features may not be to scale.
A preferred embodiment of the invention is a large format high resolutioninteractive display device10 configured as desk, which is likely to replace a desk and computer workstation. It is noted that such devices can also be configured in other forms (e.g., a table or portable case) as desired by the user. Thepreferred embodiment display10 is sized similarly to a traditional desk, providing a generous display and workspace. The desk is a single example, and the invention is not limited thereto. Thedisplay10 can be configured in other arrangements, for example on stands or mounts to present a vertical display and workspace or a horizontal display and workspace when thedevice10 is configured as a desk.
Referring now toFIGS. 1A and 1B, the preferred embodiment of thedisplay device10 is shown. Included in thedevice10 is a top layer of aclear sheet12, preferably made of an acrylic material, such as polymethyl methacrylate or another clear polymer. Theclear sheet12 presents a display surface to user. Further included in thedevice10 is afirst LCD layer14 for presenting a dynamic parallax barrier to enable autostereopsis. Dynamic parallax barriers can be built from one or more LCD Panels as shown inFIG. 2, which labels the sub parts of thebarrier14 ofFIG. 1. While it is contemplated that devices will be fabricated from new techniques, example devices have been built from components of existing LCD panels. To build a dynamic parallax barrier from existing LCD displays (left side ofFIG. 2), two LCD monitors are dissembled and re-assembles them into a common enclosure, sharing a common backlight. This creates arear polarizer14a, amid polarizer14b, and afront polarizer14c.Liquid crystal panels14eand14fare disposed between thepolarizer panels14a-14c. Angles of polarization between the two LCD screens are orthogonal to each other. This requires careful removal of the rear polarizer from the front LCD. Because the illumination reaching the front LCD is rotated 90° by the rear LCD, the front LCD behaves in the inverse, and the parallax barrier is drawn white-on-black rather than black-on-white. Preferred dynamic parallax barriers have been developed by researchers at the University of Illinois at Chicago, and are has and are described in a web-available publication by Peterka et al., entitled “Dynallax: Solid State Dynamic Parallax Barrier Autostereoscopic VR Display.” An autostereoscopic display with the dynamic parallax barrier is also described in U.S. Patent Publication 20080143895.
Referring back toFIGS. 1A and 1B, asecond LCD layer16 generates both monoscopic and/or stereoscopic images (the term screen will be used to describe a user's view of both thefirst LCD layer14 and the second LCD layer16). Alight source18, such as fluorescent tubing light emitting diodes or other backlighting, illuminates the LCD layers14 and16. Athin diffuser20 is preferably disposed between thesecond LCD layer16 and thelight source18 averages the illumination from thelight source18. One or more additional diffusers can be used to further average the illumination from the light source and eliminate illumination hot spots.
Users interact with thedevice10 via a touch interface, which preferably provides multi-touch interaction with the device. The preferred touch interface renders external devices, e.g., pucks, unnecessary, though they may be applied if desired. The touch interface permits users to use their own hands to interact with thedevice10. An example touch interface utilizesinfrared LEDs22 that are embedded in or around theclear sheet12 to sense human contact. Preferably, there is also a gesture interface. Aninfrared camera24 below thediffuser20 cooperates with theinfrared LEDs22 to provide the touch interface. Thediffuser20 can have an opening or clear section to provide thecamera24 with a view through the diffuser to theclear sheet12 for determining user interaction. Thecamera24 andlight source18 are preferably disposed in abox26 having a generally uniform glossy white interior.
The touch sensing used in thepreferred embodiment device10 is based upon use of Frustrated Total Internal Reflection (“FTIR”). See, Han, “Low-cost multi-touch sensing through frustrated total internal reflection.” 2005 Proceedings of the 18th annual ACM symposium on User interface software and technology. Seattle, Wash., USA, ACM. By this technique, theinfrared LEDs22 are embedded at the edges of theclear sheet12. When an object is brought within several wavelengths' distance, the internally reflected infrared light is able to pass through the acrylic sheet where it is detected by an infrared camera. Han's original implementation was applied to projection-based screens, but has been adapted in the invention to work with LCD panels. Even when an LCD panel is completely opaque to visible light, the infrared light is able to pass through and user interaction can be detected with thecamera24 that senses the internally reflected light caused by user interaction with the screen. An advantage of using LCD panels is that they can be viewed in a normally lit room.
Tiled-display surfaces, present a unique issue with respect to constructing FTIR touch screens in that the mullions (i.e., borders) can occlude the infrared camera's view of portions of the FTIR screen. This can overcome this by first building the FTIR screen as a single large acrylic sheet rather than as a tiling of screens, and raising it some distance (between 0.5″ and 1″) above the dynamic parallax barrier panel depending on the field of view of the camera.
Thedevice10 also preferably includes a gesture interface. A gesture interface uses sensors or cameras to detect gestures made by a user without requiring the user to touch theclear sheet12. The gesture interface can detect a hand or other an object in close proximity to thesheet12, and permit gesture interaction with thedevice10. Preferably,infrared cameras25 equipped with infrared illuminators are used for gesture tracking (FIG. 1A). Thecameras25 are mounted around the perimeter of thetop sheet12 and pointed inward so that the field of view creates a tracking volume. The cameras have a relatively short depth of field therefore only rendering relatively close objects as being sufficiently distinct to register as a trackable object.
The device's ability to display images and interact with one or more users is preferably managed by a software system. A preferred software architecture and method100 for the device is shown inFIG. 3. The method100 is preferably stored on a computer-readable storage medium included in thedisplay device10. The system100 provides afinger tracker module102 for gathering and transporting user touch data, a displayapplication manager module104, which uses the touch data to update the environment, and finally a dynamic parallaxbarrier driver module106 for providing image data to the device's LCD layers14,16.
In thefinger tracker module102, anoise filter component110 causes theinfrared cameras24 to take raw images of the user's fingers or other objects as he interacts with the device, which are smoothed with various filters to reduce noise levels. Next, afinger extractor110 examiners the contours and position of “blobs” found in the images. This will identify the finger locations of theuser108 on theclear sheet12 of the device (FIGS. 1A and 1B). Afinger mapper114 then maps finger positions from thecamera24 to a unified screen coordinate system and eliminates duplicate fingers that are picked up by any adjacent cameras if multiple camera are used. Next, agesture detection component116 analyzes the movement of fingers and their relative distances to identify certain predetermined touches or gestures. For example, theuser108 may move two fingers simultaneously to pan the image displayed on thedevice10. Finally, atouch transporter118 sends touch interface data such as finger touches, movements, and positions, and gesture interface data, such as gesture positions and speeds to theapplication module104. Such communication is conducted over anetwork120 such that thefinger tracker module102 can exist on a separate computer from the remaining modules, if desired.
Next, atouch acquisition component122 in thedisplay application manager104 acquires the touch and gesture interface data provided by thefinger tracker module102 such that anenvironmental interaction component124 can manipulate and update the virtual environment and/or any object it contains. Thereafter, a three-dimensional screen descriptor126 generates a high level description of the three-dimensional scene based on the current state of the environment. This description includes the contained three-dimensional objects, their positions, and their surface material properties. Similarly, a two-dimensional content generator128 generates all two-dimensional content including for example, overlay images. Next, aview layout manager130 generates a description of the screen and specifies the portions of the screen that have two-dimensional content, which should be rendered monoscopically, and the portions of the screen that have three-dimensional content, which should be rendered stereoscopically. As will be descried in further detail below, the dynamic parallaxbarrier driver module106 affords for this configuration to be dynamic, and thus, the number, position, and size of two-dimensional content can be changed by the displayapplication manager module104 in real-time. A user configuration component132 then generates a description of the number of autostereoscopic views to be generated and their corresponding vantage point in three-dimensional space. This vantage point can able be modified in real-time to support a variable number of users.
The last module is the dynamic parallaxbarrier driver module106, which has a parallaxbarrier generation component134 for generating a barrier by either drawing alternating opaque and transparent lines over three-dimensional content, or leaving areas over two-dimensional content transparent. Parameters of the parallax barrier are altered depending on the user configuration component132 in the displayapplication manager module104. The resulting barrier image is then displayed on thefirst LCD layer14 on thedevice10. Aview rendering component136 provides for each user, a pair of images (one for the left eye and one for the right eye) that are rendered based on the scene information generated by the three-dimensionalscene descriptor component126. The total resulting number of images equals the number of users multiplied by two. Next, a three-dimensionalimage combination component138 electronically slices the rendered images into a plurality of thin pieces, which are combined to form a single image. Finally, a two-dimensional image overlay component140 overlays the two-dimensional images onto the single image to create an image which is displayed on thesecond LCD panel16 of thedevice10.
The dynamic parallax barrier autostereoscopic technique used in devices of the invention enables an LCD display to support viewing in several simultaneous modes, with the viewing mode selectable on a per-pixel basis. A single-viewer tracked autostereo mode enables a high-resolution virtual-reality experience with first-person perspective, giving ideal viewing of stereoscopic polygonal and volumetric data. Dual-viewer tracked autostereo mode enables a shared virtual-reality experience, with a first-person perspective for each user. Panoramic autostereo mode provides a shared stereoscopic perspective to multiple users, enabling group collaboration with stereoscopic data. Monoscopic display at the LCD's full native resolution allows for the normal viewing of fine text and high-resolution monoscopic digital imagery on both instruments.
The dynamic parallax barrier technology enables these modes and utilizes a parallax barrier, which is an alternating sequence of opaque and transparent regions. An example is shown inFIG. 4. Typically, this parallax barrier is mounted in front of an LCD display, offset from it by a relatively small distance. The displayed image is correspondingly divided intosimilar regions200 of perspective views, such that all of the regions belonging to one perspective are visible only by oneeye202, and likewise a different set ofregions204 corresponding to another perspective is visible by the other eye206. Theeyes202,206 are thus simultaneously presented with two disparate views, which the brain fuses into one stereoscopic image. Parallax barriers are usually mounted in a rotated orientation relative to the pixel grid to minimize or restructure the moiré effect that results as an interference pattern between the barrier and pixel grid.
Parallax barrier autostereo displays follow one of two design paradigms. Tracked systems produce a stereo pair of views that follow the user in space, given the location of the user's eyes or head from the tracking system, these are strictly single-user systems
Another option is the untracked panoramagram where a sequence of perspective views is displayed from slightly varying vantage points. An example is shown inFIG. 5, which also hasregions200,204 as described above. However, in this approach, theregions204 are configured such that the display can be viewed by multiple users (i.e., by multiple static eye positions208). This option is well-suited to preferred embodiment large format high resolution interactive display tables. Multiple users can view this type of display, even upside-down, with limited “look-around” capability. This enables viewers to stand on the two long sides of large format high resolution interactive display table and still see correct stereoscopic views. The degree of look-around and the usable range of the display are determined by the number of views in the sequence. There is a trade-off between the number of views and the effective resolution of the three-dimensional image, and tests have demonstrated that a 9-view sequence is optimal given the native resolution of an example 30″ display and its intended pattern of use.
An example is shown inFIG. 6, in contrast to existing autostereo displays, the dynamic parallax barrier is constructed from a fullyaddressable LCD screen210 placed in front of thescreen212 used to render thestereo scene214 and to create avirtual scene216 from the viewpoint of aneye218. This approach permits greater flexibility and usability while mitigating some of the drawbacks of the previous methods. Thefront screen210 can be rendered transparent, converting the display to a full-resolution monoscopic system, and eliminating the degradation of resolution commonly associated with static-barrier displays. In stereo mode, the parameters of the parallax barrier can be updated in real time, so that optimal viewing conditions are maintained at all times, regardless of view distance. Sensitivity to system latency is reduced by accommodating rapid head movements with a translation of the front barrier pattern. Moreover, the viewing mode may be adapted in real time by modifying the barrier parameters in software. Dynamic parallax barriers can spatially multiplex more than one pair of stereo channels at the same time, so multiple tracked viewers can either view their own individual perspective of the same scene, or entirely different scenes. Any of these variations are possible on a per-tile basis or on a subset of a tile, since they are all performed at pixel scale in software. All of these features occur by virtue of the barrier being dynamic and fully addressable like any other graphical display.
For very large displays of the invention, particularly in the table embodiments, until large enough high resolution displays are available, some embodiments that exceed the size of currently available LCD panels may require tiling of multiple LCD panels in each of the first and second layers. An example embodiment of a tiled device is shown inFIG. 7. In this embodiment, thedevice10ais divided into six groups of LCD panels300 for use withmultiple users108a. Additional software may be required to permit such tiling. An example operating SAGE, which is an operating system for tiled-display environments, that lets users launch distributed visualization applications on remote clusters of computers and stream the visualizations directly to their tiled displays, where they can be viewed and manipulated.
While tiling LCDs introduces mullions, the increased resolution provided is more important. The effect of the mullions can be minimized by rendering graphics is rendered in such a way as to take them into account (e.g., by placing virtual pixels behind them so the effect is like looking out of a window). The need for mullions will disappear when LCD display technology (or another type of comparable display) can make completely seamless and scalable flat-panel displays of desirable size and necessary resolution. For comparison, an example of anon-tiled device10bis shown inFIG. 8, which is slightly angled towards and is being operated by asingle user108b.
A preferred embodiment device provides 24-Megapixel resolution, and generates 9 fixed views. The preferred embodiment device also provides 8-Megapixel resolution, and generates user-centered-perspective autostereoscopic views. However, as larger LCD displays become available with high resolution, the need, for example, for multiple LCD panels in layers of a preferred embodiment table of the invention may be alleviated.
Preferred embodiment displays provide resolution that approaches print quality (approximately 72-dpi, or higher). With current LCD technology at a reasonable cost, and example embodiment large format high resolution interactive display table can be built using twelve 30″ (4-Megapixel) LCD panels (6 for image generation, and 6 for stereo separation) providing a total resolution of 24-Megapixels.
As noted, devices of the invention will have many important applications for a variety of users. Some of these users are domain scientists who increasingly rely on digital infrastructure (also known as cyberinfrastructure) and global collaboration to conduct research. Therefore, the device is preferably equipped with 1 to 10 Gigabit/s network interfaces and switches that can enable them to connect to 10-Gigabit national and international high-speed networks, such as National Lambda Rail, Internet2, and the Global Lambda Integrated Facility. As public and private networks evolve to match speeds of these high speed networks, displays of the invention can be configured to communicate with as yet to be developed networks and protocols having suitable data communication speeds.
Preferred display devices of the invention also support life-sized distance collaboration via high-definition videoconferencing with remote participants who want to be part of a meeting, and to leverage high speed networks of National Science Foundation's cyberinfrastructure facilities, such as the TeraGrid and future Petascale Facility, over high-speed networks. Further, the devices provide spatialized audio feedback with the visuals that are presented (e.g., the audio from a videoconference is proximally located with the videoconferencing image.) As shown inFIG. 7, preferably, high-definition displays302 are positioned at the ends of the table and are equipped with high-definition video cameras304 and network controllers configured for networking thedevice10ato at least one additional remote display device for remote collaboration. When not engaged in a videoconference, side screens can be used as additional surfaces on which information can be posted. Above the users are sound projectors that enable audio to be spatialized along the length of the table.
While specific embodiments of the invention have been shown and described, it should be understood that other modifications, substitutions and alternatives are apparent to one of ordinary skill in the art. Such modifications, substitutions and alternatives can be made without departing from the spirit and scope of the invention, which should be determined from the appended claims.
Various features of the invention are set forth in the appended claims.