US20120306850A1 - Distributed asynchronous localization and mapping for augmented reality - Google Patents

Abstract

A system and method for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices is described. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.

Description

BACKGROUND
• Augmented reality (AR) relates to providing an augmented real-world environment where the perception of a real-world environment (or data representing a real-world environment) is augmented or modified with computer-generated virtual data. For example, data representing a real-world environment may be captured in real-time using sensory input devices such as a camera or microphone and augmented with computer-generated virtual data including virtual images and virtual sounds. The virtual data may also include information related to the real-world environment such as a text description associated with a real-world object in the real-world environment. An AR environment may be used to enhance numerous applications including video game, mapping, navigation, and mobile device applications.
• Some AR environments enable the perception of real-time interaction between real objects (i.e., objects existing in a particular real-world environment) and virtual objects (i.e., objects that do not exist in the particular real-world environment). In order to realistically integrate the virtual objects into an AR environment, an AR system typically performs several steps including mapping and localization. Mapping relates to the process of generating a map of the real-world environment. Localization relates to the process of locating a particular point of view or pose relative to the map. A fundamental requirement of many AR systems is the ability to localize the pose of a mobile device moving within a real-world environment in order to determine the particular view associated with the mobile device that needs to be augmented.
• In robotics, traditional methods employing simultaneous localization and mapping (SLAM) techniques have been used by robots and autonomous vehicles in order to build a map of an unknown environment (or to update a map within a known environment) while simultaneously tracking their current location for navigation purposes. Most SLAM approaches are incremental, meaning that they iteratively update the map and then update the estimated camera pose in the same process. An extension of SLAM is parallel tracking and mapping (PTAM), which separates the mapping and localization steps into parallel computation threads. Both SLAM and PTAM techniques produce sparse point clouds as maps. Sparse point clouds may be sufficient for enabling camera localization, but may not be sufficient for enabling complex augmented reality applications such as those that must handle collisions and occlusions due to the interaction of real objects and virtual objects. Both SLAM and PTAM techniques utilize a common sensing source for both the mapping and localization steps.
SUMMARY
• Technology is described for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.
• One embodiment includes determining whether a first map is required, acquiring the first map, storing the first map on a mobile device, determining a first pose associated with the mobile device, determining whether a second map is required including determining whether a virtual object is located within a field of view associated with the first pose, acquiring the second map, storing the second map on the mobile device, registering a virtual object in relation to the second map, rendering the virtual object, and displaying on the mobile device a virtual image associated with the virtual object corresponding with a view of the virtual object such that the virtual object is perceived to exist within the field of view associated with the first pose.
• This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a block diagram of one embodiment of a networked computing environment in which the disclosed technology may be practiced.
• FIG. 2 depicts one embodiment of a portion of an HMD.
• FIG. 3A depicts one embodiment of a field of view as seen by a user wearing a HMD.
• FIG. 3B depicts one embodiment of an AR environment.
• FIG. 3C depicts one embodiment of an AR environment.
• FIG. 4 illustrates one embodiment of a mapping system.
• FIG. 5 depicts one embodiment of an AR system.
• FIG. 6A is a flowchart describing one embodiment of a process for generating a 3-D map of a real-world environment and locating virtual objects within the 3-D map.
• FIG. 6B is a flowchart describing one embodiment of a process for updating a 3-D map of a real-world environment.
• FIG. 7 is a flowchart describing one embodiment of a process for rendering and displaying virtual objects.
• FIG. 8 is a block diagram of an embodiment of a gaming and media system.
• FIG. 9 is a block diagram of one embodiment of a mobile device.
• FIG. 10 is a block diagram of an embodiment of a computing system environment.
DETAILED DESCRIPTION
• Technology is described for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.
• An AR system may allow a user of a mobile device to view augmented images in real-time as the user moves about within a particular real-world environment. In order for the AR system to enable the perceived interaction between the particular real-world environment and virtual objects within the particular real-world environment, several problems must be solved. First, the AR system must generate a map of the particular real-world environment in which the virtual objects are to appear and interact (i.e., perform a mapping step). Second, the AR system must determine a particular pose associated with the mobile device (i.e., perform a localization step). Finally, the virtual objects that are to appear within the field of view of the particular pose must be registered or aligned with a coordinate system associated with the particular real-world environment. Solving these problems takes considerable processing power and providing the computer hardware necessary to accomplish real-time AR is especially difficult because of form factor and power limitations imposed on mobile devices.
• In general, solving the mapping problem is computationally much more difficult than the localization problem and may require more computational power as well as more sophisticated sensors for increased robustness and accuracy. However, the mapping problem need not be solved in real-time if landmarks or other real objects within the real-world environment are static (i.e., do not move) or semi-static (i.e., do not move often). On the other hand, mobile device localization does need to be performed in real-time in order enable the perception of real-time interaction between real objects and virtual objects.
• FIG. 1 is a block diagram of one embodiment of anetworked computing environment100 in which the disclosed technology may be practiced.Networked computing environment100 includes a plurality of computing devices interconnected through one ormore networks180. The one ormore networks180 allow a particular computing device to connect to and communicate with another computing device. The depicted computing devices includemobile device140,mobile devices110 and120,desktop computer130, andmapping server150. In some embodiments, the plurality of computing devices may include other computing devices not shown. In some embodiments, the plurality of computing devices may include more than or less than the number of computing devices shown inFIG. 1. The one ormore networks180 may include a secure network such as an enterprise private network, an unsecure network such as a wireless open network, a local area network (LAN), a wide area network (WAN), and the Internet. Each network of the one ormore networks180 may include hubs, bridges, routers, switches, and wired transmission media such as a wired network or direct-wired connection.
• A server, such asmapping server150, may allow a client to download information (e.g., text, audio, image, and video files) from the server or to perform a search query related to particular information stored on the server. In general, a “server” may include a hardware device that acts as the host in a client-server relationship or a software process that shares a resource with or performs work for one or more clients. Communication between computing devices in a client-server relationship may be initiated by a client sending a request to the server asking for access to a particular resource or for particular work to be performed. The server may subsequently perform the actions requested and send a response back to the client.
• One embodiment ofmobile device140 includes acamera148,microphone149,network interface145,processor146, andmemory147, all in communication with each other.Camera148 may capture digital images and/or videos.Microphone149 may capture sounds.Network interface145 allowsmobile device140 to connect to one ormore networks180.Network interface145 may include a wireless network interface, a modem, and/or a wired network interface.Processor146 allowsmobile device140 to execute computer readable instructions stored inmemory147 in order to perform processes discussed herein.
• Networked computing environment100 may provide a cloud computing environment for one or more computing devices. Cloud computing refers to Internet-based computing, wherein shared resources, software, and/or information are provided to one or more computing devices on-demand via the Internet (or other global network). The term “cloud” is used as a metaphor for the Internet, based on the cloud drawings used in computer network diagrams to depict the Internet as an abstraction of the underlying infrastructure it represents.
• In one example of an AR environment, one or more users may move around a real-world environment (e.g., a living room) each wearing special glasses, such asmobile device140, that allow the one or more users to observe views of the real-world overlaid with virtual images of virtual objects that maintain coherent spatial relationship with the real-world environment (i.e., as a particular user turns their head or moves within the real-world environment, the virtual images displayed to the particular user will change such that the virtual objects appear to exist within the real-world environment as perceived by the particular user). In one embodiment, environmental mapping of the real-world environment is performed by mapping server150 (i.e., on the server side) while camera localization is performed on mobile device140 (i.e., on the client side). The one or more users may also perceive 3-D localized virtual surround sounds that match the general acoustics of the real-world environment and that seem fixed in space with respect to the real-world environment.
• In another example, live video images captured using a video camera on a mobile device, such asmobile device120, may be augmented with computer-generated images of a virtual monster. The resulting augmented video images may then be displayed on a display of the mobile device in real-time such that an end user of the mobile device sees the virtual monster interacting with the real-world environment captured by the mobile device. In another example, the perception of a real-world environment may be augmented via a head-mounted display device (HMD), which may comprise a video see-through and/or an optical see-through system.Mobile device140 is one example of an optical see-through HMD. An optical see-through HMD worn by an end user may allow actual direct viewing of a real-world environment (e.g., via transparent lenses) and may, at the same time, project images of a virtual object into the visual field of the end user thereby augmenting the real-world environment perceived by the end user with the virtual object.
• FIG. 2 depicts one embodiment of a portion of an HMD, such asmobile device140 inFIG. 1. Only the right side of a head-mounted device is depicted.HMD200 includesright temple202,nose bridge204,eye glass216, andeye glass frame214. Built intonose bridge204 is amicrophone210 for recording sounds and transmitting the audio recording toprocessing unit236. Afront facing camera213 is embedded insideright temple202 for recording digital images and/or videos and transmitting the visual recordings toprocessing unit236.Front facing camera213 may capture color information, IR information, and/or depth information.Microphone210 andfront facing camera213 are in communication withprocessing unit236.
• Also embedded insideright temple202 areear phones230, motion andorientation sensor238,GPS receiver232,power supply239, andwireless interface237, all in communication withprocessing unit236. Motion andorientation sensor238 may include a three axis magnetometer, a three axis gyro, and/or a three axis accelerometer. In one embodiment, the motion andorientation sensor238 may comprise an inertial measurement unit (IMU). The GPS receiver may determine a GPS location associated withHMD200.Processing unit236 may include one or more processors and a memory for storing computer readable instructions to be executed on the one or more processors. The memory may also store other types of data to be executed on the one or more processors.
• In one embodiment,eye glass216 may comprise a see-through display, whereby virtual images generated by processingunit236 may be projected and/or displayed on the see-through display. Thefront facing camera213 may be calibrated such that the field of view captured by thefront facing camera213 corresponds with the field of view as seen by a user ofHMD200. Theear phones230 may be used to output virtual sounds associated with the virtual images. In some embodiments,HMD200 may include two or more front facing cameras (e.g., one on each temple) in order to obtain depth from stereo information associated with the field of view captured by the front facing cameras. The two or more front facing cameras may also comprise 3-D, IR, and/or RGB cameras. Depth information may also be acquired from a single camera utilizing depth from motion techniques. For example, two images may be acquired from the single camera associated with two different points in space at different points in time. Parallax calculations may then be performed given position information regarding the two different points in space.
• FIG. 3A depicts one embodiment of a field of view as seen by a user wearing a HMD such asmobile device140 inFIG. 1. The user may see within the field of view both real objects and virtual objects. The real objects may include achair16 and a mapping system10 (e.g., comprising a portion of an entertainment system). The virtual objects may include avirtual monster17. As thevirtual monster17 is displayed or overlaid over the real-world environment as perceived through the see-through lenses of the HMD, the user may perceive that thevirtual monster17 exists within the real-world environment.
• FIG. 3B depicts one embodiment of an AR environment. The AR environment includes amapping system10 andmobile devices18 and19. Themobile devices18 and19, depicted as special glasses inFIG. 3B, may include a HMD such asHMD200 inFIG. 2. Themapping system10 may include acomputing environment12, acapture device20, and adisplay14, all in communication with each other.Computing environment12 may include one or more processors.Capture device20 may include a color or depth sensing camera that may be used to visually monitor one or more targets including humans and one or more other objects within a particular environment. In one example,capture device20 may comprise an RGB or depth camera andcomputing environment12 may comprise a set-top box or gaming console.Mapping system10 may support multiple mobile devices or clients.
• Themobile devices18 and19 may include HMDs. As shown inFIG. 3B,user28 wearsmobile device18 anduser29 wearsmobile device19. Themobile devices18 and19 may receive virtual data frommapping system10 such that a virtual object is perceived to exist within a field of view as displayed through the respective mobile device. For example, as seen byuser28 throughmobile device18, the virtual object is displayed as the back ofvirtual monster17. As seen byuser29 throughmobile device19, the virtual object is displayed as the front ofvirtual monster17 appearing above the back ofchair16. The rendering ofvirtual monster17 may be performed bymapping system10 or bymobile devices18 and19. In one embodiment,mapping system10 renders images ofvirtual monster17 associated with a field of view of a particular mobile device and transmits the rendered images to the particular mobile device.
• FIG. 3C depicts one embodiment of an AR environment utilizing themapping system10 andmobile devices18 and19 depicted inFIG. 3B. Themapping system10 may track and analyze virtual objects within a particular environment such asvirtual ball27. Themapping system10 may also track and analyze real objects within the particular environment such asuser28,user29, andchair16. The rendering of images associated withvirtual ball27 may be performed bymapping system10 or bymobile devices18 and19.
• In one embodiment,mapping system10 tracks the position of virtual objects by taking into consideration the interaction between real and virtual objects. For example,user18 may move their arm such thatuser18 perceives hittingvirtual ball27. Themapping system10 may subsequently apply a virtual force tovirtual ball27 such that bothusers28 and29 perceive that the virtual ball has been hit byuser28. In one example,mapping system10 may register the placement ofvirtual ball27 within a 3-D map of the particular environment and provide virtual data information tomobile devices18 and19 such thatusers28 and29 perceive thevirtual ball27 as existing within the particular environment from their respective points of view. In another embodiment, a particular mobile device may render virtual objects that are specific to the particular mobile device. For example, if thevirtual ball27 is only rendered onmobile device18 then thevirtual ball27 would only be perceived as existing within the particular environment byuser28. In some embodiments, the dynamics of virtual objects may be performed on the particular mobile device and not on the mapping system.
• FIG. 4 illustrates one embodiment of amapping system50 including acapture device58 andcomputing environment54.Mapping system50 is one example of an implementation formapping system10 inFIGS. 3B-3C. For example, computingenvironment54 may correspond withcomputing environment12 inFIGS. 3B-3C andcapture device58 may correspond withcapture device20 inFIGS. 3B-3C.
• In one embodiment, thecapture device58 may include one or more image sensors for capturing images and videos. An image sensor may comprise a CCD image sensor or a CMOS sensor. In some embodiments,capture device58 may include an IR CMOS image sensor. Thecapture device58 may also include a depth camera (or depth sensing camera) configured to capture video with depth information including a depth image that may include depth values via any suitable technique including, for example, time-of-flight, structured light, stereo image, or the like.
• Thecapture device58 may include an image camera component32. In one embodiment, the image camera component32 may include a depth camera that may capture a depth image of a scene. The depth image may include a two-dimensional (2-D) pixel area of the captured scene where each pixel in the 2-D pixel area may represent a depth value such as a distance in, for example, centimeters, millimeters, or the like of an object in the captured scene from the camera.
• The image camera component32 may include anIR light component34, a three-dimensional (3-D)camera36, and anRGB camera38 that may be used to capture the depth image of a capture area. For example, in time-of-flight analysis, theIR light component34 of thecapture device58 may emit an infrared light onto the capture area and may then use sensors to detect the backscattered light from the surface of one or more objects in the capture area using, for example, the 3-D camera36 and/or theRGB camera38. In some embodiments, pulsed infrared light may be used such that the time between an outgoing light pulse and a corresponding incoming light pulse may be measured and used to determine a physical distance from thecapture device58 to a particular location on the one or more objects in the capture area. Additionally, the phase of the outgoing light wave may be compared to the phase of the incoming light wave to determine a phase shift. The phase shift may then be used to determine a physical distance from the capture device to a particular location associated with the one or more objects.
• In another example, thecapture device58 may use structured light to capture depth information. In such an analysis, patterned light (i.e., light displayed as a known pattern such as grid pattern or a stripe pattern) may be projected onto the capture area via, for example, theIR light component34. Upon striking the surface of one or more objects (or targets) in the capture area, the pattern may become deformed in response. Such a deformation of the pattern may be captured by, for example, the 3-D camera36 and/or theRGB camera38 and analyzed to determine a physical distance from the capture device to a particular location on the one or more objects.
• In some embodiments, two or more different cameras may be incorporated into an integrated capture device. For example, a depth camera and a video camera (e.g., an RGB video camera) may be incorporated into a common capture device. In some embodiments, two or more separate capture devices of the same or differing types may be cooperatively used. For example, a depth camera and a separate video camera may be used, two video cameras may be used, two depth cameras may be used, two RGB cameras may be used or any combination and number of cameras may be used. In one embodiment, thecapture device58 may include two or more physically separated cameras that may view a capture area from different angles to obtain visual stereo data that may be resolved to generate depth information. Depth may also be determined by capturing images using a plurality of detectors that may be monochromatic, infrared, RGB, or any other type of detector and performing a parallax calculation. Other types of depth image sensors can also be used to create a depth image.
• As shown inFIG. 4,capture device58 may include amicrophone40. Themicrophone40 may include a transducer or sensor that may receive and convert sound into an electrical signal.
• Thecapture device58 may include aprocessor42 that may be in operative communication with the image camera component32. The processor may include a standardized processor, a specialized processor, a microprocessor, or the like. Theprocessor42 may execute instructions that may include instructions for storing filters or profiles, receiving and analyzing images, determining whether a particular situation has occurred, or any other suitable instructions. It is to be understood that at least some image analysis and/or target analysis and tracking operations may be executed by processors contained within one or more capture devices such ascapture device58.
• Thecapture device58 may include amemory44 that may store the instructions that may be executed by theprocessor42, images or frames of images captured by the 3-D camera or RGB camera, filters or profiles, or any other suitable information, images, or the like. In one example, thememory44 may include random access memory (RAM), read only memory (ROM), cache, Flash memory, a hard disk, or any other suitable storage component. As shown inFIG. 4, thememory44 may be a separate component in communication with the image capture component32 and theprocessor42. In another embodiment, thememory44 may be integrated into theprocessor42 and/or the image capture component32. In other embodiments, some or all of thecomponents32,34,36,38,40,42 and44 of thecapture device58 illustrated inFIG. 4 are housed in a single housing.
• Thecapture device58 may be in communication with thecomputing environment54 via acommunication link46. Thecommunication link46 may be a wired connection including, for example, a USB connection, a FireWire connection, an Ethernet cable connection, or the like and/or a wireless connection such as a wireless 802.11b, g, a, or n connection. Thecomputing environment54 may provide a clock to thecapture device58 that may be used to determine when to capture, for example, a scene via thecommunication link46. In one embodiment, thecapture device58 may provide the images captured by, for example, the3D camera36 and/or theRGB camera38 to thecomputing environment54 via thecommunication link46.
• As shown inFIG. 4, computingenvironment54 includes image andaudio processing engine194 in communication withoperating system196. Image andaudio processing engine194 includesvirtual data engine197, gesture recognizer engine190, structure data198, processingunit191, andmemory unit192, all in communication with each other. Image andaudio processing engine194 processes video, image, and audio data received fromcapture device58. To assist in the detection and/or tracking of objects, image andaudio processing engine194 may utilize structure data198 and gesture recognition engine190.Virtual data engine197 processes virtual objects and registers the position and orientation of virtual objects in relation to various maps of a real-world environment stored inmemory unit192.
• Processing unit191 may include one or more processors for executing object, facial, and voice recognition algorithms. In one embodiment, image andaudio processing engine194 may apply object recognition and facial recognition techniques to image or video data. For example, object recognition may be used to detect particular objects (e.g., soccer balls, cars, or landmarks) and facial recognition may be used to detect the face of a particular person. Image andaudio processing engine194 may apply audio and voice recognition techniques to audio data. For example, audio recognition may be used to detect a particular sound. The particular faces, voices, sounds, and objects to be detected may be stored in one or more memories contained inmemory unit192.
• In some embodiments, one or more objects being tracked may be augmented with one or more markers such as an IR retroreflective marker to improve object detection and/or tracking. Planar reference images, coded AR markers, QR codes, and/or bar codes may also be used to improve object detection and/or tracking. Upon detection of one or more objects, image andaudio processing engine194 may report tooperating system196 an identification of each object detected and a corresponding position and/or orientation.
• The image andaudio processing engine194 may utilize structural data198 while performing object recognition. Structure data198 may include structural information about targets and/or objects to be tracked. For example, a skeletal model of a human may be stored to help recognize body parts. In another example, structure data198 may include structural information regarding one or more inanimate objects in order to help recognize the one or more inanimate objects.
• The image andaudio processing engine194 may also utilize gesture recognizer engine190 while performing object recognition. In one example, gestures recognizer engine190 may include a collection of gesture filters, each comprising information concerning a gesture that may be performed by a skeletal model. The gesture recognition engine190 may compare the data captured bycapture device58 in the form of the skeletal model and movements associated with it to the gesture filters in a gesture library to identify when a user (as represented by the skeletal model) has performed one or more gestures. In one example, image andaudio processing engine194 may use the gesture recognition engine190 to help interpret movements of a skeletal model and to detect the performance of a particular gesture.
• More information about the detection and tracking of objects can be found in U.S. patent application Ser. No. 12/641,788, “Motion Detection Using Depth Images,” filed on Dec. 18, 2009; and U.S. patent application Ser. No. 12/475,308, “Device for Identifying and Tracking Multiple Humans over Time,” both of which are incorporated herein by reference in their entirety. More information about gesture recognizer engine190 can be found in U.S. patent application Ser. No. 12/422,661, “Gesture Recognizer System Architecture,” filed on Apr. 13, 2009, incorporated herein by reference in its entirety. More information about recognizing gestures can be found in U.S. patent application Pat. No. 12/391,150, “Standard Gestures,” filed on Feb. 23, 2009; and U.S. patent application Ser. No. 12/474,655, “Gesture Tool,” filed on May 29, 2009, both of which are incorporated by reference herein in their entirety.
• FIG. 5 depicts one embodiment of anAR system600.AR system600 includesmapping server620 andmobile device630.Mapping server620 may comprise a mapping system such asmapping system50 inFIG. 4.Mapping server620 may work asynchronously to build one or more maps associated with one or more real-world environments. The one or more maps may include sparse 3-D maps and/or dense 3-D maps based on images captured from a variety of sources including sensors dedicated tomapping server620.Mobile device630 may comprise an HMD such asmobile device140 inFIG. 1. Although a singlemobile device630 is depicted,mapping server620 may support numerous mobile devices.
• The process steps621-624 associated withmapping server620 are decoupled from and are performed independently of the process steps631-635 associated with themobile device630. Instep621, images are received from one or more server sensors. One example of the one or more server sensors includes the 3-D and/or RGB cameras within the image camera component32 inFIG. 4. The one or more server sensors may be associated with one or more stationary cameras and/or one or more mobile cameras which may move about a particular environment over time. The one or more mobile cameras may move in a predetermined or predictable manner about the particular environment (e.g., a dedicated mapping server camera that runs on tracks above a room or hangs from the ceiling of a room and rotates in a deterministic manner). In one embodiment, the one or more server sensors providing images to themapping server620 may move about an environment attached to a mobile robot or autonomous vehicle.
• Instep622, images received from the one or more server sensors are registered. During image registration, themapping server620 may register different images taken within a particular environment (e.g., images from different points of view, taken at different points in time, and/or images associated with different types of information such as color or depth information) into a single real-world coordinate system associated with the particular environment. In one example, image registration into a common coordinate system may be performed using an extrinsic calibration process. The registration and/or alignment of images (or objects within the images) onto a common coordinate system allows themapping server620 to be able to compare and integrate real-world objects, landmarks, or other features extracted from the different images into one or more maps associated with the particular environment. The one or more maps associated with a particular environment may comprise 3-D maps.
• More information about generating 3-D maps can be found in U.S. patent application Ser. No. 13/017,690, “Three-Dimensional Environment Reconstruction,” incorporated herein by reference in its entirety.
• In one embodiment, one or more additional images derived from sources decoupled frommapping server620 may be used to extend or update the one or more maps associated with the particular environment. For example,client images640 derived from sensors associated withmobile device630 may be used by mappingserver620 to refine the one or more maps.
• In some embodiments, the registration of one or more images may require knowledge of one or more poses associated with the one or more images. For example, for each image being registered, a six degree of freedom (6DOF) pose may be provided including information associated with the position and orientation of the particular sensor from which the image was captured. Mapping and pose estimation (or localization) associated with a particular sensor may be determined using traditional SLAM or PTAM techniques.
• Instep623, a sparse map representing a portion of a particular environment is generated. The sparse map may comprise a 3-D point cloud and may include one or more image descriptors. Themapping server620 may identify one or more image descriptors associated with a particular map of the one or more maps by applying various image processing methods such as object recognition, feature detection, corner detection, blob detection, and edge detection methods. The one or more image descriptors may be used as landmarks in determining a particular pose, position, and/or orientation in relation to the particular map. An image descriptor may include color and/or depth information associated with a particular object (e.g., a red apple) or a portion of a particular object within the particular environment (e.g., the top of a red apple). In some embodiments, an initial sparse map is generated first and then subsequently refined and updated over time. The initial sparse map may be limited to only covering static objects within the particular environment.
• Instep624, a dense map representing a portion of the particular environment is generated. The dense map may comprise a dense 3-D surface mesh, which may be created using the 3-D point cloud generated instep623. The dense map may provide sufficient detail of the particular environment so as to enable complex augmented reality applications such as those that must handle collisions and occlusions due to the interaction of real objects and virtual objects within the particular environment. In one embodiment, a particular dense map is not generated by mappingserver620 until a request for the particular dense map is received frommobile device630. In another embodiment, amobile device630 may request a particular dense map first without requesting a related sparse map. In some embodiments, the process steps621-624 associated withmapping server620 may be performed in the cloud.
• Asmobile device630 moves around a particular environment it may requestmapping data641 and/ormapping data642 associated with a particular environment. The decision to request eithermapping data641 ormapping data642 depends on the requirements of a particular AR application running onmobile device630. Instep631, one or more client images are received from one or more client sensors. One example of the one or more client sensors includes the 3-D and/or RGB cameras associated withHMD200 inFIG. 2. In one embodiment, the one or more client sensors detect light from a patterned or structured light source associated with and projected by mappingserver620. This allows several mobile devices to each independently utilize the same structured light source for localization purposes.
• Instep632, one or more client image descriptors are extracted from the one or more client images. Themobile device630 may identify one or more client image descriptors within the one or more client images by applying various image processing methods such as object recognition, feature detection, corner detection, blob detection, and edge detection methods. Instep633, a particular map is acquired frommapping server620. The particular map may be included withinmapping data641. The map may be a 3-D map and may include one or more image descriptors associated with one or more objects located within the particular environment described by the 3-D map. An updated map may be requested bymobile device630 every time the mobile device enters a new portion of the particular environment or at a predefined frequency (e.g., every 10 minutes).
• In order to augment a real-world environment as observed from the point of view ofmobile device630, the pose associated with a particular field of view ofmobile device630 in relation to the real-world environment must be determined. Instep634, a pose (e.g., a6DOF pose) associated with a field of view ofmobile device630 is determined via localization and/or pose estimation. Pose estimation may include the matching and aligning of detected image descriptors. For example, matching may be performed between the one or more client image descriptors and the image descriptors received frommapping server620 included withinmapping data641. In some embodiments, matching may take into consideration depth and color information associated with the one or more image descriptors. In other embodiments, the one or more client images comprise color images (e.g., RGB images) and matching is performed for only image descriptors including color information, thereby enabling image-based localization for mobile devices with only RGB sensor input. Alignment of matched image descriptors may be performed using image or frame alignment techniques, which may include determining point to point correspondences between the field of view ofmobile device630 and views derived from the particular map.
• In some embodiments,mobile device630 may include a GPS receiver, such asGPS receiver232 inFIG. 2, and a motion and orientation sensor, such as motion andorientation sensor238 inFIG. 2. Prior to image processing, a first-pass estimate for the pose associated withmobile device630 may be obtained by utilizing GPS location information and orientation information generated onmobile device630.Mobile device630 may also obtain location and orientation information by detecting, tracking, and triangulating the position of physical tags (e.g., reflective markers) or emitters (e.g., LEDs) attached to one or more other mobile devices using marker-based motion capture technologies. The location and/or orientation information associated withmobile device630 may be used to improve the accuracy of the localization step.
• In one embodiment, localization ofmobile device630 includes searching for image descriptors associated with landmarks or other objects within a field of view ofmobile device630. The extracted image descriptors may subsequently be matched with image descriptors associated with the most recent map of the particular environment acquired frommapping server620 instep623. Landmarks may be associated with image features that are easily observed and distinguished from other features within the field of view. Stationary landmarks may act as anchor points or reference points in a 3-D map of a real-world environment. Other points of interest within a real-world environment (e.g., geographical elements such as land features) may also be used for both mapping and localization purposes. Landmarks and other points of interest within the real-world environment may be detected using various image processing methods such as object recognition, frame alignment, feature detection, corner detection, blob detection, and edge detection methods. Asmobile device630 moves around within an environment,mobile device630 may continuously search for and extract landmarks and associate newly discovered landmarks to those found previously.
• For each object within an environment, there may be one or more image descriptors associated with the object that can be extracted and used to identify the object. An image descriptor may comprise image information related to a portion of an object or to the entire object. In one example, an image descriptor may describe characteristics of the object such as its location, color, texture, shape, and/or its relationship to other objects or landmarks within the environment. Utilizing image processing techniques such as object and pattern matching, the one or more image descriptors may be used to locate an object in an image containing other objects. It is desirable that the image processing techniques for detecting and matching the one or more image descriptors be robust to changes in image scale, noise, illumination, local geometric distortion, and image orientation.
• In some embodiments, localization ofmobile device630 may be performed in the cloud (e.g., by the mapping server620). For example, themobile device630 may transmitclient images640 derived from sensors associated withmobile device630 and/orimage descriptors644 extracted instep632 to themapping server620. Themapping server620 may perform the mobile device localization based on the receivedclient images640 and/orimage descriptors644 utilizing process steps similar to those performed bymobile device630 instep634. Subsequently,mobile device630 may acquire a pose associated with a field of view ofmobile device630 from themapping server620 without needing to acquire the particular map instep633 or perform localization step634 (i.e., steps633 and634 may be omitted if localization of the mobile device is performed by the mapping server).
• More information regarding performing pose estimation and/or localization for a mobile device can be found in U.S. patent application Ser. No. 13/017,474, “Mobile Camera Localization Using Depth Maps,” incorporated herein by reference in its entirety.
• More information regarding differentiating objects within an environment can be found in U.S. patent application Ser. No. 13/017,626, “Moving Object Segmentation Using Depth Images,” incorporated herein by reference in its entirety.
• Referring toFIG. 5, instep635, one or more virtual objects are displayed onmobile device630. The one or more virtual objects may be rendered onmobile device630 using a processing element, such asprocessing unit236 inFIG. 2. Rendering may involve the use of ray tracing techniques (e.g., simulating the propagation of light and sound waves through a model of the AR environment) in order to generate virtual images and/or sounds from the perspective of a mobile device. Virtual data associated with the one or more virtual objects may be generated by either themapping server620 and/ormobile device630.
• In one embodiment, the one or more virtual objects are rendered locally onmobile device630. In another embodiment, the one or more virtual objects are rendered onmapping server620 and virtual images corresponding with the determined pose instep634 may be transmitted tomobile device630. Virtual images associated with the one or more virtual objects may be displayed onmobile device630 such that the one or more virtual objects are perceived to exist within a field of view associated with the previously determined pose ofstep634. The virtual data associated with the one or more virtual object may also include virtual sounds. In some embodiments,mobile device630 may request a dense 3-D map of a particular environment in order to realistically render virtual objects and to handle collisions and occlusions due to the interaction of real objects and virtual objects within the particular environment. Virtual sounds may also be registered in relation to the dense 3-D map. The dense 3-D map may be acquired frommapping server620 and included withinmapping data642.
• More information regarding AR systems and environments can be found in U.S. patent application Ser. No. 12/912,937, “Low-Latency Fusing of Virtual and Real Content,” incorporated herein by reference in its entirety.
• More information regarding generating virtual sounds in an AR environment can be found in U.S. patent application Ser. No. 12/903,610, “System and Method for High-Precision 3-Dimensional Audio for Augmented Reality,” incorporated herein by reference in its entirety.
• FIG. 6A is a flowchart describing one embodiment of a process for generating a 3-D map of a real-world environment and locating virtual objects within the 3-D map. The process ofFIG. 6A may be performed continuously and by one or more computing devices. Each step in the process ofFIG. 6A may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device. In one embodiment, the process ofFIG. 6A is performed by a mapping server such asmapping server620 inFIG. 5.
• Instep702, a 3-D map of a real-world environment is generated. Instep704, one or more virtual objects may be computer generated. In one example, the 3-D map and the one or more virtual objects are generated using a mapping server. The one or more virtual objects may be stored in a database on the mapping server. Instep706, the one or more virtual objects may be registered in relation to the 3-D map. If registration is performed by the mapping server, then the mapping server may control the location of the virtual objects being displayed by one or more mobile devices.
• Instep708, a determination is made as to whether or not the location of one or more virtual objects in relation to the 3-D map has changed. The change in location may be due to a perceived interaction between one or more virtual objects and/or one or more real objects, or due to the natural movement of a virtual object through an environment (e.g., via physics simulation). If the location of one or more virtual objects has changed, then the one or more virtual objects are reregistered in relation to the 3-D map instep706. If the one or more virtual objects are all characterized as static then step708 may be omitted.
• Instep710, location information associated with one or more mobile devices is received. The received location information may be used by a mapping server to customize a particular map associated with the location information. Instep712, at least a portion of the 3-D map is outputted to the one or more mobile devices. The 3-D map may include a customized map of an environment associated with the location information received instep710. The 3-D map may include one or more image descriptors. The one or more image descriptors may include RGB image descriptors and/or 3-D image descriptors. In one embodiment, a mapping server outputs one of more image descriptors in response to a request from a mobile device for either RGB image descriptors or 3-D image descriptors depending on the requirements of a particular application running on the mobile device. Instep714, virtual data associated with the one or more virtual objects is outputted to the one or more mobile devices.
• FIG. 6B is a flowchart describing one embodiment of a process for updating a 3-D map of a real-world environment. The process ofFIG. 6B may be performed continuously and by one or more computing devices. Each step in the process ofFIG. 6B may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device. In one embodiment, the process ofFIG. 6B is performed by a mapping server such asmapping server620 inFIG. 5.
• Instep802, one or more images of a first environment is acquired from one or more cameras associated with a mapping server. Instep804, the one or more images are registered. Instep806, a first map of the first environment is generated. Instep808, one or more objects existing within the first environment are detected and characterized. Each of the one or more objects may be characterized as either static, semi-static, or dynamic. An object characterized as static is one which does not move. An object characterized as semi-static is one which does not move often. Semi-static objects may be useful for mapping and/or localization purposes because they may act as static landmarks over a period of time within an environment. Object recognition techniques may be used to identify objects, such as furniture or appliances, that may typically be classified as semi-static objects. In one embodiment, maps of an environment are stored and time stamped (e.g., every 2 hours) in order to detect movement of landmarks and to possibly recharacterize one or more objects. In some embodiments, mapping and localization are performed using only objects classified as static objects, thereby discarding any objects characterized as either semi-static or dynamic (i.e., not taking into consideration any moving objects).
• Instep810, one or more additional images of the first environment are received from a first mobile device. Localization information associated with the one or more additional images may also be received from the first mobile device. Obtaining pose information along with one or more additional images allows a mapping server to generate one or more maps associated with an environment beyond that capable of being sensed by dedicated mapping server sensors. The one or more additional images may also include depth information. Instep812, the one or more additional images are registered. In some embodiments, the one or more additional images are registered by a mapping server. A mapping server may also accept information from one or more mobile devices in the form of registered RGB data and/or depth reconstructions. Instep814, the first map is updated based on the one or more additional images. Instep816, one or more new objects existing within the first environment are detected and characterized. Instep818, at least a subset of the updated first map is transmitted to a second mobile device. In one embodiment, a first set of mobile devices may be used to update the first map, while a second set of mobile devices perform localization in relation to the updated first map.
• FIG. 7 is a flowchart describing one embodiment of a process for rendering and displaying virtual objects. The process ofFIG. 7 may be performed continuously and by one or more computing devices. Each step in the process ofFIG. 7 may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device. In one embodiment, the process ofFIG. 7 is performed by a mobile device such asmobile device630 inFIG. 5.
• Instep902, it is determined whether a first map is required. In one example, a mobile device may determine whether a first map is required for localization purposes by determining a location of the mobile device (e.g., by acquiring a GPS location) and comparing the location with the locations associated with one or more maps stored on the mobile device. A first map may be required if none of the one or more maps stored on the mobile device provide adequate coverage of a particular environment or no map is deemed to be a valid map of the environment encompassing the location. A particular map stored on a mobile device may be deemed invalid because the map has not been refreshed within a certain period of time or an expiration date associated with the particular map has passed. In one example, a particular map stored on a mobile device may be invalidated by a mapping server upon an update to the particular map made by the mapping server.
• Instep904, the first map is acquired. The first map may be acquired from a mapping server. In one embodiment, a mobile device may transmit a request for the first map to a mapping server. In another embodiment, a mobile device may transmit a request for the first map to a second mobile device located within a particular environment. In this case, the second mobile device may act as a local mapping cache for the mobile device. The first map may include one or more image descriptors associated with one or more real objects within a particular environment. Instep906, the first map is stored. The first map may be stored in a non-volatile memory within a mobile device.
• Instep908, a first pose associated with the mobile device in relation to the first map is determined. The pose estimation or determination may be performed on a mobile device or on a mapping server. In one embodiment, localization is performed on a mobile device using the process ofstep634 inFIG. 5. When performing localization on the mobile device, image descriptors associated with non-static objects may be discarded. In another embodiment, a mobile device may transmit images of its environment to a mapping server and the mapping server may perform the localization process on behalf of the mobile device.
• Instep910, it is determined whether a second map is required. In one example, a mobile device may determine whether a second map is required for rendering and/or display purposes by determining whether a virtual object is located within (or near) a field of view associated with the first pose determined instep908. The second map may comprise a higher resolution map of a portion of the first map. Instep912, the second map is acquired. Instep914, the second map is stored. Instep916, a virtual object is registered in relation to the second map. In one example, a mobile device registers the virtual object in relation to the second map prior to rendering the virtual object.
• Instep918, the virtual object is rendered. The rendering step may include receiving virtual data associated with the virtual object from a mapping server and generating one or more virtual images of the virtual object based on the virtual data. Instep920, the virtual object is displayed. The virtual object may be displayed by displaying on a mobile device one or more virtual images associated with the virtual object. The one or more virtual images may correspond with a view of the virtual object such that the virtual object is perceived to exist within the field of view associated with the first pose determined instep908.
• The disclosed technology may be used with various computing systems.FIGS. 8-10 provide examples of various computing systems that can be used to implement embodiments of the disclosed technology.
• FIG. 8 is a block diagram of an embodiment of a gaming andmedia system7201.Console7203 has a central processing unit (CPU)7200, and amemory controller7202 that facilitates processor access to various types of memory, including a flash Read Only Memory (ROM)7204, a Random Access Memory (RAM)7206, ahard disk drive7208, and portable media drive7107. In one implementation,CPU7200 includes alevel 1cache7210 and alevel 2cache7212, to temporarily store data and hence reduce the number of memory access cycles made to thehard drive7208, thereby improving processing speed and throughput.
• CPU7200,memory controller7202, and various memory devices are interconnected via one or more buses (not shown). The one or more buses might include one or more of serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus, using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus.
• In one implementation,CPU7200,memory controller7202,ROM7204, andRAM7206 are integrated onto acommon module7214. In this implementation,ROM7204 is configured as a flash ROM that is connected tomemory controller7202 via a PCI bus and a ROM bus (neither of which are shown).RAM7206 is configured as multiple Double Data Rate Synchronous Dynamic RAM (DDR SDRAM) modules that are independently controlled bymemory controller7202 via separate buses (not shown).Hard disk drive7208 and portable media drive7107 are shown connected to thememory controller7202 via the PCI bus and an AT Attachment (ATA)bus7216. However, in other implementations, dedicated data bus structures of different types may also be applied in the alternative.
• A three-dimensionalgraphics processing unit7220 and avideo encoder7222 form a video processing pipeline for high speed and high resolution (e.g., High Definition) graphics processing. Data are carried fromgraphics processing unit7220 tovideo encoder7222 via a digital video bus (not shown). Anaudio processing unit7224 and an audio codec (coder/decoder)7226 form a corresponding audio processing pipeline for multi-channel audio processing of various digital audio formats. Audio data are carried betweenaudio processing unit7224 andaudio codec7226 via a communication link (not shown). The video and audio processing pipelines output data to an A/V (audio/video)port7228 for transmission to a television or other display. In the illustrated implementation, video and audio processing components7220-7228 are mounted onmodule7214.
• FIG. 8 showsmodule7214 including a USB host controller7230 and anetwork interface7232. USB host controller7230 is in communication withCPU7200 andmemory controller7202 via a bus (not shown) and serves as host for peripheral controllers7205(1)-7205(4).Network interface7232 provides access to a network (e.g., Internet, home network, etc.) and may be any of a wide variety of various wire or wireless interface components including an Ethernet card, a modem, a wireless access card, a Bluetooth® module, a cable modem, and the like.
• In the implementation depicted inFIG. 8,console7203 includes acontroller support subassembly7240 for supporting four controllers7205(1)-7205(4). Thecontroller support subassembly7240 includes any hardware and software components needed to support wired and wireless operation with an external control device, such as for example, a media and game controller. A front panel I/O subassembly7242 supports the multiple functionalities ofpower button7213, theeject button7215, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface ofconsole7203.Subassemblies7240 and7242 are in communication withmodule7214 via one ormore cable assemblies7244. In other implementations,console7203 can include additional controller subassemblies. The illustrated implementation also shows an optical I/O interface7235 that is configured to send and receive signals (e.g., from remote control7290) that can be communicated tomodule7214.
• MUs7241(1) and7241(2) are illustrated as being connectable to MU ports “A”7231(1) and “B”7231(2) respectively. Additional MUs (e.g., MUs7241(3)-7241(6)) are illustrated as being connectable to controllers7205(1) and7205(3), i.e., two MUs for each controller. Controllers7205(2) and7205(4) can also be configured to receive MUs (not shown). EachMU7241 offers additional storage on which games, game parameters, and other data may be stored. Additional memory devices, such as portable USB devices, can be used in place of the MUs. In some implementations, the other data can include any of a digital game component, an executable gaming application, an instruction set for expanding a gaming application, and a media file. When inserted intoconsole7203 or a controller,MU7241 can be accessed bymemory controller7202. A systempower supply module7250 provides power to the components ofgaming system7201. Afan7252 cools the circuitry withinconsole7203.
• Anapplication7260 comprising machine instructions is stored onhard disk drive7208. Whenconsole7203 is powered on, various portions ofapplication7260 are loaded intoRAM7206, and/orcaches7210 and7212, for execution onCPU7200. Other applications may also be stored onhard disk drive7208 for execution onCPU7200.
• Gaming andmedia system7201 may be operated as a standalone system by simply connecting the system to a monitor, a television, a video projector, or other display device. In this standalone mode, gaming andmedia system7201 enables one or more players to play games or enjoy digital media (e.g., by watching movies or listening to music). However, with the integration of broadband connectivity made available throughnetwork interface7232, gaming andmedia system7201 may further be operated as a participant in a larger network gaming community.
• FIG. 9 is a block diagram of one embodiment of amobile device8300. Mobile devices may include laptop computers, pocket computers, mobile phones, personal digital assistants, and handheld media devices that have been integrated with wireless receiver/transmitter technology.
• Mobile device8300 includes one ormore processors8312 andmemory8310.Memory8310 includesapplications8330 and non-volatile storage8340.Memory8310 can be any variety of memory storage media types, including non-volatile and volatile memory. A mobile device operating system handles the different operations of themobile device8300 and may contain user interfaces for operations, such as placing and receiving phone calls, text messaging, checking voicemail, and the like. Theapplications8330 can be any assortment of programs, such as a camera application for photos and/or videos, an address book, a calendar application, a media player, an internet browser, games, an alarm application, and other applications. The non-volatile storage component8340 inmemory8310 may contain data such as music, photos, contact data, scheduling data, and other files.
• The one ormore processors8312 also communicates with RF transmitter/receiver8306 which in turn is coupled to anantenna8302, with infrared transmitter/receiver8308, with global positioning service (GPS)receiver8365, and with movement/orientation sensor8314 which may include an accelerometer and/or magnetometer. RF transmitter/receiver8308 may enable wireless communication via various wireless technology standards such as Bluetooth® or the IEEE 802.11 standards. Accelerometers have been incorporated into mobile devices to enable applications such as intelligent user interface applications that let users input commands through gestures, and orientation applications which can automatically change the display from portrait to landscape when the mobile device is rotated. An accelerometer can be provided, e.g., by a micro-electromechanical system (MEMS) which is a tiny mechanical device (of micrometer dimensions) built onto a semiconductor chip. Acceleration direction, as well as orientation, vibration, and shock can be sensed. The one ormore processors8312 further communicate with a ringer/vibrator8316, a user interface keypad/screen8318, aspeaker8320, amicrophone8322, acamera8324, alight sensor8326, and atemperature sensor8328. The user interface keypad/screen may include a touch-sensitive screen display.
• The one ormore processors8312 controls transmission and reception of wireless signals. During a transmission mode, the one ormore processors8312 provide voice signals frommicrophone8322, or other data signals, to the RF transmitter/receiver8306. The transmitter/receiver8306 transmits the signals through theantenna8302. The ringer/vibrator8316 is used to signal an incoming call, text message, calendar reminder, alarm clock reminder, or other notification to the user. During a receiving mode, the RF transmitter/receiver8306 receives a voice signal or data signal from a remote station through theantenna8302. A received voice signal is provided to thespeaker8320 while other received data signals are processed appropriately.
• Additionally, aphysical connector8388 may be used to connect themobile device8300 to an external power source, such as an AC adapter or powered docking station, in order to rechargebattery8304. Thephysical connector8388 may also be used as a data connection to an external computing device. The data connection allows for operations such as synchronizing mobile device data with the computing data on another device.
• FIG. 10 is a block diagram of an embodiment of acomputing system environment2200.Computing system environment2200 includes a general purpose computing device in the form of acomputer2210. Components ofcomputer2210 may include, but are not limited to, aprocessing unit2220, asystem memory2230, and asystem bus2221 that couples various system components including thesystem memory2230 to theprocessing unit2220. Thesystem bus2221 may be any of several types of bus structures including a memory bus, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
• Computer2210 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed bycomputer2210 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed bycomputer2210. Combinations of the any of the above should also be included within the scope of computer readable media.
• Thesystem memory2230 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)2231 and random access memory (RAM)2232. A basic input/output system2233 (BIOS), containing the basic routines that help to transfer information between elements withincomputer2210, such as during start-up, is typically stored inROM2231.RAM2232 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on byprocessing unit2220. By way of example, and not limitation,FIG. 10 illustratesoperating system2234,application programs2235,other program modules2236, andprogram data2237.
• Thecomputer2210 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 10 illustrates ahard disk drive2241 that reads from or writes to non-removable, nonvolatile magnetic media, amagnetic disk drive2251 that reads from or writes to a removable, nonvolatilemagnetic disk2252, and anoptical disk drive2255 that reads from or writes to a removable, nonvolatileoptical disk2256 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. Thehard disk drive2241 is typically connected to thesystem bus2221 through an non-removable memory interface such asinterface2240, andmagnetic disk drive2251 andoptical disk drive2255 are typically connected to thesystem bus2221 by a removable memory interface, such asinterface2250.
• The drives and their associated computer storage media discussed above and illustrated inFIG. 10, provide storage of computer readable instructions, data structures, program modules and other data for thecomputer2210. InFIG. 10, for example,hard disk drive2241 is illustrated as storingoperating system2244,application programs2245,other program modules2246, andprogram data2247. Note that these components can either be the same as or different fromoperating system2234,application programs2235,other program modules2236, andprogram data2237.Operating system2244,application programs2245,other program modules2246, andprogram data2247 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information intocomputer2210 through input devices such as akeyboard2262 andpointing device2261, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to theprocessing unit2220 through auser input interface2260 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). Amonitor2291 or other type of display device is also connected to thesystem bus2221 via an interface, such as avideo interface2290. In addition to the monitor, computers may also include other peripheral output devices such asspeakers2297 andprinter2296, which may be connected through anoutput peripheral interface2295.
• Thecomputer2210 may operate in a networked environment using logical connections to one or more remote computers, such as aremote computer2280. Theremote computer2280 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to thecomputer2210, although only amemory storage device2281 has been illustrated inFIG. 10. The logical connections depicted inFIG. 10 include a local area network (LAN)2271 and a wide area network (WAN)2273, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
• When used in a LAN networking environment, thecomputer2210 is connected to theLAN2271 through a network interface oradapter2270. When used in a WAN networking environment, thecomputer2210 typically includes amodem2272 or other means for establishing communications over theWAN2273, such as the Internet. Themodem2272, which may be internal or external, may be connected to thesystem bus2221 via theuser input interface2260, or other appropriate mechanism. In a networked environment, program modules depicted relative to thecomputer2210, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 10 illustratesremote application programs2285 as residing onmemory device2281. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
• The disclosed technology is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the technology include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
• The disclosed technology may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, software and program modules as described herein include routines, programs, objects, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Hardware or combinations of hardware and software may be substituted for software modules as described herein.
• The disclosed technology may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
• For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” are used to described different embodiments and do not necessarily refer to the same embodiment.
• For purposes of this document, a connection can be a direct connection or an indirect connection (e.g., via another part).
• For purposes of this document, the term “set” of objects, refers to a “set” of one or more of the objects.
• Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (20)

1. A method for providing an augmented reality environment on a mobile device, comprising:

determining whether a localization map is required, the step of determining whether a localization map is required is performed on the mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment;

acquiring the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the step of determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the step of determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the step of determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device;

rendering the virtual object, the step of rendering includes registering the virtual object in relation to the rendering map; and

displaying on the mobile device a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view associated with the first pose.

2. The method ofclaim 1, wherein:

the step of rendering is performed on the mobile device, the step of rendering includes receiving virtual data associated with the virtual object from a mapping server, the step of rendering includes generating the virtual image of the virtual object based on the virtual data, the step of rendering includes generating virtual sounds associated with the virtual object.

3. The method ofclaim 1, wherein:

the step of determining whether a localization map is required includes determining whether another map stored on the mobile device is valid and associated with the first environment;

the step of acquiring a first location includes acquiring the first location from a GPS receiver associated with the mobile device;

each of the one or more real objects are characterized as one of static objects or semi-static objects;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the mobile device includes a head mounted display device.

4. The method ofclaim 1, wherein:

the step of determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the step of determining a first pose includes discarding each of the one or more image descriptors that are associated with semi-static objects.

5. The method ofclaim 1, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a mapping server.

6. The method ofclaim 1, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.

7. The method ofclaim 1, wherein:

the step of determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device.

8. The method ofclaim 1, wherein:

the step of acquiring the localization map includes transmitting the first location associated with the mobile device.

9. An electronic device for providing an augmented reality environment, comprising:

one or more processors, the one or more processors determine whether a localization map is required, the one or more processors request the localization map from a mapping server;

a network interface, the network interface receives the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within a first environment, the one or more processors determine a first pose associated with a field of view of the electronic device, the network interface receives virtual data associated with the virtual object, the one or more processors determine whether a rendering map is required based on the location of the virtual object within the field of view, the network interface receives the rendering map, the rendering map has a higher resolution than the localization map;

a memory, the memory stores the rendering map, the one or more processors render the virtual object in relation to the rendering map; and

a display, the display displays a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view.

10. The electronic device ofclaim 9, wherein:

the electronic device is a mobile device; and

the network interface receives virtual data from a mapping server.

11. The electronic device ofclaim 9, wherein:

each of the one or more real objects are characterized as one of static objects or semi-static objects;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the electronic device includes a head mounted display device.

12. The electronic device ofclaim 9, wherein:

the determination of the first pose includes determining a first position and a first orientation of the electronic device in relation to the localization map.

13. The electronic device ofclaim 9, wherein:

the network interface receives the localization map from a mapping server.

14. The electronic device ofclaim 9, wherein:

the network interface receives the localization map from a second electronic device, the second electronic device is a mobile device.

15. One or more storage devices containing processor readable code for programming one or more processors to perform a method comprising the steps of:

determining whether a localization map is required, the step of determining whether a localization map is required is performed on a mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment, the step of determining whether a localization map is required includes determining whether another map stored on the mobile device is valid and associated with the first environment;

acquiring the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the step of determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the step of determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the step of determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the step of determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device;

rendering the virtual object, the step of rendering includes registering the virtual object in relation to the rendering map; and

playing on the mobile device a virtual sound associated with the virtual object, the virtual sound is perceived to originate from a location associated with the virtual object.

16. The one or more storage devices ofclaim 15, wherein:

the step of rendering is performed on the mobile device, the step of rendering includes receiving virtual data associated with the virtual object from a mapping server.

17. The one or more storage devices ofclaim 16, wherein:

the step of acquiring a first location includes acquiring the first location from a GPS receiver associated with the mobile device;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the mobile device includes a head mounted display device.

18. The one or more storage devices ofclaim 16, wherein:

the step of determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the step of determining a first pose includes discarding each of the one or more image descriptors that are characterized as non-static objects.

19. The one or more storage devices ofclaim 18, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a mapping server.

20. The one or more storage devices ofclaim 18, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.

Images