US20180053315A1 - Systems and methods for utilizing anchor graphs in mixed reality environments - Google Patents

Abstract

Mixed-reality systems are provided for using anchor graphs within a mixed-reality environment. These systems utilize anchor vertexes that comprise at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element. Anchor edges comprising transformations connect the anchor vertexes.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of U.S. patent application Ser. No. 15/169,045 filed on May 31, 2016, entitled “SYSTEMS AND METHODS FOR UTILIZING ANCHOR GRAPHS IN MIXED REALITY ENVIRONMENTS,” which issued as U.S. Pat. No. ______on ______, and which application is expressly incorporated herein by reference in its entirety.
BACKGROUND
• Mixed-reality computer systems have recently received significant interest for their ability to create immersive experiences for users. Mixed-reality systems include virtual reality, augmented reality, and other systems that incorporate virtual elements into a real-world environment.
• One particular interest in mixed-reality applications is the ability for multiple users to have common shared experiences within a shared mixed-reality worldspace. For example, a first user may create a mixed-reality element that is accessed and viewed by others that are navigating through the shared worldspace. However, several technical challenges and obstacles have to be addressed to allow a seamless shared experience. For example, the respective mixed-reality systems need to establish some form of common coordinates. The recent emergence of mixed-reality applications for mobile computing devices has, at least in part, been enabled by advances in Simultaneous Location and Mapping (“SLAM”) tracking systems and the broad availability of mobile depth camera modules. Convincing and seamless experiences in a shared mixed-reality worldspace can be accomplished by sharing accurately registered SLAM coordinate systems between client devices.
• Sharing SLAM coordinate systems with large mixed-reality worldspaces can be memory and processor intensive. For example, sharing large numbers of key frames between different devices often involves sending the same large amounts of data to and from multiple devices. Transmitting such large amounts of data can lead to both network performance issues due to constrained bandwidth and memory issues at each user device. Accordingly, there is a need for improved shared SLAM coordinate systems that are band-width and memory efficient.
• The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.
BRIEF SUMMARY
• Embodiments disclosed herein comprise systems, methods, and apparatuses configured to efficiently utilize both network bandwidth and device storage within shared mixed-reality coordinate systems. In particular, disclosed embodiments comprise mixed-reality devices and mixed-reality server systems for generating anchor vertexes that comprise one or more key frames, a mixed-reality element, and at least one transform connecting the one or more key frames to the mixed-reality element. Additionally, the mixed-reality devices and mixed-reality server systems are configured to generate an anchor edge that comprises a transform between a first anchor vertex and a second anchor vertex. The various anchor vertexes and anchor edges are stored by the mixed-reality server systems and provided, as needed, to user devices.
• Disclosed embodiments include computer systems for generating an anchor graph for a mixed-reality environment. The systems comprise one or more processors and one or more computer-readable media having stored computer-executable instructions that are executable by the one or more processors to configure the computer system to perform various acts. For example, the executable instructions are operable to configure the one or more processors to identify a first anchor vertex that includes a first set of one or more key frames, a first mixed-reality element, and at least one first transform connecting at least one key frame of the first set of the one or more key frames to the first mixed-reality element.
• The executable instructions are also operable to configure the one or more processors to identify a second anchor vertex that includes a second set of one or more key frames, a second mixed-reality element, and at least one second transform connecting at least one key frame of the second set of the one or more key frames to the second mixed-reality element. Additionally, the executable instructions are further operable to configure the one or more processors to create a first anchor edge between the first anchor vertex and the second anchor vertex and to configure the one or more processors to save the first anchor vertex, the second anchor vertex, and the anchor edge within an anchor graph data structure.
• Additional disclosed embodiments include methods for using an anchor graph within a mixed-reality environment. These methods include acts for identifying a first device with a stored first anchor graph and for detecting when the first device is within a predetermined proximity to a first physical location. Then, in response to detecting that the first device is within the predetermined proximity, a first anchor vertex and a first anchor edge are transmitted to the first device. The first anchor vertex comprises at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element. The first anchor edge comprises a transformation connecting the first anchor vertex to another anchor vertex.
• Further disclosed embodiments include methods for using an anchor graph within a mixed-reality environment. These methods include a first user device communicating, to a server, location data associated with a location of the first user device. The first user device then receives, in response to the communication, a first anchor vertex that is within a predetermined proximity to the first device and a first anchor edge. The first anchor vertex includes at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element. The first anchor edge comprises a transformation connecting the first anchor vertex to another anchor vertex.
• This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
• Additional features and advantages will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the teachings herein. Features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.
BRIEF DESCRIPTION OF THE DRAWINGS
• In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments and are not therefore to be considered to be limiting in scope, embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
• FIG. 1 illustrates a perspective view of a user viewing a sculpture through an embodiment of a mixed-reality headset.
• FIG. 2 illustrates a map of a user's pathway through a sculpture park while wearing an embodiment of a mixed-reality headset.
• FIG. 3 illustrates an embodiment of an anchor map of a user's pathway through the sculpture park ofFIG. 2.
• FIG. 4A illustrates an exploded view of the anchor map ofFIG. 3.
• FIG. 4B illustrates another exploded view of the anchor map ofFIG. 3.
• FIG. 5 illustrates a schematic view of an embodiment of a mixed-reality system.
• FIG. 6 illustrates a flowchart for an embodiment of a method for generating an anchor graph for a mixed-reality environment.
• FIG. 7 illustrates a flowchart for an embodiment of a method for using an anchor graph within a mixed-reality environment.
• FIG. 8 illustrates a flowchart for another embodiment of a method for using an anchor graph within a mixed-reality environment.
DETAILED DESCRIPTION
• Disclosed embodiments include systems, methods, and apparatuses that are configured to optimize the sharing of SLAM coordinate systems. In particular, disclosed embodiments improve network bandwidth consumption between different portions of a mixed-reality system. Additionally, disclosed embodiments dramatically reduce the amount of storage required to utilize shared SLAM coordinate systems.
• Some disclosed embodiments provide significant technical improvements to the field of mixed-reality computer systems for sharing coordinate systems and for referencing objects within a mixed-reality environment. The disclosed systems are capable of providing significant reductions in the quantity of data that must be transmitted and stored for enabling the shared use of a mixed-reality environment. For instance, rather than storing and transmitting a complete mapping of all elements in the shared environment to all systems, it is possible to transmit and share only relevant data that is sufficient to identify relative locations of the different systems/components within the shared environment. This may be accomplished, in some instances, by creating and using anchor vertexes and establishing/identifying transforms between those anchor vertexes.
• For example, some embodiments comprise mixed-reality devices and mixed-reality server systems for generating anchor vertexes that comprise one or more key frames, a mixed-reality element, and at least one transform connecting the one or more key frames to the mixed-reality element. Additionally, the mixed-reality devices and mixed-reality server systems also generate an anchor edges that comprise transforms between different anchor vertexes. The various anchor vertexes and anchor edges are stored by the mixed-reality server system and provided, as needed, to user devices.
• Turning now to the figures,FIG. 1 illustrates a perspective view of auser100 viewing asculpture120 through an embodiment of a mixed-reality headset110 (also referred to herein as a “mixed-reality device”). The mixed-reality headset110 may comprise various sensors, displays, processors, storage devices, network interfaces, and other similar computer components. The mixed-reality headset110 is capable of tracking the user's surroundings and displaying three-dimensional images that overlay the real-world environment of theuser100. One will understand that there are numerous different embodiments of mixed-reality devices, such as the depicted mixed-reality headset110, that can equivalently be used within the embodiments disclosed herein. In this regard, the depicted mixed-reality headset110 should not be viewed as limiting the scope of this disclosure, but merely one form of an exemplary system that is provided for the sake of clarity and explanation.
• InFIG. 1, theuser100 is depicted as viewing asculpture120 through the mixed-reality headset110. Theuser100 is also viewing a mixed-reality element130 that is rendered by the mixed-reality headset110. In the depicted embodiment, the mixed-reality element130 comprises a hologram of an art expert that is programed to provide information about thesculpture120. In alternate embodiments, however, the mixed-reality element130 is composed of a plurality of different computer-generated objects/components including, but not limited to, two-dimensional video, text information, two-dimensional images, audio data, interactive user interfaces, and any number of other equivalent and known computer generated user interface objects within mixed-reality systems. As such, while the present disclosure primarily describes mixed-reality elements, such as the hologram ofelement130, one will understand that mixed-reality elements within the scope of this disclosure can comprise any number and type of different augmented and virtual reality objects/content.
• FIG. 2 illustrates amap200 of a user'spathway260 through a sculpture park while wearing a mixed-reality headset110 (shown inFIG. 1). The depicted sculpture park comprises a number ofdifferent sculptures210,220,230,240, includingsculpture120 fromFIG. 1. Additionally, themap200 depicts variouskey frames250 that are generated by the mixed-reality headset110 as theuser100 walks along the user'spathway260. As used herein,key frames250 comprise visual samples that the mixed-reality headset110 gathers from an environment to establish a coordinate frame. For example, thekey frames250 may comprise image data and geolocation data that is received and processed by a camera within the mixed-reality headset110 as theuser100 walks along thepathway260.
• Tracing the user'spathway260 through the sculpture park,FIG. 2 illustrates that theuser100 first walks tosculpture120 where a first mixed-reality element130 is displayed. Theuser100 then continues along thepathway260 tosculpture210 where a second mixed-reality element212 is displayed to the user. The user'spathway260 then progresses tosculpture220 and an associated third mixed-reality element220. Theuser100 then travels tosculpture230 where an associated fourth mixed-reality element232 is displayed. The user'spathway260 continues tosculpture240 and an associated fifth mixed-reality element242. Finally, theuser100 returns tosculpture220 and a sixth mixed-reality element222.
• In at least one embodiment, theuser100 is capable of generating mixed-reality element (e.g.,222) that is associated with a particular sculpture (e.g.,220). For example, theuser100 may have access to a pre-made hologram of an artexpert describing sculpture220. In an alternative embodiment, theuser100 may create a wholly newhologram describing sculpture220. In any case, theuser100 provides a mixed-reality element222 that is associated withsculpture220. In at least one embodiment, theuser100 also determines the physical location of the mixed-reality element222 with respect to thesculpture220. This information can be entered intosystem110 and/or into a connected server system.
• Using methods that will be described more fully herein, a second user (not shown) may at a future time also approachsculpture220. If the second user is wearing a mixed-reality headset110, the mixed-reality element220 that was created byuser100 can be displayed to the second user (as provided throughsystem110 or the connected server system, based on the positioning of the second user). As such, a user is capable of both generating mixed-reality elements and receiving mixed-reality elements that were generated by others.
• FIG. 3 illustrates ananchor map300 corresponding to the user's traversedpathway260 through the sculpture park ofFIG. 2, starting fromlocation310.FIG. 3 also depicts thekey frames250, mixed-reality elements130,212,222,232,242, and various other anchor map elements proximate the user'spathway260.
• As used herein, ananchor map300 comprises digital data that maps mixed-reality elements (e.g.,130) to locations within the real world. Theanchor map300 depicted inFIG. 3 is merely exemplary and is simply being provided for the sake of discussion. One of skill in the art will understand that in practice an anchor map comprises a different form, function, and structure than thevisual anchor map300 that is currently depicted.
• In addition to the elements depicted within themap200 ofFIG. 2, theanchor map300 also depicts afirst anchor vertex320, asecond anchor vertex330, athird anchor vertex340, afourth anchor vertex350, and afifth anchor vertex360. Each anchor vertex comprises a first set of one or more key frames (e.g.,key frames322a,322b,322c), a mixed-reality element (e.g., mixed-reality element130), and at least one transform (e.g., transform324) connecting at least one key frame (e.g.,key frame322a) to the mixed-reality element (e.g., mixed-reality element130).
• As used herein, a transform comprises a mathematical relationship between a location and viewing direction of the key frame and the location and pose of the mixed-reality element. The mathematical relationship may be a SE3 transformation (i.e., 3D rotation and translation) transform or other transform capable of providing relative location and position of objects. The transform may be based on any location referencing scheme, including vector relationships, spatial coordinates, rotational and/or translational matrices, and so forth.
• The mixed-reality headset110 uses the transform and its associated key frame to determine the correct location and pose to render a mixed-reality element with respect to the location of the associated key frame. The rendering of the mixed-reality element may be performed with any augmented reality or virtual reality technology that is known, wherein the relative positioning and pose of the rendered element is based on the techniques described within this disclosure, based at least in part on the determined anchor vertexes and connecting transforms.
• Each anchor vertex (e.g., first anchor vertex320) is associated with at least one anchor edge (e.g., first anchor edge372) that connects the anchor vertex to another anchor vertex. As used herein, an anchor edge comprises a direct mathematical transform that maps the relative location of a first anchor vertex (e.g.,320) to the relative location of a second anchor vertex (e.g.,330). In at least one embodiment, an anchor edge also omitskey frames250 that are known to exist along the pathways between two respective anchor vertexes or other map locations near the anchor vertexes. In particular, the anchor edges represent transformations which may have been initially estimated by analysis of co-visible key frames, but are encoding the total transformation between anchors without requiring the transmission of individual key frames and pose-links (transforms) between the same. In this regard, the anchor edge comprises a composition or replacement of many intervening transforms. By omitting the key frames, it is possible to significantly reduce the data that needs to be stored/processed to identify a relative location within a mixed-reality environment. This is a significant change and improvement over known systems that map/reference all key frames traversed between different locations in a virtual environment.
• Additionally, in at least one embodiment, an anchor edge comprises a rotational and/or translational matrix that represents the physical rotation and translation between two different anchor vertexes. Further, an anchor edge can comprise an SE(3) transform between a source and a target anchor vertex, a three-dimensional vector, or any other mathematical construct that is capable of describing a pathway between two objects located within a physical space.
• Upon identifying the location of afirst anchor vertex320, the mixed-reality headset110 uses thefirst anchor edge372 to correctly determine the location of the second anchor vertex330 (and its associated mixed-reality element212).
• In at least one embodiment, as theuser100 walks along thepathway260, the mixed-reality headset110 identifies afirst anchor vertex320. The first anchor vertex includes a first set of one or more key frames322(a-c), a first mixed-reality element130, and at least one first transform connecting at least onekey frame322ato the first mixed-reality element130. Additionally, as the user continues along thepathway260, the mixed-reality headset110 identifies asecond anchor vertex330. Thesecond anchor vertex330 also includes a second set of one or more key frames332(a, b), a second mixed-reality element212, and at least onetransform334 connecting at least one key frame332ato the second mixed-reality elements212.
• Upon identifying thefirst anchor vertex320 and thesecond anchor vertex330, the mixed-reality headset110 creates afirst anchor edge372 between thefirst anchor vertex320 and thesecond anchor vertex330. In at least one embodiment, the mixed-reality headset110 is then capable of saving thefirst anchor vertex320, thesecond anchor vertex330, and thefirst anchor edge372 within an anchor graph data structure. In various embodiments, the anchor graph data structure may be stored within local memory in the mixed-reality headset110, within remote memory on a remote server, or within a combination thereof. As used herein, identifying an anchor vertex can comprise receiving an anchor vertex from a remote server and/or creating an anchor vertex.
• Continuing the example above, in at least one embodiment, theuser100 continues along thepathway260 to a third sculpture220 (shown inFIG. 2) that is associated with athird anchor vertex340. Thethird anchor vertex340 is associated with mixed-reality element222, key frames342 (a, b), and transform344. Upon identifying thethird anchor vertex340, the mixed-reality headset110 can create asecond anchor edge374 between thesecond anchor vertex330 and thethird anchor vertex340.
• As depicted inFIGS. 2 and 3, thesecond anchor vertex374 substantially aligns with apathway260 that theuser100 walked between thesecond anchor vertex330 and thethird anchor vertex340. Creating an anchor edge (e.g.,374) comprises calculating a direct mathematical transformation between two anchor vertexes. In at least one embodiment, the mixed-reality headset110 calculates the mathematical transformation by calculating rotational and/or translational matrices based upon data gathered by the mixed-reality headset110. For example, the mixed-reality headset110 may comprise sensors such as accelerometers, gyroscopes, depth cameras, vision sensors, magnetometers, GPS units, and other similar navigational and positional sensors—one or more of which can be utilized to generatekey frames250. As such, in at least one embodiment, the mixed-reality headset110 calculates an anchor edge by calculating rotational and/or translational matrices that link two anchor vertexes based upon the received sensor data, such as key frames250.
• After creating thesecond anchor edge374, the mixed-reality headset110 creates athird anchor edge376 between thefirst anchor vertex320 and thethird anchor vertex340. Of note, theuser100 never walked directly from thefirst anchor vertex320 to thethird anchor vertex340. In this case, instead of generating thethird anchor edge376 based upon sensor data that the mixed-reality headset110 gathered as theuser100 walked between thefirst anchor vertex320 and thethird anchor vertex340, the mixed-reality headset110 generates thethird anchor edge376 by combining thefirst anchor edge372 and thesecond anchor edge374. For example, combining thefirst anchor edge372 and thesecond anchor edge374 may comprise adding together the respective rotational and translational matrices that are associated with thefirst anchor edge372 and thesecond anchor edge374. Similarly, instead of directly combining thefirst anchor edge372 and thesecond anchor edge374, the mixed-reality headset110 can generate thethird anchor edge376 by generating rotational and translational matrices based upon the various sensor readings (e.g., key frames250) gathered along the user'spathway260 from thefirst anchor vertex320 to thesecond anchor vertex330 and then on to thethird anchor vertex340. As such, in at least one embodiment, the mixed-reality headset110 is capable of creating an anchor edge between any two anchor vertexes within an anchor graph by combining intermediate anchor vertexes or intermediate sensor readings together.
• In at least one embodiment, identifying an anchor vertex comprises creating an anchor vertex. For example,user100, using methods described above, may create mixed-reality element130 that is associated withsculpture120. After creating mixed-reality element130, the mixed-reality headset110 automatically calculates atransform324 that associates at least onekey frame322awith the mixed-reality element130. In at least one embodiment, atransform324 between akey frame322aand a mixed-reality element130 comprises translational and rotational matrices that orient the mixed-reality element130 with respect to the key frame322A. Additionally, in at least one embodiment, the transform is based upon an interpolation of a coordinate location associated with the mixed-reality element130, which coordinate location is derived from coordinates associated with each of the one or more key frames (e.g.,362(a-c). For example, inanchor vertex360, multiple transforms364(a-c) are depicted as relating the location of the mixed-reality element242 to multiple key frames362(a-c). As such, in at least one embodiment, a location of a mixed-reality element can be derived from coordinate systems associated with multiple key frames362(a-c).
• After calculating thetransform324, the mixed-reality headset110 creates thefirst anchor vertex320 by creating a data structure that contains a reference to the mixed-reality element130, thetransform324, the associatedkey frame322a, and various other key frames322(b, c) that are also proximate to the location of the mixed-reality element130.
• In at least one embodiment, the number of key frames322(a-c) associated with a particular anchor vertex is determined by a pre-defined memory size cap for anchor vertexes. For example, a mixed-reality headset110 may limit the size of any given anchor vertex to 10 MB, 15 MB, 20 MB or any other predetermined size. As such, the mixed-reality headset110 only includes a restricted number of key frames to a predetermined number of key frames (e.g., less than 2, less than 3, less than 5, or another quantity) and/or to a predetermined storage size that will maintain a total memory size of the anchor vertex (e.g., less than or equal to about 10 MB). This may include omitting key frames, as previously discussed. In various embodiments, capping the memory size of anchor vertexes allows a mixed-reality system to operate within optimized bandwidth and memory parameters. For example, whether creating and uploading an anchor vertex or receiving an anchor vertex from a remote server, the mixed-reality headset110 only needs to store and/or communicate data files that are limited to a predetermined size (e.g., 10 MB).
• In addition to the various methods described above for creating anchor vertexes, in at least one embodiment, the mixed-reality device110 receives anchor vertexes from a mixed-reality server system (shown as550 inFIG. 5). For example, in at least one embodiment, the mixed-reality headset110 communicates data to the mixed-reality server system550. Using the data received from the mixed-reality device110, the mixed-reality server system550 identifies that the mixed-reality device110 comprises a stored anchor graph. In at least one embodiment, the mixed-reality server system550 receives an index of anchor vertexes stored within the anchor graph on the mixed-reality device110. Using the index, the mixed-reality server system550 determines what additional anchor vertexes to send to the mixed-reality device110 and that are related to the stored anchor vertexes (e.g., related by location or context). Thesystem550 may also create transforms between the stored and new vertexes that are being sent to theuser device110. Alternatively, theuser device110 may create the transforms.
• In some embodiments, the mixed-reality server system550 triggers the transmission of vertexes to the user device in response to detecting the mixed-reality device110 is within a predetermined proximity to a first physical location. The detection of thedevice110 location can be based on sensors in thedevice100 and other systems proximate the first location. Alternatively, the location can be extrapolated from mapping traversal of the user'spath260. In yet other embodiments, the location is determined by location information entered by the user at thedevice100 that specifies their location.
• The first physical location may comprise the location of mixed-reality element130,sculpture120,first anchor vertex320, or any other physical location otherwise associated withsculpture120. In response to detecting that the mixed-reality device110 is within the predetermined proximity, the mixed-reality server system550 transmits thefirst anchor vertex320 and thefirst anchor edge372 to the mixed-reality device110.
• As described above, in at least one embodiment, the mixed-reality server system550 is also configured to receive one or more anchor vertexes from the mixed-reality device110, which may be subsequently stored at theserver system550 for use by one or more other user systems. For example, theuser100, in conjunction with the mixed-reality device110, may create a new anchor vertex and its associated at least one new key frame, new mixed-reality element, andnew transform344 connecting the at least one new key frame to the new mixed-reality elements. Even more specifically, the mixed-reality device may create thethird anchor vertex340 and its associated at least one key frame342(a, b), mixed-reality element222, and transform344 connecting the at least onekey frame342ato the mixed-reality elements222.
• After the mixed-reality device110 creates the new anchor vertexes (e.g., the third anchor vertex340), the mixed-reality server system550 receives the new anchor vertex from the mixed-reality device110. Additionally, the mixed-reality server system550 also receives a physical location that is associated with the real-world physical location of the new anchor vertex and a new anchor edge that comprises a transformation between thefirst anchor vertex320 and the new anchor vertex. For example, the mixed-reality server550 may receive thethird anchor vertex340, the physical location associated with thethird anchor vertex340, and thethird anchor edge376 that links thatthird anchor vertex340 to thefirst anchor edge320.
• When a second user (not shown) who has an associated second mixed-reality device (not shown) enters the sculpture park depicted inFIGS. 2 and 3, the mixed-reality server system550 identifies that the second mixed-reality device is storing a second anchor graph. Additionally, the mixed-reality server system550 detects that the second mixed-reality device is within a predetermined proximity to the physical location that is associated with the new anchor vertex (e.g., the third anchor vertex340). In response to detecting that the second mixed-reality device is within the predetermined proximity to the physical location, the mixed-reality server system550 transmits thethird anchor vertex340 and thethird anchor edge376 to the second mixed-reality device. Accordingly, the second mixed-reality device receives anchor vertexes and anchor edges that were identified by aseparate user100 and mixed-reality device110. As such, in at least one embodiment, the mixed-reality server system550 stores and transmits a plurality of anchor vertexes and associated anchor edges that are created by various different devices.
• Turning now toFIG. 4A,FIG. 4A illustrates an explodedview400 of theanchor map300 ofFIG. 3. In particular,FIG. 4A depicts the user'spathway260 in relation to thethird anchor vertex340, thefourth anchor vertex350, and thefifth anchor vertex360. Thethird anchor vertex340 comprises mixed-reality element222, thefourth anchor vertex350 comprises mixed-reality element232, and thefifth anchor vertex360 comprises mixed-reality element242. Additionally, thethird anchor vertex340 comprises key frames342(a, b) that are linked to the mixed-reality element222 throughtransform344. Similarly, thefourth anchor vertex350 comprises a single key frame352athat is linked to the mixed-reality element232 throughtransform354. Further, thefifth anchor vertex360 comprises key frames362(a-c) that are each separately linked to the mixed-reality element242 through transforms364(a-c).
• As described above, as theanchor vertexes340,350,360 are identified, respective anchor edges410,420,430 can be created by the mixed-reality headset110. For example,anchor edge410 andanchor edge420 both primarily follow the user'spathway260 between thethird anchor vertex340 and thefourth anchor vertex350 and between thefourth anchor edge350 and the fifth anchor edge306, respectively. As such, using methods described above, the mixed-reality headset110 can calculateanchor edge410 andanchor edge420 using sensor data gathered from the user's mixed-reality headset110 as the user traveled from thethird anchor vertex340 to thefourth anchor vertex350 and from thefourth anchor vertex350 to thefifth anchor vertex360.
• Additionally, using methods described above, the mixed-reality headset110 also creates acalculated anchor edge430 that extends from thefifth anchor vertex360 to thethird anchor vertex340. In particular, the mixed-reality headset110 creates the calculatedanchor edge430 by combininganchor edge410 andanchor edge420.
• In contrast, the mixed-reality headset110 can create an anchor edge between thefifth anchor vertex360 and thethird anchor vertex340 also using sensor data gathered from the user's mixed-reality headset110 as theuser100 traveled from thefifth anchor vertex360 to thethird anchor vertex340. For example,FIG. 4B illustrates another explodedview450 of theanchor map300 ofFIG. 3. In the depicted embodiment, the mixed-reality headset110 utilizes the user'spathway470 between thefifth anchor vertex360 and thethird anchor vertex340 to create a derivedanchor edge460. As such, in the depicted embodiment, the mixed-reality headset110 has created a continuous loop of anchor vertexes and anchor edges between thethird anchor vertex340, the fouranchor vertex350, and thefifth anchor vertex360.
• In at least one embodiment after creating the loop, the mixed-reality headset110 tunes one or more of the respective anchor edges410,420,460. For example, as described above, in at least one embodiment the anchor edges comprise rotational and translational matrices. In an ideal case, mathematical relationships exist between the respective anchor edges within the loop. For instance, a combination ofanchor edge410 andanchor edge420 should produce derivedanchor edge460. Using this mathematical relationship, the mixed-reality headset110 can tune the respective anchor edges410,420,460 based upon any discrepancies with the above stated mathematic relationship.
• In at least one embodiment, when tuning the respective anchor edges410,420,460, the mixed-reality headset110 weights the accuracy of one or more of the anchor edges higher. For instance,anchor edge410 andanchor edge420 may have previously been tuned based upon data received from multiple different mixed-reality devices worn by multiple different users who traveled from thethird anchor vertex340 to thefourth anchor vertex350 and from thefourth anchor vertex350 to thefifth anchor vertex360. In contrast, derivedanchor edge460 may only have data from a single mixed-reality device associated with a single user who walkedpathway470. In such a case, the mixed-reality device110 tunes anchoredge460 to conform with the mathematical relationship determined by a combination ofanchor edge410 andanchor edge240. Accordingly, in various embodiments, the mixed-reality device110 can tune various anchor edges over time such that greater accuracy is achieved as more anchor edges are added to an anchor graph and are validated by multiple users.
• FIG. 5 illustrates a schematic view of an embodiment of a mixed-reality system. In particular,FIG. 5 illustrates a schematic view of a user system500 in communication with a mixed-reality server system550 through anetwork connection540. The user system500 may correspond with various hardware and software components within a mixed-reality device, such as mixed-reality headset110. As depicted, the user system500 comprisesstorage510, processor(s)520,output interface524,sensory input interface526,graphics processing unit528,camera530,display532, andsensors534.Storage510 containsanchor graph storage512, localanchor vertex storage514, andkey frame storage516.
• Mixed-reality server system550 comprisesstorage560, processor(s)570, and network interfaces580.Storage560 containsremote anchor storage562,maps564, and in some additional or alternative embodiments remotekey frame storage566. The user system500 and/or the mixed-reality server system550 may be executed on various local, remote, and/or distributed systems542(a-c) that communicate throughnetwork connection540. One will understand that the various modules and components depicted inFIG. 5 are provided for the sake of example and that in various additional or alternative embodiments different combinations, descriptions, and depictions of modules and components can be equivalently used and described.
• In at least one embodiment of user system500, processor(s)520 execute various software applications within the mixed-reality device110. For example, the processor(s)520 direct thegraphics processing unit528 to render a hologram and provide the hologram through theoutput interface524 to adisplay532 within the mixed-reality device. The processor(s)520 also receive through thesensor input interface526 sensor data fromvarious sensors534, including acamera530.
• As the processor(s)520 identify anchor vertexes, the processor(s)520 store the anchor vertexes within localanchor vertex storage514. Similarly, as the mixed-reality device110 generateskey frames250, the processor(s)520 store the generatedkey frames250 withinkey frame storage516. As such, as the mixed-reality device110 gathers and generates information related to the mixed-reality worldspace, the processor(s)520 store the various SLAM coordinate data withinanchor graph storage512.
• Additionally, the processor(s)520 communicate information to the mixed-reality server system550. For example, in at least one embodiment, the processor(s)520 causes thenetwork interface536 to communicate one or more anchor vertexes to the mixed-reality server system550.
• The mixed-reality server system550 receives the communicated anchor vertexes throughnetwork interface580. Upon receiving an anchor vertex, the processor(s)570 store the anchor vertex withinremote anchor storage562. Additionally, in at least one embodiment, the processor(s)570 also store location data associated with a physical location of one or more anchor vertexes withinmaps564. For example, the processor(s)570 can store GPS coordinates associated with a particular anchor vertex within themaps564.
• In at least one additional or alternative embodiment, theprocessors570 also store one or more receivedkey frames250 within remotekey frame storage566. For example, the processor(s)570 may store somekey frames250 that are not associated with either an anchor vertex or an anchor edge. In contrast, in at least one embodiment, the mixed-reality server system550 does not store anykey frames250, except those that are contained with anchor vertexes. Accordingly, in at least one embodiment, the mixed-reality server system550 is only transmitting anchor vertexes to mixed-reality devices110, and is not transmitting stand-alonekey frames250 that are not otherwise contained within an anchor vertex.
• The form factor of the user system500 and the mixed-reality server system550 may vary to accommodate different needs and preferences. For instance, as described herein, the different systems may be standalone systems or distributed systems. The user system500 may be configured as a head mounted display, another wearable or portable device, a vehicle, robot and/or autonomous system.
• One will appreciate that embodiments disclosed herein can also be described in terms of flowcharts comprising one or more acts for accomplishing a particular result. For example,FIGS. 6-8 and the corresponding text describe acts in various systems for performing methods and/or stand-alone methods for segmenting scenes into semantic components. The acts ofFIGS. 6-8 are described below.
• For example,FIG. 6 illustrates aflowchart600 of acts associated with methods for generating an anchor graph for a mixed-reality environment. The illustrated acts comprise anact610 of identifying a first anchor vertex that includes a first set of one or more key frames, a first mixed-reality element, and at least one first transform connecting at least one key frame of the first set of one or more key frames to the first mixed-reality element. For example, as depicted and described inFIGS. 2 and 3, auser100 with a mixed-reality headset110 can identify thefirst anchor vertex320. Identifying thefirst anchor vertex320 includes either creating thefirst anchor vertex320 or receiving thefirst anchor vertex320 from mixed-reality server system550.
• The next illustrated act, act620, comprises identifying a second anchor vertex that includes a second set of one or more key frames, a second mixed-reality element, and at least one second transform connecting at least one key frame of the second set of the one or more key frames to the second mixed-reality element. For example, as depicted and described inFIGS. 2 and 3 and the accompanying description, auser100 with a mixed-reality headset110 can identify thesecond anchor vertex330. Identifying thesecond anchor vertex330 includes either creating thesecond anchor vertex330 or receiving thesecond anchor vertex320 from mixed-reality server system550.
• The next illustrated act, act630, includes creating a first anchor edge between the first anchor vertex and the second anchor vertex. For example, as depicted and described inFIG. 3 and the accompanying description, a mixed-reality headset110 calculates afirst anchor vertex372 in the form of a rotational matrix and/or a translational matrix that are derived from sensor data received as the user traveled from thefirst anchor vertex320 to thesecond anchor vertex330.
• The next illustrated act, act640, includes saving the first anchor vertex, the second anchor vertex, and the anchor edge within an anchor graph data structure. For example, as depicted and described inFIG. 5 and in the accompanying description, the user system500 (i.e., the mixed-reality headset110) stores anchor vertex data within ananchor graph storage512. Theanchor graph store512 comprises various anchor graph storage structures such as, but not limited to, flat file data, relational data, linked data, and various other known data structures.
• FIG. 7 illustrates aflowchart700 of acts associated with related methods for using anchor graphs within mixed-reality environments. The first illustrated act, act710, includes identifying a first device with a stored first anchor graph. For example, as depicted and described inFIGS. 2 and 5 and the accompanying description, the mixed-reality server system550 identifies a mixed-reality device110 that is associated with ananchor graph storage512, which comprises a stored first anchor graph.
• Next, act720 includes detecting that the first device is near a first physical location, which may include determining the device is within a predetermined proximity to a first physical location. For example, as depicted and described inFIG. 3 and the accompanying description, the mixed-reality server system550 identifies when auser100 comes within a predetermined proximity to a physical location associated with thefirst anchor vector320. The mixed-reality server system550 identifies the relative location of theuser100 based upon information received from the mixed-reality user device100 and information stored withinmaps564.
• Next, act730 includes transmitting a first anchor vertex and a first anchor edge to thefirst device730. This may be performed in response to detecting that the first device is within the predetermined proximity and may be performed by transmitting afirst anchor vertex320 and afirst anchor edge372 to the first device (e.g., mixed-reality headset110), wherein the first anchor vertex comprises at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element and wherein the first anchor edge comprises a transformation connecting the first anchor vertex to a second anchor vertex. For example, as depicted and described inFIGS. 3 and 5 and the accompanying description, the mixed-reality server system550 can transmit an anchor vertex (e.g., first anchor vertex320) and an anchor edge (e.g., first anchor edge372) to a mixed-reality device110.
• FIG. 8 illustrates aflowchart800 of acts associated with related methods for using an anchor graph within a mixed-reality environment. The first illustrated act, anact810, includes communicating location data associated with a first device to a server. This location data is associated with a location of a first device. For example, as depicted and described inFIGS. 3 and 5 and the accompanying description, user system500 (e.g., mixed-reality device110) communicates through anetwork interface536 to the mixed-reality server system550 location data gathered from one ormore sensors534. Alternatively, the location may be entered by the user into the user device.
• Next, the device receives (act820) a first anchor vertex and afirst anchor edge820 in response to communicating the location data to the server. The first anchor vertex is a vertex that is determined to be within a predetermined proximity to the first device and a first anchor edge or that is otherwise related to a stored vertex at the user device. The first anchor vertex comprises at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element. The first anchor edge comprises a transformation connecting the first anchor vertex to another anchor vertex (e.g., a stored vertex). For example, as depicted and described inFIGS. 3 and 5 and the accompanying description, when it is determined that auser100 is within a predetermined proximity to a physical location associated with an anchor vertex, the mixed-reality server system550 communicates the anchor vertex to the user device.
• Based on the foregoing description, it will be appreciated that the disclosed embodiments provide systems, methods, and apparatuses for optimizing the sharing of sparse SLAM coordinates within a mixed-reality worldspace. As described herein, the use, storage, and transmission of anchor vertexes provide significant technical improvements to at least bandwidth consumption and storage space requirements within mixed-reality applications. The improved bandwidth and storage space requirements provides for significantly improved user experience within mixed-reality worldspaces because the use of the location vertexes provides less of a burden on network bandwidth and storage capacities.
• Additionally, by using anchor vertexes that are linked together by anchor edges within an anchor graph map, it is possible to create a system of graphs within graphs. A “graph of graphs” representation enables sparse representation of SLAM data, which is locally dense in regions where shared mixed-reality content is displayed. Each anchor vertex represents a relatively small SLAM map component with tracking and relocalization data, and contains at least one associated world-fixed coordinate frame or “anchor” for displaying accurately registered mixed-reality content. Each anchor edge encodes relative pose between the coordinate system(s) represented by respective anchor vertexes. The physical size of the mapped environment becomes virtually unlimited via cloud-based storage of exported SLAM tracking data (i.e. “anchor vertexes”).
• Further, the methods may be practiced by a computer system including one or more processors and computer-readable media such as computer memory. In particular, the computer memory may store computer-executable instructions that when executed by one or more processors cause various functions to be performed, such as the acts recited in the embodiments.
• Embodiments of the present invention may comprise or utilize a special purpose or general-purpose computer including computer hardware, as discussed in greater detail below. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: physical computer-readable storage media and transmission computer-readable media.
• Physical computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage (such as CDs, DVDs, etc.), magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
• A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry out desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above are also included within the scope of computer-readable media.
• Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission computer-readable media to physical computer-readable storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer-readable physical storage media at a computer system. Thus, computer-readable physical storage media can be included in computer system components that also (or even primarily) utilize transmission media.
• Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
• Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, and the like. The invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
• Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
• The present invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims (20)

What is claimed is:

1. A computer system comprising:

one or more processors; and

one or more computer-readable hardware storage media having stored thereon computer-executable instructions, the computer-executable instructions being executable by the one or more processors to generate an anchor graph for a mixed-reality environment by causing the computer system to:

identify an anchor vertex that includes a mixed-reality element, the mixed-reality element being linked by the anchor vertex to a key frame;

identify a second anchor vertex that includes a second mixed-reality element, the second mixed-reality element being linked by the second anchor vertex to a second key frame; and

generate the anchor graph using information corresponding to the anchor vertex, the second anchor vertex, and a link between the anchor vertex and the second anchor vertex.

2. The computer system ofclaim 1, wherein the mixed-reality element is linked to the key frame by a transform, the transform defining a relationship between (1) a location and viewing direction associated with the key frame and (2) a location and pose associated with the mixed-reality element.

3. The computer system ofclaim 1, wherein the anchor graph is transmitted to a second computer system that is identified as being worn by a user who is physically proximate to a location associated with the key frame.

4. The computer system ofclaim 1, wherein the key frame comprises image data and geolocation data, the image data being captured by a camera of the computer system as a user who is wearing the computer system traverses a path, and wherein the geolocation data corresponds to a plurality of locations that are sampled from the user's traversed path, whereby the key frame includes location information corresponding to the user's traversed path.

5. The computer system ofclaim 1, wherein the mixed-reality element is a pre-made hologram.

6. The computer system ofclaim 5, wherein the pre-made hologram is viewable only when the computer system is oriented in a particular pose and situated at a particular location.

7. The computer system ofclaim 1, wherein a size of the anchor vertex is configurable.

8. The computer system ofclaim 1, wherein the anchor graph corresponds to a traversal path of a user who is wearing the computer system, and wherein execution of the computer-executable instructions further causes the computer system to:

receive a previously generated anchor graph, the previously generated anchor graph corresponding to a different path that was traversed previously by a different user who was wearing a different computer system, wherein a location of the path is determined to be sufficiently proximate to a location of the different path such that the key frame is included in both the anchor graph and the previously generated anchor graph; and

update the anchor graph to include additional anchor vertexes that are included in the previously generated anchor graph.

9. The computer system ofclaim 1, wherein the anchor graph includes a third anchor vertex, the third anchor vertex including a third mixed-reality element that is linked, in the third anchor vertex, to a third key frame, and wherein the anchor graph further includes an established link between the third anchor vertex and the second anchor vertex.

10. The computer system ofclaim 9, wherein, after identifying (1) the link between the anchor vertex and the second anchor vertex and (2) the established link between the third anchor vertex and the second anchor vertex, a third link is generated, the third link linking the anchor vertex to the third anchor vertex.

11. The computer system ofclaim 1, wherein the link between the anchor vertex and the second anchor vertex is structured so as to define a path of the user, the path originating at a location associated with the anchor vertex and ending at a location associated with the second anchor vertex.

12. The computer system ofclaim 1, wherein the anchor vertex further includes geographic location associated with the key frame, the geographic location being generated by one or more of (1) a global positioning sensor included as a part of the computer system, (2) an extrapolated mapping of a traversal path of a user wearing the computer system, or (3) location information entered by the user.

13. The computer system ofclaim 1, wherein the anchor vertex and the second anchor vertex are stored in an index of anchor vertexes.

14. The computer system ofclaim 1, wherein execution of the computer-executable instructions further causes the computer system to:

upon an occurrence of a triggering event, receive an additional anchor vertex from a remote server, the additional anchor vertex being added to the anchor graph.

15. The computer system ofclaim 14, wherein the triggering event occurs when a location of the computer system is determined to be sufficiently proximate to a location associated with the additional anchor vertex.

16. A method for generating an anchor graph for a mixed-reality environment, the method being implemented by one or more processors of a computer system, the method comprising:

identifying an anchor vertex that includes a mixed-reality element, the mixed-reality element being linked by the anchor vertex to a key frame;

identifying a second anchor vertex that includes a second mixed-reality element, the second mixed-reality element being linked by the second anchor vertex to a second key frame; and

generating the anchor graph using information corresponding to the anchor vertex, the second anchor vertex, and a link between the anchor vertex and the second anchor vertex.

17. The method ofclaim 16, wherein the link between the anchor vertex and the second anchor vertex is an anchor edge, the anchor edge comprising a rotational matrix that represents a physical rotation between the anchor vertex and the second anchor vertex.

18. The method ofclaim 17, wherein the anchor edge further includes a translational matrix that represents a translation between the anchor vertex and the second anchor vertex.

19. The method ofclaim 16, wherein the method further includes:

upon determining that a location of the computer system is determined to be sufficiently proximate to a location associated with a different anchor vertex, receive the different anchor vertex from a remote server, the different anchor vertex being added to the anchor graph.

20. One or more hardware storage devices having stored thereon computer-executable instructions, the computer-executable instructions being executable by one or more processors of a computer system to cause the computer system to:

identify an anchor vertex that includes a mixed-reality element, the mixed-reality element being linked by the anchor vertex to a key frame;

identify a second anchor vertex that includes a second mixed-reality element, the second mixed-reality element being linked by the second anchor vertex to a second key frame; and

generate the anchor graph using information corresponding to the anchor vertex, the second anchor vertex, and a link between the anchor vertex and the second anchor vertex.

Images