US20220068038A1 - Systems and Methods for Facilitating Display of Augmented Reality Content - Google Patents

Abstract

An exemplary method includes an augmented reality system acquiring an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area; identifying at least part of the performance area of the event within the image of the event; determining a three-dimensional (3D) pose of the performance area of the event within the image of the event; and providing, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.

Description

BACKGROUND INFORMATION
• Advances in computing and networking technology have made new forms of media content available. For example, people may utilize mobile computing devices (e.g., smartphones, tablet computers, etc.) to experience augmented reality content that is provided for display together with an image of a real-world event.
BRIEF DESCRIPTION OF THE DRAWINGS
• The accompanying drawings illustrate various embodiments and are a part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the disclosure. Throughout the drawings, identical or similar reference numbers designate identical or similar elements.
• FIG. 1 illustrates an exemplary augmented reality system according to principles described herein.
• FIG. 2 illustrates an exemplary implementation of the augmented reality system ofFIG. 1 according to principles described herein.
• FIG. 3 illustrates an exemplary image of an event according to principles described herein.
• FIGS. 4-5 illustrate exemplary image processing operations according to principles described herein.
• FIG. 6 illustrates an exemplary image of an event that includes augmented reality content according to principles described herein.
• FIG. 7 illustrates another exemplary image of an event according to principles described herein.
• FIGS. 8-9 illustrate exemplary image processing operations according to principles described herein.
• FIG. 10 illustrates another exemplary implementation of the augmented reality system ofFIG. 1 according to principles described herein.
• FIGS. 11-12 illustrate exemplary flow diagrams according to principles described herein.
• FIGS. 13-14 illustrate exemplary methods for facilitating display of augmented reality content according to principles described herein.
• FIG. 15 illustrates an exemplary computing device according to principles described herein.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
• Systems and methods for facilitating display of augmented reality content are described herein. For example, an exemplary augmented reality system may provide augmented reality content for display together with an image of a real-world event to enhance a user's experience of the real-world event.
• In certain examples, for instance, an exemplary augmented reality system may acquire an image of an event from a viewpoint of a camera of a computing device in proximity to the event. The event may include a performance area (e.g., for participants in the event). The augmented reality system may identify at least part of the performance area of the event within the image of the event and determine a three-dimensional (3D) pose of the performance area of the event within the image of the event. The augmented reality system may then provide, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.
• To illustrate an example, a user of a computing device (e.g., a smartphone, a tablet computer, etc.) may be viewing a live sporting event (e.g., a football game, a soccer match, etc.) from a distance in the stands of a stadium. While viewing the live sporting event, the user may hold up the computing device to use a camera of the computing device to capture a live image of a performance area (e.g., the football field, soccer field, etc.) of the live sporting event. While the user is viewing the live image on a display device of the computing device, the augmented reality system may analyze the live image in real-time in any suitable manner such as described herein to identify at least part of the performance area in the live image. The augmented reality system may then determine the 3D pose of the performance area. Based on the 3D pose of the performance area, the augmented reality system may provide, for concurrent display by the display device of the computing device, the live image of the live sporting event and augmented reality content (e.g., graphics, statistics, etc. associated with the live sporting event and/or one or more participants in the live sporting event) that is oriented according to the 3D pose of the performance area of the live sporting event within the live image of the live sporting event.
• Various advantages and benefits are associated with the systems and methods for facilitating display of augmented reality content described herein. For example, systems and methods such as those described herein may provide a generalized approach that may be extended and/or scaled to use any type of performance area of an event as a canvas on which augmented reality content may be provided for display. The systems and methods such as those described herein overcome long-distance limitations of known augmented reality frameworks to successfully determine a 3D pose of a performance area and track a portion (e.g., a center of the field of play of a sporting event) of the performance area in real-time. In addition, systems and methods such as those described herein facilitate providing augmented reality content even when only a portion of the performance area is viewable and/or when the performance area is viewed from a variety of different vantage points (e.g., various angles, distances, heights, etc. with respect to the performance area). These and other benefits that may be provided by systems and methods described herein will be evident from the disclosure that follows.
• FIG. 1 illustrates an exemplary augmented reality system100 (“system100”). As shown,system100 may include, without limitation, amemory102 and aprocessor104 selectively and communicatively coupled to one another.Memory102 andprocessor104 may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). In some examples,memory102 and/orprocessor104 may be implemented by any suitable computing device. In other examples,memory102 and/orprocessor104 may be distributed between multiple devices and/or multiple locations as may serve a particular implementation. An illustrative implementation ofsystem100 is described herein.
• Memory102 may maintain (e.g., store) executable data used byprocessor104 to perform any of the operations described herein. For example,memory102 maystore instructions106 that may be executed byprocessor104 to perform any of the operations described herein.Instructions106 may be implemented by any suitable application, software, code, and/or other executable data instance.
• Memory102 may also maintain any data received, generated, managed, used, and/or transmitted byprocessor104.Memory102 may store any other suitable data as may serve a particular implementation. For example,memory102 may store data representative of different types of events (e.g., sporting events, concerts, etc.), data indicative of dimensions (e.g., length, width, shape, etc.) and/or any other attribute of performance areas (e.g., playing fields, sport courts, performance stages, etc.) associated with the events, data (e.g., statistics, pictures, history, etc.) associated with participants (e.g., athletes, performers, etc.) of the events, data indicative of known outer contour shapes and/or 3D poses of performance areas of events, and/or any other suitable data. In addition,memory102 may also store any suitable data associated with graphical user interface content, and/or any other suitable data that may be used to provide augmented reality content for display to a user.
• Processor104 may be configured to perform (e.g., executeinstructions106 stored inmemory102 to perform) various processing operations associated with facilitating display of augmented reality content. For example,processor104 may determine a three-dimensional (3D) pose of a performance area of an event within an image of the event.Processor104 may provide, for concurrent display by a display device of a computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event. These and other operations that may be performed byprocessor104 are described herein.
• System100 (e.g., processor104) may be configured to provide any suitable service and/or feature that may be associated with augmented reality content as may serve a particular implementation. For example,system100 may be configured to provide participant information, provide user interfaces configured to facilitate user selection of augmented reality content, distribute (e.g., stream) augmented reality content for consumption by users of computing devices (e.g., smartphones, tablet computers, etc.), and/or provide any other suitable service and/or feature that may provide information regarding and/or access to augmented reality content.
• As used herein, the term “augmented reality content” may refer to any form of media that may be provided for display with an image of an event and/or distributed bysystem100 and experienced by a user ofsystem100. Augmented reality content may be provided for concurrent display with an image of an event in any suitable manner. For example, augmented reality content may be provided for display as an overlay over the image. Alternatively, augmented reality content may be incorporated into the image in any suitable manner. Examples of augmented reality content are described herein.
• FIG. 2 shows anexemplary implementation200 in whichsystem100 may be provided in certain examples. As shown inFIG. 2,implementation200 includes acomputing device202 that is configured to capture an image of anevent204 from a viewpoint in proximity toevent204. To that end,computing device202 includes acamera206.Computing device202 also includes adisplay device208 configured to display an image captured bycamera206.Computing device202 may correspond to any suitable mobile computing device that may be associated with a user. For example,computing device202 may correspond to a smartphone, a head mounted display device, a tablet computer, and/or any other suitable device.
• Event204 may correspond to any suitable type of event that includes a performance area. In certain examples, a performance area may be for participants in the event. For example, incertain implementations event204 may correspond to a sporting event such as a football game, a soccer match, a basketball game, a baseball game, a tennis match, etc. In certain alternative implementations,event204 may correspond to a concert, a political rally, a conference, etc.
• In certain examples,event204 may correspond to a live event in which a user ofcomputing device204 is physically located at event204 (e.g., in the stands) and is viewingevent204 in real-time. In certain alternative implementations,event204 may correspond to an event that a user ofcomputing device202 may experience while not being physically located atevent204. For example, the user may experienceevent204 by viewing a display screen of an additional computing device (e.g., a television, a projector screen, etc.) that is displaying an image ofevent204. In such examples, the additional computing device may display either a live image ofevent204 or a recorded image ofevent204.
• As used herein, a “performance area” refers to a region having known dimensions and that is associated with an event (e.g., that is configured to be used by one or more participants in the event). The known dimensions of a performance area may include any suitable spatial attribute or combination of spatial attributes of the performance area as may serve a particular implementation. In certain examples, the known dimensions may include a known size (e.g., length, width, etc.) of the performance area. Additionally or alternatively, the known dimensions may include a known shape (e.g., rectangular, square, circular, oval, an irregular shape, etc.) of the performance area. In certain examples, a performance area may have a generally planar surface. For example, a performance area associated with a basketball game may be considered as having a generally planar surface. In certain alternative examples, a performance area may include convex portions, concave portions, angled portions, and/or planar portions in any suitable combination. For example, a performance area associated with a baseball game may have a combination of convex surfaces (e.g., a pitching mound) and planar surfaces (e.g., infield areas, outfield grass areas, etc.).
• In certain examples, a performance area of an event may correspond to an area of play of a sporting event where athletes and/or other individuals participate in the sporting event. For example, an area of play may correspond to a football field, a soccer field, a basketball court, a baseball field, a tennis court, etc. In certain alternative examples, the performance area may correspond to a stage for participants (e.g., performers, speakers, etc.) of events such as concerts, conferences, political rallies, etc.
• As described herein,system100 may use known dimensions of a performance area to facilitate using the performance area as a canvas for augmented reality content. For example, while a user ofcomputing device202 is experiencingevent204,system100 may acquire an image ofevent204.System100 may acquire the image ofevent204 in any suitable manner. For example, a user ofcomputing device202 may directcamera206 towardsevent204 to capture an image ofevent204 andsystem100 may acquire data representative of the captured image fromcamera206. Accordingly,system100 may acquire the image ofevent204 from a viewpoint ofcamera206 ofcomputing device202.
• Camera206 may be configured to capture any suitable type of image ofevent204 as may serve a particular implementation. For example,camera206 may be configured to capture a still image, a live video image, and/or any other suitable type of image ofevent204. Additionally or alternatively,camera206 may be configured to capture a depth image ofevent204. In certain implementations, such a depth image may be used bysystem100 in any suitable manner to create a 3D point cloud of the performance area.
• Display device208 may be configured to display the image ofevent204 in real-time as the image is captured bycamera206.Display device208 may be implemented in any suitable manner. For example, in implementations wherecomputing device202 corresponds to a smartphone,display device208 may correspond to a display screen of the smartphone. In certain alternative implementations wherecomputing device202 corresponds to a head mounted display device,display device208 may correspond to a see through lens configured to direct an image to one or more eyes of a user while allowing the user to view the environment external to the head mounted display device.
• Aftersystem100 acquires the image ofevent204,system100 may identify at least part of a performance area within the image ofevent204.System100 may identify at least part of a performance area ofevent204 within the image ofevent204 in any suitable manner. For example,system100 may perform one or more image processing operations on the image captured bycamera206 to determine that a region included in the image ofevent204 represents a performance area. In certain examples, such image processing operations may includesystem100 determining that the region included in the image ofevent204 represents a performance area based on an average color of pixels included in the region. For example, ifevent204 corresponds to a soccer match,system100 may identify a region of the image as corresponding to the soccer field based on an average color of the pixels in the region being green.
• In certain examples, such image processing operations may includesystem100 determining an outer contour shape of the performance area. In certain examples,system100 may identify a performance area by comparing an outer contour shape of the performance area in the image to a plurality of known outer contour shapes of performance areas. In such examples,system100 may determine that the outer contour shape corresponds to a known outer contour shape from the plurality of known outer contour shapes when a degree of similarity between the outer contour shape and the known outer contour shape satisfies a predefined threshold. For example,system100 may determine that the outer contour shape corresponds to a known outer contour shape when the degree of similarity between the outer contour shape and the known outer contour shape is above 75%, 80%, 90%, or 95%.
• In certain examples,system100 may use object segmentation and/or deep learning technology to identify a portion of an image that corresponds toperformance area212. For example,system100 may use a supervised machine learning algorithm to generate a database of candidate outer contour shapes. In such examples, the training inputs to the supervised machine learning algorithm may include a plurality of images of an event that include labeled x and y coordinates of visible corners of performance areas. This facilitates marking corners of the performance area so as to define boundaries of the performance area as the ground truth. In addition, one or more images of the plurality of images may include labeled x and y coordinates of intersections of borders of a performance area and edges of the images.System100 may use the supervised machine learning algorithm in any suitable manner during a training phase to analyze the candidate outer contour shapes with the labeled corners. After the training phase,system100 may obtain, as an operational input, an image of an event in which a performance area is located.System100 may use the supervised machine learning algorithm in any suitable manner (e.g., by using a deep neural network) to analyze the image of the event including the performance area and determine an outer contour shape for the performance area.
• To illustrate an example, whenevent204 corresponds to a football game, a deep neural network model may be trained with videos of a football stadium (e.g., during real football games and/or at any other suitable time). In such an example, the training set for the deep neural network model may include videos from multiple different viewing locations inside the stadium (e.g., from various viewpoints at various locations in the stands of the stadium).System100 may use the videos in any suitable manner to generate the database of candidate outer contour shapes.
• As shown inFIG. 2, a processedimage210 is shown in which aperformance area212 is depicted in relation to other portions of processedimage210. In the example shown inFIG. 2, the outer contour shape ofperformance area212 may be considered as the boundary betweenperformance area212 and the black portions of processedimage210 that surroundperformance area212.Processed image210 may also be referred to as a mask ofperformance area212 in certain implementations.
• Performance area212 is depicted as a convex quadrilateral in the example shown inFIG. 2. However,performance area212 may have any other suitable shape in certain alternative implementations. For example,performance area212 may have the shape of a baseball diamond in implementations whereevent204 corresponds to a baseball game.
• In certain examples,system100 may determine, based on an outer contour shape of a performance area, that the performance area of an event is completely viewable within an image of the event. For example, as shown inFIG. 2,performance area212 may be considered as being completely viewable within the image ofevent204. In such examples,system100 may perform any suitable image processing operations to determine a location of corners ofperformance area212 that are viewable within processedimage210.
• In certain alternative examples, at least a portion of a performance area may not be currently viewable within an image of an event. For example, one or more corners or other portions of a performance area may not be currently viewable within the image of the event. In such examples,system100 may determine, based on the outer contour shape, a location where the performance area crosses a boundary of the image of the event.System100 may then infer, based on the location where the performance area crosses the boundary, a location of the one or more corners of the performance area that are not currently viewable within the image of the event. In certain examples,system100 may additionally or alternatively access information (e.g., from memory102) indicating the dimensions (e.g., shape, size, etc.) of the performance area to facilitate determining the location of the one or more corners of the performance area that are not currently viewable within the image of the event.
• In certain alternative examples,system100 may determine, based on an outer contour shape of a performance area, that each corner of the performance area is outside of an image of an event. For example, the performance area may have the shape of a convex quadrilateral similar to what is shown inFIG. 2. However, an additional image of the sporting event may only include a view of the middle of the field such that each of the four corners of the convex quadrilateral are not currently viewable in the additional image of the event. In such examples,system100 may access the known dimensions (e.g., from memory102) of the performance area to facilitatesystem100 identifying at least part of the performance area in the additional image of the event.
• To facilitate determining where to provide augmented reality content within an image of an event,system100 may determine a 3D pose of a performance area of an event within an image of the event. As used herein, the “3D pose” of a performance area refers to the combination of the orientation of the performance area and the position of the performance area in 3D space.System100 may determine the 3D pose of a performance area in any suitable manner. For example,system100 may determine in any suitable manner the x and y coordinates of each of the corners ofperformance area212 shown inFIG. 2. Based on the x and y coordinates of the corners,system100 may determine the 3D pose ofperformance area212 within the image ofevent204.
• In examples where an outer contour shape of a performance area is completely viewable within an image of an event,system100 may determine the 3D pose of the performance area based on the outer contour shape indicating that the performance area is completely viewable within the image of the event.
• In examples where a portion of a performance area is not currently viewable within an image of an event,system100 may determine the 3D pose of the performance area based on the outer contour shape and a determined location of the portion (e.g., a corner) that is not currently within the image of the event.
• In certain implementations,system100 may determine the 3D pose by determining 3D locations of corners of a performance area. For example, in implementations where the performance area has four corners, as shown in processedimage210,system100 may determine the x and y coordinates of each of the four corners. Aftersystem100 determines the x and y coordinates of each of the corners,system100 may create a camera matrix based on the following expression 1):
• 
  camera_matrix=np.array([[focal_length,0,center [0]],[0,focal_length,center [1]],[0,0,1]],dtype=“double”)  1)
• In expression 1), “center” is the center of the image.
• System100 may then create the four corresponding 3D points in a certain order (e.g., counterclockwise from the upper right corner). The following expression 2) is an exemplary expression for each of the four corners of the performance area.
• 
  model_points=np.array([(600.0,260.0,0.0),#upper right (−600.0,260.0,0.0),#upper left (−600.0,−260.0,0.0),#lower left (600.0,−260.0,0.0)#lower right])  2)
• Based on expressions 1) and 2)system100 may determine the 3D pose ofperformance area212 in relation to the viewpoint fromcamera206. Based on the determined 3D pose,system100 may provide, for concurrent display bydisplay device208, an image ofevent204 and augmented reality content that is oriented according to the 3D pose ofperformance area212. To illustrate,FIG. 2 shows animage214 ofevent204 that includes augmentedreality content216 that is oriented according to the 3D pose ofperformance area212.Augmented reality content216 may be oriented according to the 3D pose ofperformance area212 in any suitable manner. For example,system100 may determine, based on the 3D pose ofperformance area212, an anchor vector for augmented reality content. Such an anchor vector may be configured to intersect a surface ofperformance area212 and extend perpendicularly with respect to the surface. In such examples,augmented reality content216 may be provided for display along the anchor vector.
• In the example shown inFIG. 2,augmented reality content216 is provided for display at a center ofperformance area212. Accordingly, in certain examples,system100 may be configured to determine a center ofperformance area212. This may be accomplished in any suitable manner. For example,system100 may determine the center ofperformance area212 based on the 3D positions of each of the corners ofperformance area212. In certain examples,system100 may be configured to track the center of the performance area in real-time. In certain alternative implementations,augmented reality content216 may be provided for display withinimage214 at a portion ofperformance area212 other than the center ofperformance area212.
• In certain examples, an orientation of augmented reality content may be fixed in relation to a performance area of an event regardless of a change in viewing angle of a camera of a computing device. For example,augmented reality content216 may be fixed at the location shown inFIG. 2 relative toperformance area212. As such, even if the user panscamera206 to the left, to the right, up, or down, the position ofaugmented reality content216 stays at a same position relative toperformance area216.
• In certain examples,system100 may provide a notification to a user of a computing device to inform the user that augmented reality content associated with an event is available.System100 may provide such a notification in any suitable manner. For example,system100 may provide a text notification to a user by way ofdisplay screen208 ofcomputing device202. The notification may instruct the user to hold upcomputing device202 so as to capture an image ofperformance area212 withcamera206 ofcomputing device202.
• Additionally or alternatively, in certain examples,system100 may provide one or more graphical user interfaces for display by way ofcomputing device202 that facilitates accessing augmented reality content. Such graphical user interfaces may include a plurality of options that are selectable by the user ofcomputing device202 to experience different available augmented reality content. For example, ifevent204 corresponds to a football game, the plurality of options may include one or more options to view augmented reality content associated with a big play that occurred (e.g., a fumble, an interception, a touchdown, etc.), players by position (e.g., quarterback, wide receiver, running back, etc.), and/or any other suitable option.System100 may detect a user selection of one of the plurality of options and may provide augmented reality content associated with the selected option for display together with the image of the football field. For example,system100 may detect a user selection of an option to experience augmented reality content associated with the top running back in the football game. In such an example, the augmented reality content may include, for example, an image of the top running back, statistics (e.g., rushing yards) associated with the top running back, team graphics, and/or any other suitable information.
• FIG. 3 shows an implementation in whichcomputing device202 is implemented as asmartphone302 that that is being used to capture an image of a football game. As shown inFIG. 3,smartphone302 includes a display screen that is currently displaying alive image304 of the football game. In the example shown inFIG. 3,live image304 is captured by a camera ofsmartphone302 while a user ofsmartphone302, who may be in the stands of the stadium where the football game is being played, holds upsmartphone302 towards the football field.
• System100 may analyzelive image304 to determine which region inlive image304 corresponds to the football field and which region oflive image304 corresponds to something (e.g., stands, spectators, etc.) other than the football field. This may be accomplished in any suitable manner using any suitable image processing technique. For example,system100 may perform any suitable image processing operations to identify portions oflive image304 that correspond to the football field and other portions oflive image304 that correspond to regions oflive image304 other than the football field. In certain examples,system100 may generate one or more processed images in which portions (e.g., pixels) that represent the football field are white and portions (e.g., pixels) that do not represent the football field are black. To illustrate,FIG. 4 showsexemplary implementation400 depicting processed images402 (e.g., processed images402-1 and402-2) thatsystem100 may generate while identifying the football field. As shown inFIG. 4, processed image402-1 includes aperformance area404 that is white and that represents the football field. Processed image402-1 also includes black splotches withinperformance area404 that may be filled in, in any suitable manner, bysystem100 to generate processed image402-2. Processed images402 are provided for illustrative purposes only. It is understood that various other intermediate processing operations may be performed bysystem100 prior to achieving processed image402-2. In addition, it is understood that processed images402 may represent background processing performed bysystem100. As such,system100 may not provide processed images402 for display to a user on the display screen ofsmartphone302.
• System100 may use processed image402-2 in any suitable manner to identify whether at least part ofperformance area404 is withinlive image304. For example,system100 may determine an outer contour shape ofperformance area404. In certain implementations,system100 may compare the outer contour shape of processedarea404 in processed image402-2 to a plurality of known outer contour shapes. To illustrate,FIG. 5 shows animplementation500 that depicts a comparison between a processed image of an event and a plurality of known outer contour shapes502 (e.g., known outer contour shapes502-1 through502-N). Known outer contour shapes502 may represent views of the football stadium from different viewpoints of the football field. Known outer contour shapes502 may be considered as masks that are encoded bysystem100 as binary images where black (0) represents a region that does not include a performance area and white (255) represents a region that includes a performance area.System100 may compare the outer contour shape ofperformance area404 in processed image402-2 to known outer contour shapes502 in any suitable manner, such as described herein, to determine whether the football field is withinlive image304.
• Based on an outer contour shape ofperformance area404 shown in processed image402-2,system100 may determine x and y locations of each of the corners ofperformance area404.System100 may then use the x and y locations of each of the corners to determine 3D positions of the corners ofperformance area404 in relation tocomputing device302. Based on the 3D positions of the corners,system100 may then determine a center of performance area and an anchor vector that extends perpendicularly from a surface ofperformance area404.System100 may then provide augmented reality content for display together with the image of the football game oriented along the anchor vector in any suitable manner.
• To illustrate,FIG. 6 shows exemplary augmentedreality content602 that may be provided for display bysystem100 together with the image of the football game shown inFIG. 3.Augmented reality content602 may include any suitable information and/or graphic associated with the football game. For example,augmented reality content602 may include an image of a player in the football game, information indicating a score of the football game, team graphics, and/or any other suitable information.
• FIG. 7 shows another implementation in whichcomputing device202 is implemented as asmartphone302 that is being used to capture an image of a football game. As shown inFIG. 7,smartphone302 includes a display screen that is currently displaying alive image304 of the football game. In the example shown inFIG. 7,live image304 is captured by a camera ofsmartphone302 while the user, who may be in the stands of the stadium where the football game is being played, holds upsmartphone302 towards the football field.
• Unlike the example shown inFIG. 3, in the example shown inFIG. 7, the entire football field is not completely viewable withinlive image304 displayed bysmartphone302. In particular, the bottom right corner of the football field crosses a border oflive image304 and is not currently displayed. In such an example,system100 may be configured to determine an x and a y position of each of the visible corners of the football field as well as locations where the football field crosses the border oflive image304.System100 may then generate various processed images to infer the location of the missing corner based on the locations where the football field crosses the border and/or any other suitable information associated with the football field. To illustrate,FIG. 8 shows anexemplary implementation800 in which processed images802 (e.g., processed images802-1 through802-3) may be generated bysystem100. In the example shown inFIG. 8, corners804-1,804-2, and804-3 ofperformance area806 are within processed image802-1. However, the bottom right corner is not. To determine the location of the missing corner,system100 may determine intersection locations808-1 and808-2 where performance area crosses the border of image802-1.System100 may then infer the location of corner804-4 in any suitable manner. For example,system100 may extrapolate the location of corner804-4 based on locations808-1 and808-2 (e.g., by extending the line that connects corner804-1 and intersection location808-1 beyond intersection location808-1, extending the line that connects corner804-3 and intersection location808-2 beyond intersection location808-1, and finding and setting the location at which the extended lines intersect to be the missing corner). Additionally or alternatively,system100 may infer the location of corner804-4 based on the known dimensions of the football field. Based on corners804-1 through804-4,system100 may determine the 3D pose ofperformance area806 and a center ofperformance area806 in any suitable manner. For example,system100 may use, in any suitable manner, 3D locations of each corner of the football field to find a center of the football field and a surface normal vector extending from the center of the football field. As shown inFIG. 8, ananchor vector810 extends from the center ofperformance area806 and is oriented perpendicular to the surface ofperformance area806.Anchor vector810 may be used bysystem100 to orient augmented reality content with respect the surface ofperformance area806 such as described herein.
• In certain examples,system100 may determine that two or more 3D poses may be candidates for being determined as a 3D pose for a performance area. For example,system100 may determine that a first 3D pose and a second 3D pose included in a plurality of 3D poses are candidates for being determined as the 3D pose of the performance area of the event. In such examples,system100 may determine a first anchor vector for the first 3D pose and a second anchor vector for the second 3D pose. The first anchor vector may be perpendicular to a surface of the performance area represented in the first 3D pose and the second anchor vector may be perpendicular to a surface of the performance area represented in the second 3D pose. To illustrate,FIG. 9 shows anexemplary implementation900 in which there are two possible 3D poses for a performance area represented in an image of an event. As shown inFIG. 9, processed images902-1 have similar x and y coordinates for corners904 (e.g., corners904-1 through904-4) ofperformance area906. However, as indicated by anchor vectors908-1 and908-2 the orientation ofperformance area906 is different in each of processed images902-1 and902-2. In such an example,system100 may select the 3D pose represented in processed image902-2 as the 3D pose of the performance area of the event based on anchor vector908-2 being relatively more vertical than anchor vector908-1 when positioned within the image of the football game.
• System100 may be implemented in any suitable manner as may serve a particular application.FIG. 10 illustrates anexemplary implementation1000 ofsystem100. As shown inFIG. 10,implementation1000 may include an augmentedreality provider system1002 configured to communicate withcomputing device202 by way of anetwork1004. Inimplementation1000,memory102 andprocessor104 ofsystem100 may be implemented by augmentedreality provider system1002,computing device202, or distributed across augmentedreality provider system1002 andcomputing device202. Augmentedreality provider system1002 andcomputing device202 may communicate using any communication platforms and technologies suitable for transporting data and/or communication signals, including known communication technologies, devices, media, and protocols supportive of remote communications, examples of which include, but are not limited to, data transmission media, communications devices, and data transmission protocols.
• Network1004 may include, but is not limited to, one or more wireless networks (Wi-Fi networks), wireless communication networks, mobile telephone networks (e.g., cellular telephone networks), mobile phone data networks, broadband networks, narrowband networks, the Internet, local area networks, wide area networks, live television transmission networks, and any other networks capable of carrying media content, data, and/or communications signals betweencomputing device202 and augmentedreality provider system1002. Communications between augmentedreality provider system1002 andcomputing device202 may be transported using any one of the above-listed networks, or any combination or sub-combination of the above-listed networks. Alternatively, augmentedreality provider system1002 andcomputing device202 may communicate in another way such as by one or more direct connections between augmentedreality provider system1002 andcomputing device202. Augmentedreality provider system1002 may include one or more server-side computing devices.Computing device202 may include one or more mobile computing/processing devices such as described herein that are capable of accessing and presentingaugmented reality content216 for experiencing by a user1006 (e.g., an end user of an augmented reality service) using the device(s).
• Augmentedreality provider system1002 and/orcomputing device202 may be configured to perform one or more operations to provideaugmented reality content216 for concurrent display together withimage214 that includesperformance area212.Augmented reality content216 may include any suitable content associated with the event represented inimage214.
• FIG. 11 is a flow diagram1100 that depicts exemplary operations that may be performed bysystem100 to determine an anchor (e.g., an anchor vector) for augmented reality content with respect to a performance area in an image of an event. As shown inFIG. 11,system100 obtains a new image frame of an event inoperation1102. In certain examples,operation1102 may be preceded by a notification provided bysystem100 that instructs a user to point a camera of a computing device (e.g., computing device302) towards a performance area of an event. In certain examples,system100 may make a deep learning model inference to identify the performance area in the new image frame.System100 may then determine an average color (e.g., of pixels) of what the deep learning model sees as the performance area. For example, if the average color of a region in the new image frame is green,system100 may determine that the region corresponds to a field of play of a sporting event (e.g., a football game).
• Inoperation1104,system100 may determine whether the performance area identified, for example, by the deep learning model inference is a predetermined color (e.g., green). If the answer tooperation1104 is “NO,” the process may return to beforeoperation1102. However, if the answer tooperation1104 is “Yes,” the process may proceed tooperation1106 in whichsystem100 obtains an additional new image frame. In certain examples,system100 may provide an indication to the user thatsystem100 is scanning the additional new image frame for a performance area. For example,system100 may provide a scanning animation for display by way of a display device that informs the user thatsystem100 is currently scanning an image displayed by a computing device for a performance area.
• System100 may process the additional new image frame in any suitable manner such as described herein to identify the performance area. For example,system100 may make an additional deep learning model inference to identify the performance area. Based on the additional deep learning model inference,system100 may estimate a polygon that represents the performance area in operation1108.
• If there is a clear view of the performance area (e.g., all four corners of the performance area are viewable within the additional new frame),system100 may use the 4-point quadrilateral inoperation1110 to estimate the 3D pose inoperation1112. If one of the corners of the polygon is not represented in the additional new frame,system100 may infer the missing 4th corner and use the 4-point quadrilateral with the inferred missing 4th corner inoperation1114 to estimate the 3D pose of the performance area inoperation1112. Based on the 3D pose estimated inoperation1112,system100 may determine an anchor for augmented reality content with respect to the performance area within the additional new frame inoperation1116.
• Ifsystem100 determines that it is not possible to detect the polygon in operation1108,system100 may determine whether a defined threshold period (e.g., five seconds) has passed inoperation1118. If the answer tooperation1118 is “NO,”system100 may return the process flow to beforeoperation1106.
• On the other hand, if the answer inoperation1118 is “YES,”system100 may determine that it is not possible for the user to obtain a full or almost full view of the performance area. As such,system100 may perform a close distance pose estimation of the performance area inoperation1120.System100 may perform the close distance pose estimation in any suitable manner. For example,system100 may estimate the close distance pose of the performance area based on known dimensions and/or a 3D point cloud of the performance area at the event. For instance, if the event corresponds to a football game,system100 may use computer vision to detect yard lines of the football field.System100 may then estimate the 3D pose of the football field based on the yard lines. Additionally or alternatively,system100 may generate a 3D point cloud (e.g., an ARcore-based 3D point cloud) of the area of play that may be used to estimate the 3D pose.
• Inoperation1122,system100 may determine whetheroperation1120 was successful. If the answer inoperation1122 is “YES,”system100 may create an anchor inoperation1124 for augmented reality content. The anchor may be oriented with respect to the performance area based on the close distance pose estimation performed inoperation1120. On the other hand, if the answer inoperation1122 is “NO,”system100 may determine inoperation1126 whether a defined threshold period (e.g., ten seconds) has passed. If the answer inoperation1126 is “NO,”system100 may return the process flow to beforeoperation1120. On the other hand, if the answer tooperation1126 is “YES,”system100 may select a distance for a manual anchor inoperation1128. For example,system100 may select a distance of 30 meters in front of a user of a computing device as a distance to place an anchor for augmented reality content inoperation1128. Based on the selected distance,system100 may then provide an anchor for the augmented reality content inoperation1130.
• FIG. 12 is a flow diagram1200 that depicts exemplary operations that may be performed bysystem100 in determining a 3D pose of a performance area in an image of an event. As shown inFIG. 12,system100 may generate a processed image of a performance area inoperation1202. The processed image may be processed in any suitable manner such as described herein. For example, the processed image may correspond to processed image402-2 shown inFIG. 4.
• Inoperation1204,system100 may compute an outer contour shape of the performance area in the processed image. This may be accomplished in any suitable manner, such as described herein.
• Inoperation1206,system100 may detect, based on the computed outer contour shape, border crossings of the performance area where the performance area extends beyond a border of the processed image. A border crossing may be indicative of one or more corners of the performance area not currently being viewable in the processed image. As part ofoperation1206,system100 may determine the number and the locations of each of the border crossings.
• Inoperation1208,system100 may determine whether there are zero border crossings. If the answer inoperation1208 is “YES,”system100 may approximate the outer contour shape with a quadrilateral in operation1210.System100 may then estimate the 3D pose inoperation1212 using the quadrilateral determined inoperation1212.
• On the other hand, if the answer inoperation1208 is “NO,”system100 may determine whether there is a single border crossing inoperation1214. If the answer tooperation1214 is “YES,”system100 may approximate the outer contour shape of the performance area with a pentagon. Inoperation1218,system100 may compute the edges of the pentagon.
• Inoperation1220,system100 may determine whether one edge of the pentagon is on a border of the processed image. If the answer inoperation1220 is “NO,”system100 may determine that there are no edges on the border inoperation1222 and proceed to operation1210. On the other hand, if the answer inoperation1220 is “YES,”system100 may infer a missing corner inoperation1224, similar to what is shown, for example, inimplementation800 depicted inFIG. 8.
• Inoperation1226,system100 may determine whether the missing corner was successfully inferred. If the answer inoperation1226 is “YES,”system100 may replace the edge on the border with the missing corner in operation1228. An example of operation1228 is shown in processed image802-2 shown inFIG. 8. After operation1228,system100 may proceed to operation1210. On the other hand, if the answer inoperation1226 is “NO,”system100 may determine that there has been a failure to determine the 3D pose of the performance area inoperation1230.
• If the answer inoperation1214 is “NO,”system100 may determine inoperation1232 whether there is a double border crossing of the performance area with respect to edges of the processed image. If the answer inoperation1232 is “NO,”system100 may determine that there has been a failure to determine the 3D pose of the performance area inoperation1234. On the other hand, if the answer inoperation1232 is “YES,”system100 may proceed tooperation1236 and determine whether the border crossings occur on opposite sides of the processed image.
• If the answer inoperation1236 is “NO,”system100 may determine that there has been a failure to determine the 3D pose of the performance area inoperation1234. On the other hand, if the answer inoperation1236 is “YES,”system100 may approximate the outer contour shape representing the performance area with a quadrilateral inoperation1238.
• Inoperation1240,system100 may compute the locations of the edges of the quadrilateral in the processed image.
• Inoperation1242,system100 may determine whether two edges of the quadrilateral are on a border of the processed image. If the answer inoperation1242 is “YES,” system may compute the width of the performance area using the known dimensions of the performance area.System100 may then use the width to estimate the 3D pose of the performance area inoperation1246. On the other hand, if the answer inoperation1242 is “NO,”system100 may determine whether there are no edges on the border inoperation1248. If the answer inoperation1248 is “NO,”system100 may determine that there has been a failure to determine the 3D pose of the performance area inoperation1230.
• If the answer inoperation1248 is “YES,”system100 may estimate the 3D pose of the performance area inoperation1212 using the quadrilateral determined inoperation1238.
• FIG. 13 illustrates an exemplary method for facilitating display of augmented reality content. WhileFIG. 13 illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown inFIG. 13. One or more of the operations shown inFIG. 13 may be performed bysystem100, any components included therein, and/or any implementation thereof.
• Inoperation1302, an augmented reality system (e.g., augmented reality system100) may acquire an image of an event from a viewpoint of a camera of a computing device in proximity to the event. As described herein, the event may include a performance area that may, in certain examples, be for participants in the event.Operation1302 may be performed in any of the ways described herein.
• Inoperation1304, the augmented reality system may identify at least part of the performance area of the event within the image of the event.Operation1304 may be performed in any of the ways described herein.
• Inoperation1306, the augmented reality system may determine a 3D pose of the performance area of the event within the image of the event.Operation1306 may be performed in any of the ways described herein.
• Inoperation1308, the augmented reality system may provide, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.Operation1308 may be performed in any of the ways described herein.
• FIG. 14 illustrates another exemplary method for facilitating display of augmented reality content. WhileFIG. 14 illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown inFIG. 14. One or more of the operations shown inFIG. 14 may be performed bysystem100, any components included therein, and/or any implementation thereof.
• Inoperation1402, an augmented reality system (e.g., augmented reality system100) may acquire an image of an event from a viewpoint of a camera of a computing device in proximity to the event. As described herein, the event may include a performance area that may, in certain examples, be for participants in the event.Operation1402 may be performed in any of the ways described herein.
• Inoperation1404, the augmented reality system may determine a region included in the image of the event that represents the performance area of the event.Operation1404 may be performed in any of the ways described herein.
• Inoperation1406, the augmented reality system may determine, based on the region in the image of the event that represents a performance area, an outer contour shape of the performance area of the event within the image of the event.Operation1406 may be performed in any of the ways described herein.
• Inoperation1408, the augmented reality system may determine, based on the outer contour shape, a three-dimensional (3D) pose of the performance area of the event within the image of the event.Operation1408 may be performed in any of the ways described herein.
• Inoperation1410, the augmented reality system may provide, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented according to the 3D pose of the performance area of the event within the image of the event.Operation1410 may be performed in any of the ways described herein.
• In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.
• A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g. a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (RAM), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).
• FIG. 15 illustrates anexemplary computing device1500 that may be specifically configured to perform one or more of the processes described herein. As shown inFIG. 15,computing device1500 may include acommunication interface1502, aprocessor1504, astorage device1506, and an input/output (I/O)module1508 communicatively connected one to another via acommunication infrastructure1510. While anexemplary computing device1500 is shown inFIG. 15, the components illustrated inFIG. 15 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components ofcomputing device1500 shown inFIG. 15 will now be described in additional detail.
• Communication interface1502 may be configured to communicate with one or more computing devices. Examples ofcommunication interface1502 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.
• Processor1504 generally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein.Processor1504 may perform operations by executing computer-executable instructions1512 (e.g., an application, software, code, and/or other executable data instance) stored instorage device1506.
• Storage device1506 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example,storage device1506 may include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored instorage device1506. For example, data representative of computer-executable instructions1512 configured to directprocessor1504 to perform any of the operations described herein may be stored withinstorage device1506. In some examples, data may be arranged in one or more databases residing withinstorage device1506.
• I/O module1508 may include one or more I/O modules configured to receive user input and provide user output. One or more I/O modules may be used to receive input for a single virtual experience. I/O module1508 may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O module1508 may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.
• I/O module1508 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O module1508 is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.
• In some examples, any of the systems, computing devices, and/or other components described herein may be implemented bycomputing device1500. For example,memory102 may be implemented bystorage device1506, andprocessor104 may be implemented byprocessor1504.
• In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

Claims (20)

1. A method comprising:

acquiring, by an augmented reality system, an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area;

identifying, by the augmented reality system, at least part of the performance area of the event within the image of the event;

determining, by the augmented reality system, a three-dimensional (3D) pose of the performance area of the event within the image of the event;

determining, by the augmented reality system and based on the 3D pose of the performance area, an anchor vector that is perpendicular to the performance area; and

providing, by the augmented reality system for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented with respect to the anchor vector and according to the 3D pose of the performance area of the event within the image of the event.

2. The method ofclaim 1, wherein the identifying of the at least part of the performance area includes determining an outer contour shape of the performance area of the event within the image of the event.

3. The method ofclaim 2, wherein:

the identifying of the at least part of the performance area further includes determining, based on the outer contour shape, that the performance area of the event is completely viewable within the image of the event; and

the determining of the 3D pose of the performance area of the event is based on the outer contour shape indicating that the performance area is completely viewable within the image of the event.

4. The method ofclaim 2, wherein:

the identifying of the at least part of the performance area further includes:

determining, based on the outer contour shape, a location where the performance area crosses a border of the image of the event; and

determining a location of a corner of the performance area based on the location where the outer contour shape of the performance area crosses the border of the image; and

the determining of the 3D pose of the performance area of the event is based on the outer contour shape and the location of the corner of the performance area.

5. The method ofclaim 2, wherein:

the identifying of the at least part of the performance area further includes determining, based on the outer contour shape of the performance area, that each corner of the performance area of the event is outside of the image of the event; and

the augmented reality content is provided for display with the image of the event based on known dimensions of the performance area of the event.

6. The method ofclaim 2, wherein the identifying of the at least part of the performance area further includes:

comparing the outer contour shape of the performance area of the event to a plurality of known outer contour shapes of the performance area of the event; and

determining that the outer contour shape corresponds to a known outer contour shape from the plurality of known outer contour shapes when a degree of similarity between the outer contour shape and the known outer contour shape satisfies a predefined threshold.

7. The method ofclaim 1, wherein:

the determining of the 3D pose of the performance area of the event includes determining that a first 3D pose and a second 3D pose included in a plurality of 3D poses are candidates for being determined as the 3D pose of the performance area of the event;

the determining of the anchor vector includes determining a first anchor vector for the first 3D pose and a second anchor vector for the second 3D pose, the first anchor vector being perpendicular to a surface of the performance area represented in the first 3D pose and the second anchor vector being perpendicular to a surface of the performance area represented in the second 3D pose; and

the determining of the 3D pose of the performance area of the event further includes selecting the first 3D pose as the 3D pose of the performance area of the event based on the first anchor vector being relatively more vertical than the second anchor vector when positioned within the image of the event.

8. The method ofclaim 7, wherein the augmented reality content is oriented with respect to the first anchor vector.

9. The method ofclaim 8, wherein the first anchor vector is provided at a center of the performance area of the event within the image of the event.

10. The method ofclaim 1, wherein an orientation of the augmented reality content is fixed in relation to the performance area of the event regardless of a change in viewing angle of the camera of the computing device.

11. The method ofclaim 1, further comprising determining, by the augmented reality system, a center of the performance area within the image of the event,

wherein the providing of the image and the augmented reality content for concurrent display includes providing the augmented reality content at the center of the performance area within the image of the event.

12. The method ofclaim 1, wherein the augmented reality content is associated with one or more participants in the event.

13. A method comprising:

acquiring, by an augmented reality system, an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area;

determining, by the augmented reality system, a region included in the image of the event that represents the performance area of the event;

determining, by the augmented reality system based on the region in the image of the event that represents a performance area, an outer contour shape of the performance area of the event within the image of the event;

determining, by the augmented reality system based on the outer contour shape, a three-dimensional (3D) pose of the performance area of the event within the image of the event;

determining, based on the 3D pose of the performance area, an anchor vector that is perpendicular to the performance area; and

providing, by the augmented reality system for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented with respect to the anchor vector and according to the 3D pose of the performance area of the event within the image of the event.

14. The method ofclaim 13, wherein the determining of the outer contour shape includes:

determining that at least one corner of the performance area of the event is not within the image of the event; and

determining a location of the at least one corner of the performance area of the event.

15. (canceled)

16. The method ofclaim 13, wherein the determining of the region in the image of the event includes determining an average color of pixels included in the image of the event.

17. A system comprising:

a memory that stores instructions;

a processor communicatively coupled to the memory and configured to execute the instructions to:

acquire an image of an event from a viewpoint of a camera of a computing device in proximity to the event, the event including a performance area;

identify at least part of the performance area of the event within the image of the event;

determine a three-dimensional (3D) pose of the performance area of the event within the image of the event;

determine, based on the 3D pose of the performance area, an anchor vector that is perpendicular to the performance area; and

provide, for concurrent display by a display device of the computing device, the image of the event and augmented reality content that is oriented with respect to the anchor vector and according to the 3D pose of the performance area of the event within the image of the event.

18. The system ofclaim 17, wherein the identifying of the at least part of the performance area includes determining an outer contour shape of the performance area of the event within the image of the event.

19. The system ofclaim 17, wherein:

the processor is further configured to execute the instructions to determine, based on the 3D pose of the performance area of the event, a center of the performance area of the event; and

the augmented reality content is provided for display at the center of the performance area of the event within the image of the event.

20. The system ofclaim 17, wherein an orientation of the augmented reality content is fixed in relation to the performance area of the event regardless of a change in viewing angle of the camera of the computing device.

Images