A man using an augmented reality headset to view a life-size virtual model of a buildingAn augmented reality mapping application
Augmented reality (AR), also known asmixed reality (MR), is a technology that overlays real-time3D-rendered computer graphics onto a portion of the real world through a display, such as a handheld device orhead-mounted display. This experience is seamlessly interwoven with the physical world such that it is perceived as animmersive aspect of the real environment.[1] In this way, augmented reality alters one's ongoing perception of a real-world environment, compared tovirtual reality, which aims to completely replace the user's real-world environment with a simulated one.[2][3] Augmented reality is typicallyvisual, but can span multiple sensorymodalities, includingauditory,haptic, andsomatosensory.[4]
The primary value of augmented reality is the manner in which components of a digital world blend into a person's perception of the real world, through the integration of immersive sensations, which are perceived as real in the user's environment. The earliest functional AR systems that provided immersive mixed reality experiences for users were invented in the early 1990s, starting with theVirtual Fixtures system developed at the U.S. Air Force'sArmstrong Laboratory in 1992.[1][5][6] Commercial augmented reality experiences were first introduced in entertainment and gaming businesses.[7] Subsequently, augmented reality applications have spanned industries such as education, communications, medicine, and entertainment.
Augmented reality can be used to enhance natural environments or situations and offers perceptually enriched experiences. With the help of advanced AR technologies (e.g. addingcomputer vision, incorporating AR cameras into smartphone applications, andobject recognition) the information about the surrounding real world of the user becomesinteractive and digitally manipulated.[8] Information about the environment and its objects is overlaid on the real world. This information can be virtual or real, e.g. seeing other real sensed or measured information such as electromagnetic radio waves overlaid in exact alignment with where they actually are in space.[9][10][11] Augmented reality also has a lot of potential in the gathering and sharing of tacit knowledge. Immersive perceptual information is sometimes combined with supplemental information like scores over a live video feed of a sporting event. This combines the benefits of both augmented reality technology andheads up display technology (HUD).
Augmented realityframeworks includeARKit andARCore. Commercial augmented reality headsets include theMagic Leap 1 andHoloLens. A number of companies have promoted the concept ofsmartglasses that have augmented reality capability.
Augmented reality can be defined as a system that incorporates three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.[12] The overlaid sensory information can be constructive (i.e. additive to the natural environment), or destructive (i.e. masking of the natural environment).[1] As such, it is one of the key technologies in thereality-virtuality continuum.[13] Augmented reality refers to experiences that are artificial and that add to the already existing reality.[14][15][16]
Augmented reality (AR) is largely synonymous with mixed reality (MR). There is also overlap in terminology withextended reality andcomputer-mediated reality. However, In the 2020s, the differences between AR and MR began to be emphasized.[17][18]
Types of extended reality
Mixed reality (MR) is an advanced technology that extends beyond augmented reality (AR) by seamlessly integrating the physical and virtual worlds.[19] In MR, users are not only able to view digital content within their real environment but can also interact with it as if it were a tangible part of the physical world.[20] This is made possible through devices such asMeta Quest 3S andApple Vison Pro, which utilize multiple cameras and sensors to enable real-time interaction between virtual and physical elements.[21] Mixed reality that incorporateshaptics has sometimes been referred to as visuo-haptic mixed reality.[22][23]
Invirtual reality (VR), the users' perception is completely computer-generated, whereas with augmented reality (AR), it is partially generated and partially from the real world.[24][25] For example, in architecture, VR can be used to create a walk-through simulation of the inside of a new building; and AR can be used to show a building's structures and systems super-imposed on a real-life view. Another example is through the use of utility applications. Some AR applications, such asAugment, enable users to apply digital objects into real environments, allowing businesses to use augmented reality devices as a way to preview their products in the real world.[26] Similarly, it can also be used to demo what products may look like in an environment for customers, as demonstrated by companies such asMountain Equipment Co-op orLowe's who use augmented reality to allow customers to preview what their products might look like at home.[27]
Augmented reality (AR) differs fromvirtual reality (VR) in the sense that in AR, the surrounding environment is 'real' and AR is just adding virtual objects to the real environment. On the other hand, in VR, the surrounding environment is completely virtual and computer generated. A demonstration of how AR layers objects onto the real world can be seen with augmented reality games.WallaMe is an augmented reality game application that allows users to hide messages in real environments, utilizing geolocation technology in order to enable users to hide messages wherever they may wish in the world.[28]
In a physics context, the term "interreality system" refers to a virtual reality system coupled with its real-world counterpart.[29] A 2007 paper describes an interreality system comprising a real physical pendulum coupled to a pendulum that only exists in virtual reality.[30] This system has two stable states of motion: a "dual reality" state in which the motion of the two pendula are uncorrelated, and a "mixed reality" state in which the pendula exhibit stable phase-locked motion, which is highly correlated. The use of the terms "mixed reality" and "interreality" is clearly defined in the context of physics and may be slightly different in other fields, however, it is generally seen as, "bridging the physical and virtual world".[31]
1901: AuthorL. Frank Baum, in his science-fiction novelThe Master Key, first mentions the idea of an electronic display/spectacles that overlays data onto real life (in this case 'people'). It is named a 'character marker'.[32]
Heads-up displays (HUDs), a precursor technology to augmented reality, were first developed for pilots in the 1950s, projecting simple flight data into their line of sight, thereby enabling them to keep their "heads up" and not look down at the instruments. It is a transparent display.
1980: The research by Gavan Lintern of the University of Illinois is the first published work to show the value of aheads up display for teaching real-world flight skills.[34]
1980:Steve Mann creates the first wearable computer, a computer vision system with text and graphical overlays on a photographically mediated scene.[35]
1986: Within IBM, Ron Feigenblatt describes the most widely experienced form of AR today (viz. "magic window," e.g.smartphone-basedPokémon Go), use of a small, "smart" flat panel display positioned and oriented by hand.[36][37]
1987: Douglas George and Robert Morris create a working prototype of an astronomical telescope-based "heads-up display" system (a precursor concept to augmented reality) which superimposed in the telescope eyepiece, over the actual sky images, multi-intensity star, and celestial body images, and other relevant information.[38]
1990: The termaugmented reality is attributed to Thomas P. Caudell, a formerBoeing researcher.[39]
1992:Louis Rosenberg developed one of the first functioning AR systems, calledVirtual Fixtures, at the United States Air Force Research Laboratory—Armstrong, that demonstrated benefit to human perception.[40]
1992:Steven Feiner,Blair MacIntyre and Doree Seligmann present an early paper on an AR system prototype, KARMA, at the Graphics Interface conference.
1993: Mike Abernathy, et al., report the first use of augmented reality in identifying space debris usingRockwell WorldView by overlaying satellite geographic trajectories on live telescope video.[41]
1993: A widely cited version of the paper above is published inCommunications of the ACM – Special issue on computer augmented environments, edited by Pierre Wellner, Wendy Mackay, and Rich Gold.[42]
1993:Loral WDL, with sponsorship fromSTRICOM, performed the first demonstration combining live AR-equipped vehicles and manned simulators. Unpublished paper, J. Barrilleaux, "Experiences and Observations in Applying Augmented Reality to Live Training", 1999.[43]
1995: S. Ravela et al. at University of Massachusetts introduce a vision-based system using monocular cameras to track objects (engine blocks) across views for augmented reality.[44][45]
1996: General Electric develops system for projecting information from 3D CAD models onto real-world instances of those models.[46]
1999: Frank Delgado, Mike Abernathy et al. report successful flight test of LandForm software video map overlay from a helicopter at Army Yuma Proving Ground overlaying video with runways, taxiways, roads and road names.[48][49]
1999: TheUS Naval Research Laboratory engages on a decade-long research program called the Battlefield Augmented Reality System (BARS) to prototype some of the early wearable systems for dismounted soldier operating in urban environment for situation awareness and training.[50]
2000:Rockwell International Science Center demonstrates tetherless wearable augmented reality systems receiving analog video and 3D audio over radio-frequency wireless channels. The systems incorporate outdoor navigation capabilities, with digital horizon silhouettes from a terrain database overlain in real time on the live outdoor scene, allowing visualization of terrain made invisible by clouds and fog.[52][53]
2004: An outdoor helmet-mounted AR system was demonstrated byTrimble Navigation and the Human Interface Technology Laboratory (HIT lab).[54]
2006: Outland Research develops AR media player that overlays virtual content onto a users view of the real world synchronously with playing music, thereby providing an immersive AR entertainment experience.[55][56]
2008: Wikitude AR Travel Guide launches on 20 Oct 2008 with theG1 Android phone.[57]
2009: ARToolkit was ported toAdobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser.[58]
2012: Launch ofLyteshot, an interactive AR gaming platform that utilizes smart glasses for game data
2015:Microsoft announced theHoloLens augmented reality headset, which uses various sensors and a processing unit to display virtual imagery over the real world.[59]
2016:Niantic releasedPokémon Go foriOS andAndroid in July 2016. The game quickly became one of the most popular smartphone applications and in turn spikes the popularity of augmented reality games.[60]
2018:Magic Leap launched theMagic Leap One augmented reality headset.[61] Leap Motion announced the Project North Star augmented reality headset, and later released it under an open source license.[62][63][64][65]
2019:Microsoft announcedHoloLens 2 with significant improvements in terms of field of view and ergonomics.[66]
2022: Magic Leap launched the Magic Leap 2 headset.[67]
AR visuals appear on handheld devices (video passthrough) and head-mounted displays (optical see-through or video passthrough). Systems pair a display with sensors (e.g., cameras and IMUs) to register virtual content to the environment; research also explores near-eye optics, projection-based AR, and experimental concepts such as contact-lens or retinal-scanned displays.[70][71]
AR systems estimate device pose and scene geometry so virtual graphics stay aligned with the real world. Common approaches include visual–inertial odometry and SLAM for markerless tracking, and fiducial markers when known patterns are available; image registration and depth cues (e.g., occlusion, shadows) maintain realism.[71][81][82]
AR runtimes provide sensing, tracking, and rendering pipelines; mobile platforms expose SDKs with camera access and spatial tracking. Interchange/geospatial formats such as ARML standardize anchors and content.[83][84][73]
Input commonly combines head/gaze with touch, controllers, voice, or hand tracking; audio and haptics can reduce visual load. Human-factors studies report performance benefits but also workload and safety trade-offs depending on task and context.[85][82]
Key usability factors include stable registration, legible contrast under varied lighting, and low motion-to-photon latency. Visual design often uses depth cues (occlusion, shadows) to support spatial judgment; safety-critical uses emphasize glanceable prompts and minimal interaction.[86][87][71]
Augmented reality has been explored for many uses, including gaming, medicine, and entertainment. It has also been explored for education and business.[88] Some of the earliest cited examples include augmented reality used to support surgery by providing virtual overlays to guide medical practitioners, to AR content for astronomy and welding.[6][89] Example application areas described below include archaeology, architecture, commerce and education.
Overlays models and step-by-step guidance in real settings (e.g., anatomy, maintenance); systematic reviews report learning benefits alongside design and implementation caveats that vary by context and task.[90][91][92]
Guidance overlays and image fusion support planning and intraoperative visualization across several specialties; reviews note accuracy/registration constraints and workflow integration issues.[93][94][95]
Hands-free work instructions, inspection, and remote assistance tied to assets; evidence highlights productivity gains alongside limits around tracking robustness, ergonomics, and change management.[96][97][98]
Location-based and camera-based play place virtual objects in real spaces; recent surveys cover design patterns, effectiveness, and safety/attention trade-offs.[99][100][101]
Augmented reality navigation overlays route guidance or hazard cues onto the real scene, typically via smartphone "live view" or in-vehicle heads-up displays. Research finds AR can improve wayfinding and driver situation awareness, but human-factors trade-offs (distraction, cognitive load, occlusion) matter for safety-critical use.[102][103][104][105]
In the AEC sector, AR is used for design visualization, on-site verification against BIM models, clash detection, and guided assembly/inspection. Systematic reviews report benefits for communication and error reduction, while noting limits around tracking robustness and workflow integration.[106][107][108]
AR has been used to aidarchaeological research. By augmenting archaeological features onto the modern landscape, AR allows archaeologists to formulate possible site configurations from extant structures.[109] Computer generated models of ruins, buildings, landscapes or even ancient people have been recycled into early archaeological AR applications.[110][111][112] For example, implementing a system like VITA (Visual Interaction Tool for Archaeology) will allow users to imagine and investigate instant excavation results without leaving their home. Each user can collaborate by mutually "navigating, searching, and viewing data". Hrvoje Benko, a researcher in the computer science department atColumbia University, points out that these particular systems and others like them can provide "3D panoramic images and 3D models of the site itself at different excavation stages" all the while organizing much of the data in a collaborative way that is easy to use. Collaborative AR systems supplymultimodal interactions that combine the real world with virtual images of both environments.[113]
AR is used to integrate print and video marketing. Printed marketing material can be designed with certain "trigger" images that, when scanned by an AR-enabled device using image recognition, activate a video version of the promotional material. A major difference between augmented reality and straightforward image recognition is that one can overlay multiple media at the same time in the view screen, such as social media share buttons, the in-page video even audio and 3D objects. Traditional print-only publications are using augmented reality to connect different types of media.[114][115][116][117][118]
AR can enhance product previews such as allowing a customer to view what's inside a product's packaging without opening it.[119] AR can also be used as an aid in selecting products from a catalog or through a kiosk. Scanned images of products can activate views of additional content such as customization options and additional images of the product in its use.[120]
In 2018,Apple announcedUniversal Scene Description (USDZ) AR file support for iPhones and iPads with iOS 12. Apple has created an AR QuickLook Gallery that allows people to experience augmented reality through their own Apple device.[121]
In 2018,Shopify, the Canadian e-commerce company, announced AR Quick Look integration. Their merchants will be able to upload 3D models of their products and their users will be able to tap on the models inside the Safari browser on their iOS devices to view them in their real-world environments.[122]
In 2018,Twinkl released a free AR classroom application. Pupils can see howYork looked over 1,900 years ago.[123] Twinkl launched the first ever multi-player AR game,Little Red[124] and has over 100 free AR educational models.[125]
Augmented reality is becoming more frequently used for online advertising. Retailers offer the ability to upload a picture on their website and "try on" various clothes which are overlaid on the picture. Even further, companies such as Bodymetrics install dressing booths in department stores that offerfull-body scanning. These booths render a 3D model of the user, allowing the consumers to view different outfits on themselves without the need of physically changing clothes.[126] For example,JC Penney andBloomingdale's use "virtual dressing rooms" that allow customers to see themselves in clothes without trying them on.[127] Another store that uses AR to market clothing to its customers isNeiman Marcus.[128] Neiman Marcus offers consumers the ability to see their outfits in a 360-degree view with their "memory mirror".[128] Makeup stores likeL'Oreal,Sephora,Charlotte Tilbury, andRimmel also have apps that utilize AR.[129] These apps allow consumers to see how the makeup will look on them.[129] According to Greg Jones, director of AR and VR at Google, augmented reality is going to "reconnect physical and digital retail".[129]
AR technology is also used by furniture retailers such asIKEA,Houzz, andWayfair.[129][127] These retailers offer apps that allow consumers to view their products in their home prior to purchasing anything.[129][130]In 2017,Ikea announced the Ikea Place app. It contains a catalogue of over 2,000 products—nearly the company's full collection of sofas, armchairs, coffee tables, and storage units which one can place anywhere in a room with their phone.[131] The app made it possible to have 3D and true-to-scale models of furniture in the customer's living space. IKEA realized that their customers are not shopping in stores as often or making direct purchases anymore.[132][133] Shopify's acquisition of Primer, an ARapp aims to push small and medium-sized sellers towards interactive AR shopping with easy to use AR integration and user experience for both merchants and consumers. AR helps the retail industry reduce operating costs. Merchants upload product information to the AR system, and consumers can use mobile terminals to search and generate 3D maps.[134]
AR hardware and software for use in fitness includessmart glasses made for biking and running, with performance analytics and map navigation projected onto the user's field of vision,[135] and boxing, martial arts, and tennis, where users remain aware of their physical environment for safety.[136] Fitness-related games and software includePokémon Go andJurassic World Alive.[137]
Augmented reality systems are used inpublic safety situations, fromsuper storms to suspects at large.
As early as 2009, two articles fromEmergency Management discussed AR technology for emergency management. The first was "Augmented Reality—Emerging Technology for Emergency Management", by Gerald Baron.[138] According to Adam Crow,: "Technologies like augmented reality (ex: Google Glass) and the growing expectation of the public will continue to force professional emergency managers to radically shift when, where, and how technology is deployed before, during, and after disasters."[139]
Another early example was a search aircraft looking for a lost hiker in rugged mountain terrain. Augmented reality systems provided aerial camera operators with a geographic awareness of forest road names and locations blended with the camera video. The camera operator was better able to search for the hiker knowing the geographic context of the camera image. Once located, the operator could more efficiently direct rescuers to the hiker's location because the geographic position and reference landmarks were clearly labeled.[140]
AR can be used to facilitate social interaction, however, use of an AR headset can inhibit the quality of an interaction between two people if one isn't wearing one if the headset becomes a distraction.[141]
Augmented reality also gives users the ability to practice different forms of social interactions with other people in a safe, risk-free environment. Hannes Kauffman, Associate Professor for virtual reality at TUVienna, says: "In collaborative augmented reality multiple users may access a shared space populated by virtual objects, while remaining grounded in the real world. This technique is particularly powerful for educational purposes when users are collocated and can use natural means of communication (speech, gestures, etc.), but can also be mixed successfully with immersive VR or remote collaboration."[This quote needs a citation] Hannes citeseducation as a potential use of this technology.
One of the first applications of augmented reality was in healthcare, particularly to support the planning, practice, and training of surgical procedures. As far back as 1992, enhancing human performance during surgery was a formally stated objective when building the first augmented reality systems at U.S. Air Force laboratories.[1] AR provides surgeons with patient monitoring data in the style of a fighter pilot's heads-up display, and allows patient imaging records, including functional videos, to be accessed and overlaid. Examples include a virtualX-ray view based on priortomography or on real-time images fromultrasound andconfocal microscopy probes,[142] visualizing the position of a tumor in the video of anendoscope,[143] or radiation exposure risks from X-ray imaging devices.[144][145] AR can enhance viewing afetus inside a mother'swomb.[146] Siemens, Karl Storz and IRCAD have developed a system forlaparoscopic liver surgery that uses AR to view sub-surface tumors and vessels.[147]AR has been used for cockroach phobia treatment[148] and to reduce the fear of spiders.[149] Patients wearing augmented reality glasses can be reminded to take medications.[150] Augmented reality can be very helpful in the medical field.[151] It could be used to provide crucial information to a doctor or surgeon without having them take their eyes off the patient.
On 30 April 2015, Microsoft announced theMicrosoft HoloLens, their first attempt at augmented reality. The HoloLens is capable of displaying images for image-guided surgery.[152] As augmented reality advances, it finds increasing applications in healthcare. Augmented reality and similar computer based-utilities are being used to train medical professionals.[153][154] In healthcare, AR can be used to provide guidance during diagnostic and therapeutic interventions e.g. during surgery. Magee et al.,[155] for instance, describe the use of augmented reality for medical training in simulating ultrasound-guided needle placement. Recently, augmented reality began seeing adoption inneurosurgery, a field that requires heavy amounts of imaging before procedures.[156]
Smartglasses can be incorporated into the operating room to aide in surgical procedures; possibly displaying patient data conveniently while overlaying precise visual guides for the surgeon.[157][158] Augmented reality headsets like theMicrosoft HoloLens have been theorized to allow for efficient sharing of information between doctors, in addition to providing a platform for enhanced training.[159][158] This can, in some situations (i.e. patient infected with contagious disease), improve doctor safety and reducePPE use.[160] While mixed reality has lots of potential for enhancing healthcare, it does have some drawbacks too.[158] The technology may never fully integrate into scenarios when a patient is present, as there are ethical concerns surrounding the doctor not being able to see the patient.[158] Mixed reality is also useful for healthcare education. For example, according to a 2022 report from the World Economic Forum, 85% of first-year medical students at Case Western Reserve University reported that mixed reality for teaching anatomy was "equivalent" or "better" than the in-person class.[161]
Augmented reality applications, running on handheld devices utilized as virtual reality headsets, can also digitize human presence in space and provide a computer generated model of them, in a virtual space where they can interact and perform various actions. Such capabilities are demonstrated by Project Anywhere, developed by a postgraduate student at ETH Zurich, which was dubbed as an "out-of-body experience".[162][163][164]
Building on decades of perceptual-motor research in experimental psychology, researchers at the Aviation Research Laboratory of theUniversity of Illinois at Urbana–Champaign used augmented reality in the form of a flight path in the sky to teach flight students how to land an airplane using a flight simulator. An adaptive augmented schedule in which students were shown the augmentation only when they departed from the flight path proved to be a more effective training intervention than a constant schedule.[34][165] Flight students taught to land in the simulator with the adaptive augmentation learned to land a light aircraft more quickly than students with the same amount of landing training in the simulator but with constant augmentation or without any augmentation.[34]
Augmented reality system for soldier ARC4 (U.S. Army 2017)
The first fully immersive system was theVirtual Fixtures platform, which was developed in 1992 by Louis Rosenberg at theArmstrong Laboratories of theUnited States Air Force.[166] It enabled human users to controlrobots in real-world environments that included real physical objects and 3D virtual overlays ("fixtures") that were added enhance human performance of manipulation tasks. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators.[166][167][168]
An interesting early application of AR occurred whenRockwell International created video map overlays of satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System. In their 1993 paper "Debris Correlation Using the Rockwell WorldView System" the authors describe the use of map overlays applied to video from space surveillance telescopes. The map overlays indicated the trajectories of various objects in geographic coordinates. This allowed telescope operators to identify satellites, and also to identify and catalog potentially dangerous space debris.[41]
Starting in 2003 the US Army integrated the SmartCam3D augmented reality system into the Shadow Unmanned Aerial System to aid sensor operators using telescopic cameras to locate people or points of interest. The system combined fixed geographic information including street names, points of interest, airports, and railroads with live video from the camera system. The system offered a "picture in picture" mode that allows it to show a synthetic view of the area surrounding the camera's field of view. This helps solve a problem in which the field of view is so narrow that it excludes important context, as if "looking through a soda straw". The system displays real-time friend/foe/neutral location markers blended with live video, providing the operator with improved situational awareness.
Combat reality can be simulated and represented using complex, layered data and visual aides, most of which arehead-mounted displays (HMD), which encompass any display technology that can be worn on the user's head.[169] Military training solutions are often built oncommercial off-the-shelf (COTS) technologies, such asImprobable's synthetic environment platform, Virtual Battlespace 3 and VirTra, with the latter two platforms used by theUnited States Army. As of 2018[update], VirTra is being used by both civilian and military law enforcement to train personnel in a variety of scenarios, including active shooter, domestic violence, and military traffic stops.[170][171]
In 2017, the U.S. Army was developing the Synthetic Training Environment (STE), a collection of technologies for training purposes that was expected to include mixed reality. As of 2018[update], STE was still in development without a projected completion date. Some recorded goals of STE included enhancing realism and increasing simulation training capabilities and STE availability to other systems.[172]
It was claimed that mixed-reality environments like STE could reduce training costs,[173][174] such as reducing the amount ofammunition expended during training.[175] In 2018, it was reported that STE would include representation of any part of the world's terrain for training purposes.[176] STE would offer a variety of training opportunities for squad brigade and combat teams, includingStryker, armory, and infantry teams.[177]
Researchers at USAF Research Lab (Calhoun, Draper et al.) found an approximately two-fold increase in the speed at which UAV sensor operators found points of interest using this technology.[178] This ability to maintain geographic awareness quantitatively enhances mission efficiency. The system is in use on the US Army RQ-7 Shadow and the MQ-1C Gray Eagle Unmanned Aerial Systems.
In combat, AR can serve as a networked communication system that renders useful battlefield data onto a soldier's goggles in real time. From the soldier's viewpoint, people and various objects can be marked with special indicators to warn of potential dangers. Virtual maps and 360° view camera imaging can also be rendered to aid a soldier's navigation and battlefield perspective, and this can be transmitted to military leaders at a remote command center.[179] The combination of 360° view cameras visualization and AR can be used on board combat vehicles and tanks ascircular review system.
AR can be an effective tool for virtually mapping out the 3D topologies of munition storages in the terrain, with the choice of the munitions combination in stacks and distances between them with a visualization of risk areas.[180][unreliable source?] The scope of AR applications also includes visualization of data from embedded munitions monitoring sensors.[180]
LandForm video map overlay marking runways, road, and buildings during 1999 helicopter flight test
TheNASA X-38 was flown using a hybrid synthetic vision system that overlaid map data on video to provide enhanced navigation for the spacecraft during flight tests from 1998 to 2002. It used the LandForm software which was useful for times of limited visibility, including an instance when the video camera window frosted over leaving astronauts to rely on the map overlays.[48] The LandForm software was also test flown at the ArmyYuma Proving Ground in 1999. In the photo at right one can see the map markers indicating runways, air traffic control tower, taxiways, and hangars overlaid on the video.[49]
AR can augment the effectiveness of navigation devices. Information can be displayed on an automobile's windshield indicating destination directions and meter, weather, terrain, road conditions and traffic information as well as alerts to potential hazards in their path.[181][182][183] Since 2012, a Swiss-based companyWayRay has been developing holographic AR navigation systems that use holographic optical elements for projecting all route-related information including directions, important notifications, and points of interest right into the drivers' line of sight and far ahead of the vehicle.[184][185] Aboard maritime vessels, AR can allow bridge watch-standers to continuously monitor important information such as a ship's heading and speed while moving throughout the bridge or performing other tasks.[186]
In a research project, AR was used to facilitate collaboration among distributed team members via conferences with local and virtual participants. AR tasks included brainstorming and discussion meetings utilizing common visualization via touch screen tables, interactive digital whiteboards, shared design spaces and distributed control rooms.[187][188][189]
In industrial environments, augmented reality is proving to have a substantial impact with use cases emerging across all aspect of the product lifecycle, starting from product design and new product introduction (NPI) to manufacturing to service and maintenance, to material handling and distribution. For example, labels were displayed on parts of a system to clarify operating instructions for a mechanic performing maintenance on a system.[190][191] Assembly lines benefited from the usage of AR. In addition to Boeing, BMW and Volkswagen were known for incorporating this technology into assembly lines for monitoring process improvements.[192][193][194] Big machines are difficult to maintain because of their multiple layers or structures. AR permits people to look through the machine as if with an x-ray, pointing them to the problem right away.[195]
As AR technology has progressed, the impact of AR in enterprise has grown. In theHarvard Business Review, Magid Abraham and Marco Annunziata discussed how AR devices are now being used to "boost workers' productivity on an array of tasks the first time they're used, even without prior training".[196] They contend that "these technologies increase productivity by making workers more skilled and efficient, and thus have the potential to yield both more economic growth and better jobs".[196]
Machine maintenance can also be executed with the help of mixed reality. Larger companies with multiple manufacturing locations and a lot of machinery can use mixed reality to educate and instruct their employees. The machines need regular checkups and have to be adjusted every now and then. These adjustments are mostly done by humans, so employees need to be informed about needed adjustments. By using mixed reality, employees from multiple locations can wear headsets and receive live instructions about the changes. Instructors can operate the representation that every employee sees, and can glide through the production area, zooming in to technical details and explaining every change needed. Employees completing a five-minute training session with such a mixed-reality program have been shown to attain the same learning results as reading a 50-page training manual.[197] An extension to this environment is the incorporation of live data from operating machinery into the virtual collaborative space and then associated with three dimensional virtual models of the equipment. This enables training and execution of maintenance, operational and safety work processes, which would otherwise be difficult in a live setting, while making use of expertise, no matter their physical location.[198]
Product content management before the advent of augmented reality consisted largely of brochures and little customer-product engagement outside of this 2-dimensional realm.[199] With augmented reality technology improvements, new forms of interactive product content management has emerged. Most notably, 3-dimensional digital renderings of normally 2-dimensional products have increased reachability and effectiveness of consumer-product interaction.[200]
Augmented reality allows sellers to show the customers how a certain commodity will suit their demands. A seller may demonstrate how a certain product will fit into the homes of the buyer. The buyer with the assistance of the VR can virtually pick the item, spin around and place to their desired points. This improves the buyer's confidence of making a purchase and reduces the number of returns.[201] Architectural firms can allow customers to virtually visit their desired homes.
Augmented reality can be used to buildmockups that combine physical and digital elements. With the use ofsimultaneous localization and mapping (SLAM), mockups can interact with the physical world to gain control of more realistic sensory experiences[202] likeobject permanence, which would normally be infeasible or extremely difficult to track and analyze without the use of both digital and physical aides.[203]
Weather visualizations were the first application of augmented reality in television. It has now become common in weather casting to display full motion video of images captured in real-time from multiple cameras and other imaging devices. Coupled with 3D graphics symbols and mapped to a common virtual geospatial model, these animated visualizations constitute the first true application of AR to TV.
AR has become common in sports telecasting. Sports and entertainment venues are provided with see-through and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow "first down" line seen in television broadcasts ofAmerican football games showing the line the offensive team must cross to receive a first down. AR is also used in association with football and other sporting events to show commercial advertisements overlaid onto the view of the playing area. Sections ofrugby fields andcricket pitches also display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. Other examples include hockey puck tracking and annotations of racing car performance[204] and snooker ball trajectories.[205][206]
AR has been used to enhance concert and theater performances. For example, artists allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.[207][208][209]
Travelers may use AR to access real-time informational displays regarding a location, its features, and comments or content provided by previous visitors. Advanced AR applications include simulations of historical events, places, and objects rendered into the landscape.[210][211][212]
AR applications linked to geographic locations present location information by audio, announcing features of interest at a particular site as they become visible to the user.[213][214][215]
AR applications such asWord Lens can interpret the foreign text on signs and menus and, in a user's augmented view, re-display the text in the user's language. Spoken words of a foreign language can be translated and displayed in a user's view as printed subtitles.[216][217][218]
Recent advances in mixed-reality technologies have renewed interest in alternative modes of communication for human-robot interaction.[223] Human operators wearing augmented reality headsets such asHoloLens can interact with (control and monitor) e.g. robots and lifting machines[224] on site in a digital factory setup. This use case typically requires real-time data communication between a mixed reality interface with the machine / process / system, which could be enabled by incorporatingdigital twin technology.[224]
Snapchat users have access to augmented reality features. In September 2017, Snapchat announced a feature called "Sky Filters" that will be available on its app. This new feature makes use of augmented reality to alter the look of a picture taken of the sky, much like how users can apply the app's filters to other pictures. Users can choose from sky filters such as starry night, stormy clouds, beautiful sunsets, and rainbow.[225]
Google launched an augmented reality feature forGoogle Maps on Pixel phones that identifies users' location and places signs and arrows on the device screen to show a user navigation directions.[226]
MessageAR launched an augmented reality feature that allows users to send personalized video greetings which appear directly in the recipient's surroundings through their smartphone camera. The platform lets users record a short video, generate a shareable link, and have recipients view the message in augmented reality, creating the effect of the sender being present in the same space.[227]
In a paper titled"Death by Pokémon GO", researchers atPurdue University'sKrannert School of Management claim the game caused "a disproportionate increase in vehicular crashes and associated vehicular damage, personal injuries, and fatalities in the vicinity of locations, called PokéStops, where users can play the game while driving."[228] Using data from one municipality, the paper extrapolates what that might mean nationwide and concluded "the increase in crashes attributable to the introduction of Pokémon GO is 145,632 with an associated increase in the number of injuries of 29,370 and an associated increase in the number of fatalities of 256 over the period of 6 July 2016, through 30 November 2016." The authors extrapolated the cost of those crashes and fatalities at between $2bn and $7.3 billion for the same period. Furthermore, more than one in three surveyed advanced Internet users would like to edit out disturbing elements around them, such as garbage or graffiti.[229] They would like to even modify their surroundings by erasing street signs, billboard ads, and uninteresting shopping windows. Consumers want to use augmented reality glasses to change their surroundings into something that reflects their own personal opinions. Around two in five want to change the way their surroundings look and even how people appear to them.[citation needed]
Augmented reality devices that use cameras for 3D tracking or video passthrough depend on the ability of the device to record and analyze the environment in real time. Because of this, there are potential legal concerns over privacy.
In late 2024, Meta's collaboration with Ray-Ban on smart glasses faced heightened scrutiny due to significant privacy concerns. A notable incident involved two Harvard students who developed a program named I-XRAY, which utilized the glasses' camera in conjunction with facial recognition software to identify individuals in real-time.[230]
According to recent studies, users are especially concerned that augmented reality smart glasses might compromise the privacy of others, potentially causing peers to become uncomfortable or less open during interactions.[231]
While theFirst Amendment to the United States Constitution allows for such recording in the name of public interest, the constant recording of an AR device makes it difficult to do so without also recording outside of the public domain. Legal complications would be found in areas where a right to a certain amount of privacy is expected or where copyrighted media are displayed.
In terms of individual privacy, there exists the ease of access to information that one should not readily possess about a given person. This is accomplished through facial recognition technology. Assuming that AR automatically passes information about persons that the user sees, there could be anything seen from social media, criminal record, and marital status.[232]
Ronald Azuma is a scientist and author of works on AR.
Jeri Ellsworth headed a research effort forValve on augmented reality (AR), later taking that research to her own start-upCastAR. The company, founded in 2013, eventually shuttered. Later, she created another start-up based on the same technology called Tilt Five; another AR start-up formed by her with the purpose of creating a device for digitalboard games.[233]
Steve Mann formulated an earlier concept ofmediated reality in the 1970s and 1980s, using cameras, processors, and display systems to modify visual reality to help people see better (dynamic range management), building computerized welding helmets, as well as "augmediated reality" vision systems for use in everyday life. He is also an adviser toMeta.[234]
Dieter Schmalstieg and Daniel Wagner developed a marker tracking systems for mobile phones and PDAs in 2009.[235]
^Time-frequency perspectives, with applications, in Advances in Machine Vision, Strategies and Applications, World Scientific Series in Computer Science: Volume 32, C Archibald and Emil Petriu, Cover + pp 99–128, 1992.
^Mann, Steve; Feiner, Steve; Harner, Soren; Ali, Mir Adnan; Janzen, Ryan; Hansen, Jayse; Baldassi, Stefano (15 January 2015). "Wearable Computing, 3D Aug* Reality, Photographic/Videographic Gesture Sensing, and Veillance".Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14. ACM. pp. 497–500.doi:10.1145/2677199.2683590.ISBN978-1-4503-3305-4.S2CID12247969.
^Wu, Hsin-Kai; Lee, Silvia Wen-Yu; Chang, Hsin-Yi; Liang, Jyh-Chong (March 2013). "Current status, opportunities and challenges of augmented reality in education...".Computers & Education.62:41–49.doi:10.1016/j.compedu.2012.10.024.S2CID15218665.
^Rokhsaritalemi, S., Sadeghi-Niaraki, A., & Choi, S. M. (2020). A review on mixed reality: Current trends, challenges and prospects.Applied Sciences,10(2), 636.
^Buhalis, D., & Karatay, N. (2022). Mixed reality (MR) for generation Z in cultural heritage tourism towards metaverse. InInformation and communication technologies in tourism 2022: Proceedings of the ENTER 2022 eTourism conference, January 11–14, 2022 (pp. 16-27). Springer International Publishing.
^Cosco, F.; Garre, C.; Bruno, F.; Muzzupappa, M.; Otaduy, M. A. (January 2013). "Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration".IEEE Transactions on Visualization and Computer Graphics.19 (1):159–172.Bibcode:2013ITVCG..19..159C.doi:10.1109/TVCG.2012.107.PMID22508901.
^Carmigniani, Julie; Furht, Borko; Anisetti, Marco; Ceravolo, Paolo; Damiani, Ernesto; Ivkovic, Misa (1 January 2011). "Augmented reality technologies, systems and applications".Multimedia Tools and Applications.51 (1):341–377.doi:10.1007/s11042-010-0660-6.ISSN1573-7721.S2CID4325516.
^Ma, Minhua; C. Jain, Lakhmi; Anderson, Paul (2014).Virtual, Augmented Reality and Serious Games for Healthcare 1.Springer Publishing. p. 120.ISBN978-3-642-54816-1.
^Sutherland, Ivan E. (1968). "A head-mounted three dimensional display".Proceedings of the December 9-11, 1968, fall joint computer conference, part I on - AFIPS '68 (Fall, part I). p. 757.doi:10.1145/1476589.1476686.S2CID4561103.
^George, Douglas B.; Morris, L. Robert (1989). "A computer-driven astronomical telescope guidance and control system with superimposed star field and celestial coordinate graphics display".Journal of the Royal Astronomical Society of Canada.83: 32.Bibcode:1989JRASC..83...32G.
^Louis B. Rosenberg. "The Use ofVirtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments." Technical Report AL-TR-0089, USAF Armstrong Laboratory (AFRL), Wright-Patterson AFB OH, 1992.
^abAbernathy, M., Houchard, J., Puccetti, M., and Lambert, J,"Debris Correlation Using the Rockwell WorldView System", Proceedings of 1993 Space Surveillance Workshop 30 March to 1 April 1993, pages 189–195
^Ramesh Raskar, Greg Welch, Henry FuchsSpatially Augmented Reality, First International Workshop on Augmented Reality, Sept 1998.
^abDelgado, F., Abernathy, M., White J., and Lowrey, B.Real-Time 3-D Flight Guidance with Terrain for the X-38, SPIE Enhanced and Synthetic Vision 1999, Orlando Florida, April 1999, Proceedings of the SPIE Vol. 3691, pages 149–156
^abDelgado, F., Altman, S., Abernathy, M., White, J.Virtual Cockpit Window for the X-38, SPIE Enhanced and Synthetic Vision 2000, Orlando Florida, Proceedings of the SPIE Vol. 4023, pages 63–70
^AviationNow.com Staff, "X-38 Test Features Use of Hybrid Synthetic Vision" AviationNow.com, 11 December 2001
^Behringer, R.; Tam, C.; McGee, J.; Sundareswaran, S.; Vassiliou, M. (2000). "A wearable augmented reality testbed for navigation and control, built solely with commercial-off-the-shelf (COTS) hardware".Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). pp. 12–19.doi:10.1109/ISAR.2000.880918.ISBN0-7695-0846-4.S2CID18892611.
^Behringer, R.; Tam, C.; McGee, J.; Sundareswaran, S.; Vassiliou, M. (2000). "Two wearable testbeds for augmented reality: ItWARNS and WIMMIS".Digest of Papers. Fourth International Symposium on Wearable Computers. pp. 189–190.doi:10.1109/ISWC.2000.888495.ISBN0-7695-0795-6.S2CID13459308.
^7732694, "United States Patent: 7732694 - Portable music player with synchronized transmissive visual overlays", published 9 August 2006, issued 8 June 2010Archived 27 April 2019 at theWayback Machine
^Moro, Christian; Štromberga, Zane; Raikos, Athanasios; Stirling, Allan (2017). "The effectiveness of virtual and augmented reality in health sciences and medical anatomy".Anatomical Sciences Education.10 (6):549–559.doi:10.1002/ase.1696.ISSN1935-9780.PMID28419750.S2CID25961448.
^Chi, H.-L.; Kang, S.-C.; Wang, X. (2013). "Research trends and opportunities of AR applications in AEC".Automation in Construction.33:116–122.doi:10.1016/j.autcon.2012.12.017.
^Papagiannakis, George; Schertenleib, Sébastien; O'Kennedy, Brian; Arevalo-Poizat, Marlene; Magnenat-Thalmann, Nadia; Stoddart, Andrew; Thalmann, Daniel (1 February 2005). "Mixing virtual and real scenes in the site of ancient Pompeii".Computer Animation and Virtual Worlds.16 (1):11–24.CiteSeerX10.1.1.64.8781.doi:10.1002/cav.53.ISSN1546-427X.S2CID5341917.
^Benko, H.; Ishak, E.W.; Feiner, S. (2004). "Collaborative Mixed Reality Visualization of an Archaeological Excavation".Third IEEE and ACM International Symposium on Mixed and Augmented Reality. pp. 132–140.doi:10.1109/ISMAR.2004.23.ISBN0-7695-2191-6.S2CID10122485.
^Pavlik, John V., and Shawn McIntosh. "Augmented Reality."Converging Media: a New Introduction to Mass Communication, 5th ed.,Oxford University Press, 2017, pp. 184–185.
^Loy Rodas, Nicolas; Padoy, Nicolas (2014). "3D Global Estimation and Augmented Reality Visualization of Intra-operative X-ray Dose".Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014. Lecture Notes in Computer Science. Vol. 8673. pp. 415–422.doi:10.1007/978-3-319-10404-1_52.ISBN978-3-319-10403-4.PMID25333145.S2CID819543.
^Cui, Nan; Kharel, Pradosh; Gruev, Viktor (8 February 2017). "Augmented reality with Microsoft HoloLens holograms for near-infrared fluorescence based image guided surgery". In Pogue, Brian W; Gioux, Sylvain (eds.).Augmented reality with Microsoft HoloLens holograms for near-infrared fluorescence based image guided surgery. Molecular-Guided Surgery: Molecules, Devices, and Applications III. Vol. 10049. International Society for Optics and Photonics. pp. 100490I.doi:10.1117/12.2251625.S2CID125528534.
^Lintern, Gavan; Roscoe, Stanley N.; Sivier, Jonathan E. (June 1990). "Display Principles, Control Dynamics, and Environmental Factors in Pilot Training and Transfer".Human Factors.32 (3):299–317.doi:10.1177/001872089003200304.S2CID110528421.
^abRosenberg, Louis B. (1992). "The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments". Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.
^Rosenberg, Louis B. (21 December 1993). "Virtual fixtures as tools to enhance operator performance in telepresence environments".Telemanipulator Technology and Space Telerobotics.2057:10–21.Bibcode:1993SPIE.2057...10R.doi:10.1117/12.164901.
^Hughes, C.E.; Stapleton, C.B.; Hughes, D.E.; Smith, E.M. (November 2005). "Mixed reality in education, entertainment, and training".IEEE Computer Graphics and Applications.25 (6):24–30.Bibcode:2005ICGA...25f..24H.doi:10.1109/MCG.2005.139.PMID16315474.
^Pandher, Gurmeet Singh (2 March 2016). "Microsoft HoloLens Preorders: Price, Specs Of The Augmented Reality Headset". The Bitbag. Archived from the original on 4 March 2016. Retrieved 1 April 2016.
^Smith, Roger (February 2010). "The Long History of Gaming in Military Training".Simulation & Gaming.41 (1):6–19.doi:10.1177/1046878109334330.
^Shufelt, Jr., J.W. (2006) A Vision for Future Virtual Training. In Virtual Media for Military Applications (pp. KN2-1 – KN2-12). Meeting Proceedings RTO-MP-HFM-136, Keynote 2. Neuilly-sur-Seine, France: RTO. Available from:Mixed Reality (MR)Archived 13 June 2007 at theWayback Machine
^"STAND-TO!".www.army.mil. Retrieved22 August 2018.
^Calhoun, G. L., Draper, M. H., Abernathy, M. F., Delgado, F., and Patzek, M. "Synthetic Vision System for Improving Unmanned Aerial Vehicle Operator Situation Awareness," 2005 Proceedings of SPIE Enhanced and Synthetic Vision, Vol. 5802, pp. 219–230.
^Bruno, Fabio; Barbieri, Loris; Muzzupappa, Maurizio (September 2020). "A Mixed Reality system for the ergonomic assessment of industrial workstations".International Journal on Interactive Design and Manufacturing (IJIDeM).14 (3):805–812.doi:10.1007/s12008-020-00664-x.
^Pair, J.; Wilson, J.; Chastine, J.; Gandy, M. (2002). "The Duran Duran project: The augmented reality toolkit in live performance".The First IEEE International Workshop Agumented Reality Toolkit. p. 2.doi:10.1109/ART.2002.1107010.ISBN0-7803-7680-3.S2CID55820154.
^Farbiz, Farzam; Tang, Ka Yin; Wang, Kejian; Ahmad, Waqas; Manders, Corey; Jyh Herng, Chong; Kee Tan, Yeow (2007). "A multimodal augmented reality DJ music system".2007 6th International Conference on Information, Communications & Signal Processing. pp. 1–5.doi:10.1109/ICICS.2007.4449564.ISBN978-1-4244-0982-2.S2CID17807179.
^Chakraborti, Tathagata; Sreedharan, Sarath; Kulkarni, Anagha; Kambhampati, Subbarao (2018). "Projection-Aware Task Planning and Execution for Human-in-the-Loop Operation of Robots in a Mixed-Reality Workspace".2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4476–4482.doi:10.1109/IROS.2018.8593830.ISBN978-1-5386-8094-0.
