US20200400959A1 - Augmented reality monitoring of border control systems - Google Patents

Abstract

An augmented-reality device is provided. The augmented-reality device is configured to retrieve security information for a plurality of persons at respective security checkpoints. In addition, the augmented-reality device is configured to concurrently display the security information for each of the plurality of persons on an augmented-reality display. Security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the display configured to maintain visibility of the security checkpoint.

Description

PRIORITY APPLICATION
• This application claims the benefits of U.S. Provisional Patent Application No. 62/458,732, filed on Feb. 14, 2017, which is incorporated herein by reference in its entirety.
FIELD OF INVENTION
• The systems, methods, and apparatuses described herein relate to improved checkpoint security and control using computer-implemented techniques, and more particularly, toward the use of augmented-reality (AR) for identification of a person of interest at a security checkpoint such as a national border or other secure entry checkpoint.
BACKGROUND
• For identification and security purposes, persons travelling via air, land, and/or water typically must pass through one or more checkpoints, at which their travel documents, such as driver licenses or passports, are verified. Border officers face challenges posed by a wide range of demographics. Travelers include people from all walks of life and of all ages. Some demographics, such as children, may pose no security threat while other demographics may pose higher security risks. The conventional way of verifying the travelers is to check each traveler's identification information (e.g., name, date of birth, etc.) against a watch list maintained in a computer database. If a traveler's identification matches an entry in the watch list, the traveler may receive special treatment, such as being further questioned, detained, rejected, and deported, etc.
• While checking travelers' information against the watch list, border and/or security officers have limitations as to how much information they can mentally process while maintaining situational awareness. In the border security area, for example, the border officer is frequently placed in charge of monitoring travelers passing through an immigration checkpoint. The immigration checkpoint may contain multiple manned immigration booths as well as unmanned automated border control systems, such as kiosks or electronic gates. The border officer must maintain full situational awareness to detect potential threats while assisting travelers having difficulty with the systems and making clearance decisions on travelers when issues with their identity documentation are identified. Typically, the border officer monitors no more than seven systems at once to prevent the border officer from being overwhelmed with information that would cause them to miss potential security or traveler issues.
• As security or border officers grapple with security decisions, they are often limited by the lack of technology. Accordingly, systems, methods, and apparatuses that enable security or border officers to monitor multiple systems within immigration or other security checkpoints are needed. To this end, the embodiments of the present invention enable the use of augmented-reality to monitor checkpoints and to identify a person of interest.
SUMMARY OF THE INVENTION
• Embodiments of the present invention are directed toward electronic devices configured to provide augmented-reality monitoring for border control systems and/or other security checkpoints that substantially improve upon the related art.
• Features and advantages of the embodiments are set forth in the description which follows, or will be apparent from the description, or may be learned by practice of the invention.
• In one example, the augmented-reality device includes a processor, and a non-transitory memory storing one or more programs for execution by the processor, the one or more programs including instructions for retrieving security information for a plurality of persons at respective security checkpoints, and concurrently displaying the security information for each of the plurality of persons on an augmented-reality display, wherein the security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the displaying configured to maintain visibility of the security checkpoint.
• It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the invention to the described examples.
BRIEF DESCRIPTION OF THE DRAWINGS
• Further embodiments, details, advantages, and modifications will become apparent from the following detailed description of the preferred embodiments, which is to be taken in conjunction with the accompanying drawings.
• FIG. 1 is a block diagram of a computer server/system according to an example embodiment of the present invention.
• FIG. 2 illustrates an augmented-reality device that is communicatively coupled to a remote server according to an example embodiment of the present invention.
• FIG. 3 illustrates an officer using an augmented-reality device to monitor a plurality of security checkpoints according to an example embodiment of the present invention.
• FIG. 4 illustrates an officers view of security checkpoints through an augmented-reality device according to an example embodiment of the present invention.
• FIG. 5 illustrates a network that is configured to connect various components according to an example embodiment of the present invention.
• FIG. 6 illustrates a flow diagram of functionality for operating an augmented-reality device according to an example embodiment of the present invention.
DETAILED DESCRIPTION
• According to the embodiments of the present invention, an augmented-reality device is used to enable a security officer to concurrently (e.g., simultaneously) monitor multiple physically-manned or automated border control devices or entry control devices, such as immigration kiosks, electronic gates, or manned posts while maintaining full visibility of the area being monitored. Use of augmented-reality enables the real-time monitoring of detailed information that is visually superimposed on one or more persons (e.g., travelers) transiting the system, without losing situational awareness of the entire area being monitored.
• In some embodiments, a monitoring application is provided using augmented-reality glasses. Here, the status of all systems in the immigration or security checkpoint, the processing status of the persons (e.g., travelers) passing through the checkpoint, and/or other information regarding the environment around the persons, may be superimposed in an officer's field of view. Alternatively, other embodiments provide a monitoring application on a computer or mobile device (e.g., mobile phone or tablet) that graphically illustrates the area being monitored, the status of all systems in the immigration or security checkpoint, the processing status of the persons (e.g., travelers) passing through the checkpoint, and/or other information regarding the environment around the persons. Additional detailed information about each person may be retrieved if the border or security officer requests it.
• By contrast, using known systems, the officer needs to maintain situational awareness on a device screen, which requires taking their eyes off the travelers passing through the immigration point. This loss of visual monitoring opens a window of time for a security threat to arise and potentially evade the security measures in place at the immigration point. This is a challenge that is difficult to overcome without assigning multiple officers to each immigration checkpoint.
• Another challenge with known monitoring applications is that the screen layout of a terminal used by the border officer is often significantly different from the physical layout of the immigration point. For example, the gates may be laid out in a right-to-left scheme while the monitor of the terminal shows information in a top-to-bottom layout to maximize screen space. This may lead to confusion about which traveler and which system may be involved when a security or device issue arises. This is difficult to solve in a general way because each immigration checkpoint is laid out differently. It is also difficult to draw the border officer's attention to an area rather than a device. For example, in the event that a chemical signature indicating a hazardous material is detected in the vicinity of a device, the border officer has to further determine the physical location of the device.
• In view of the drawbacks of known security solutions, the embodiments enable the use of augmented-reality to monitor security checkpoints. By using augmented-reality devices, a single officer may maintain visibility across the security checkpoint while being able to get detailed information on each system and person simply by looking at them. Implementation of the embodiments also removes the mismatch between the screen layout of the monitoring application and the layout of the physical space, since the officer may look directly at the physical system about which they want to see information.
• Some augmented-reality applications use “fiduciary markers” in the environment to determine location. WO2015176163A1, entitled “Fiducial Marker Patterns, Their Automatic Detection in Images, and Applications Thereof”, which is hereby incorporated by reference in its entirety, uses fiduciary markers for determining configuration of a virtual space. In the various embodiments of the instant application, a single-camera application with multiple markers is used rather than a multi-camera application with a single marker. Processing functions may be split between the augmented-reality device and the other network connected computers. In addition, hyper-local location sensors and direct image-processing may be used to improve on the use of fiduciary markers alone.
• Accordingly, the embodiments of the present invention provide an augmented-reality experience for border or security officers charged with monitoring a security checkpoint. Although a border is often used as example security checkpoint, the embodiments of the invention are not so limited and may be readily applied to any security checkpoints (e.g., borders, concert entry, building entry, etc.). As discussed below, the various embodiments may overlay status information about each system and person in the border officers visual field through an augmented-reality device, and may provide detailed information about systems or persons that the border officer is directly observing.
• Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Wherever possible, like reference numbers will be used for like elements.
• User interfaces and associated methods for using a portable electronic device are described. In some embodiments, the device is a mobile phone, tablet, or wearable computer. The user interface may include a touch screen and/or other input/output devices. In the discussion that follows, a portable communications device is used as an example embodiment. It should be understood, however, that the user interfaces and associated methods may be applied to other devices, especially wearable computers and/or augmented-reality glasses, which may include one or more other physical user-interface devices, such as voice command and various sensors.
• The portable communication device may support a variety of applications. The various applications that may be executed on the device may use at least one common physical user-interface device. A common physical architecture of the portable communication device may support a variety of applications. In the discussion that follows, a border security application is used as an example embodiment, but it should be understood that the user interfaces and associated methods may be applied to other applications.
• FIG. 1 is a block diagram of acomputer server system100 in accordance with an example embodiment of the present invention.
• As shown inFIG. 1,system100 may include abus112 and/or other communication mechanism(s) configured to communicate information between the various components ofsystem100, such as aprocessor122 and amemory114. In addition, acommunication device120 may enable connectivity betweenprocessor122 and other devices by encoding data to be sent fromprocessor122 to another device over a network (as shown inFIG. 5) and decoding data received from another system over the network forprocessor122.
• For example,communication device120 may include a network interface card that is configured to provide wireless network communications. A variety of wireless communication techniques may be used including infrared, radio, Bluetooth, Wi-Fi, and/or cellular communications. Alternatively,communication device120 may be configured to provide wired network connection(s), such as an Ethernet connection.
• Processor122 may comprise one or more general or specific purpose processors to perform computation and control functions ofsystem100.Processor122 may include a single integrated circuit, such as a micro-processing device, or may include multiple integrated circuit devices and/or circuit boards working in cooperation to accomplish the functions ofprocessor122.
• System100 may includememory114 for storing information and instructions for execution byprocessor122.Memory114 may contain various components for retrieving, presenting, modifying, and storing data. For example,memory114 may store software modules that provide functionality when executed byprocessor122. The software modules may include anoperating system115 that provides operating system functionality forsystem100. The software modules may further include augmented-reality modules116 configured to concurrently (e.g., simultaneously) monitor multiple physical manned or automated border control devices or entry control devices, as well as otherfunctional modules118.
• Augmented-reality modules116 are configured to cause a person (e.g., traveler, concert-goer, employee, etc.) and device information to be visually superimposed, using a wearable display, on one or more persons transiting through a security checkpoint. In some instances, augmented-reality modules116 may illustrate the movement of travelers and respective traveler information within a security checkpoint. Augmented-reality modules116 may include one or more application program interfaces (“API”) that enable the security or border officers to monitor persons or travelers and their respective identifying information.
• Memory114 may include a variety of computer-readable media that may be accessed byprocessor122. For example,memory114 may include any combination of random access memory (“RAM”), dynamic RAM (“DRAM”), static RAM (“SRAM”), read only memory (“ROM”), flash memory, cache memory, and/or any other type of non-transitory or transitory computer-readable medium.
• Processor122 is further coupled viabus112 to adisplay124, such as a wearable display or augmented-reality glasses. Akeyboard126 and acursor control device128, such as a computer mouse, are further coupled tocommunication device120 to enable a user to interface withsystem100.
• System100 may be part of a larger system. Therefore,system100 may include one or more additional functional modules, such asfunctional module118 to include additional functionality, such as other applications. Otherfunctional modules118 may include various modules for identifying a person of interest as described in U.S. Patent Application Publication No. 2014/0279640A1, which is incorporated by reference in its entirety.
• Adatabase117 is coupled tobus112 to provide centralized storage for augmented-reality modules116 andfunctional modules118 and to store a person's or traveler's identifying and/or threat data.Database117 may store data in an integrated collection of logically-related records or files.Database117 may be an operational database, an analytical database, a data warehouse, a distributed database, an end-user database, an external database, a navigational database, an in-memory database, a document-oriented database, a real-time database, a relational database, an object-oriented database, or any other database known in the art.
• Although illustrated as a single system, the functionality ofsystem100 may be implemented as a distributed system. Further, the functionality disclosed herein may be implemented on separate servers or devices that may be coupled together over a network. Further, one or more components ofsystem100 may not be included. For example,system100 may be a smartphone or tablet device that includes a processor, memory and a display, but may not include one or more of the other components shown inFIG. 1.
• FIG. 2 illustrates an augmented-reality device200 that is communicatively coupled to aserver210, which can be located remotely, according to an example embodiment of the present invention. As shown inFIG. 2, augmented-reality device200 may includeprocessor202, global positioning system (GPS)receiver203, hyper-local positioning system204,accelerometer205,gyroscope206,network device207,camera208,image processor209, anddisplay211. As further shown inFIG. 2,server210 may includeserver processor212,network device217, andimage processor219. In addition, either of augmented-reality device200 orserver210 may include the additional elements, such as the various elements and software modules depicted and described in connection withFIG. 1. Augmented-reality device200 may be communicatively coupled toserver210 through a network, for example.
• Returning toFIG. 2, the security or border officer may wear an augmented-reality device200, which can be head-mounted and does not impede their normal field of vision, such as augmented-reality glasses. Alternatively, the security or border officers also may use a handheld device capable of augmented-reality, such as a tablet or smartphone. In either embodiment, the use of augmented-reality enables the real-time monitoring of detailed information that is visually superimposed on one or more persons (e.g., travelers, concert-goers, employees, etc.) transiting the system, without losing situational awareness of the entire area being monitored.
• Augmented-reality device200 may include on-board processing capabilities, such asprocessor202 andimage processor209.Processor202 and/orimage processor209 are configured to ensure that detailed information is visually superimposed on one or more persons transiting the security checkpoint. Cooperation between the on-board processing capabilities andGPS receiver203 and/or hyper-local positioning system204 is configured to ensure that the displayed information matches the security or border officer's field of view. In this manner, situational awareness is not compromised when providing the displayed information to the officer. Here,accelerometer205 and/orgyroscope206 are further configured to gather information to determine the orientation and movement of augmented-reality device200 relative to the security checkpoint.
• Camera208 may be configured to periodically or continuously (and in real-time) gather image data from the security checkpoint. Use ofcamera208 is optional in connection with embodiments directed to augmented-reality glasses. However,camera208 is typically used in connection with embodiments directed to a mobile phone or tablet device. In some instances,camera208 may be configured to operate across spectra, to include non-visible light such as infrared. Alternatively, or additionally,camera208 may be stereoscopic.
• Camera208 of augmented-reality device200 may transmit the image data to imageprocessor209 of augmented-reality device200 orimage processor219 ofserver210. Image processing functions may be executed atimage processor209 orimage processor219, or a combination ofimage processor209 andimage processor219. Augmented-reality device200 also may be configured to transmit its orientation and movement data, global position data, and/or hyper-local position information toprocessor202 orserver processor212.
• The various systems deployed at the security checkpoint may be integrated with fiduciary marks, which are markers that may be recognized by processing algorithms to provide data about which system or systems are being looked at by the security or border officer, the distance to that system, the relative orientation of the security or border officer to the system, and the relative orientation of that system to other systems at the security checkpoint. The systems may be recognized by their physical geometry by the processing algorithms when fiduciary marks are not present or are obscured. Alternatively, or additionally, the systems may be recognized by hyper-local positioning information provided by hyper-local positioning system204.
• Image processor209 and/orimage processor219 use the image data, and may use locational data, to determine which system, person, traveler, or area of interest that the security or border officer is observing.Image processor209 and/orimage processor219 then overlay data related to that system, person, traveler, or area of interest ondisplay211, in augmented-reality device200, so that the security or border officer may see both the real-world view of the system and the augmented display of information. This information may take the form of a floating screen, page, or window containing information. Alternatively, or additionally, this information may take the form of an icon, graphics, text, and/or a combination thereof. This information may be updated at or near real-time. In some embodiments,image processor209 and/orimage processor219 use the image data along with available augmented-reality device orientation data to analyze the security or border officer's location, the fiduciary points and any system geometry within the officer's field of view.Image processor209,219 are configured to provide one or more images to overlay ondisplay211 of augmented-reality device200.
• FIG. 3 illustrates anofficer310 using an augmented-reality device300 to monitorpersons320 at a plurality ofsecurity checkpoints330 according to an example embodiment of the present invention.
• Atsecurity checkpoints330, which may include an immigration control point or an event admission gate, one or more systems may be monitored. As depicted inFIG. 3,security checkpoints330 may include multiple system types, such as immigration kiosks, electronic gates, and the like. Systems to be monitored withinsecurity checkpoints330 are not limited to electronic gates and automated kiosks, but may further include immigration booths, environmental sensors, and the like.
• Using augmented-reality device300,officer310 monitorssecurity checkpoints330 while maintaining situational awareness.Officer310 may maintain direct visual awareness ofpersons320 interacting with the systems within thesecurity checkpoints330. For example,officer310 may wear augmented-reality device300, which can be augmented-reality glasses, that are configured to superimpose information associated withpersons320 who are passing throughsecurity checkpoints330. As discussed above,fiduciary marks340 may be included on the systems being monitored atsecurity checkpoints330.
• FIG. 4 illustrates an officer'sview400 ofsecurity checkpoints430 through an augmented-reality device according to an example embodiment of the present invention.
• As depicted inFIG. 4,officer410 equipped with the augmented-reality device, as described herein, may maintain situational awareness and direct view ofsecurity checkpoints430 while viewing information (e.g., security information) associated withpersons420. The augmented-reality device may utilize one or more of image data, location data, and/or orientation data to superimpose information associated withpersons420, such asperson data415, onview400.
• Person data415 and/or other displayed information may include real-time detailed traveler information, environmental alarms, real-time status symbols, or any other type of information that provides the security orborder officer410 with requested or pre-determined information while maintaining situational awareness ofsecurity checkpoints430, the systems within it, and the persons or travelers using the system. As shown inFIG. 4,officer410 is able to retrieve detailed, real-time information on each system and/or person in an intuitive and real-time manner without losing situational awareness and while maintaining direct visibility of the area under control.
• FIG. 5 illustrates anetwork500 that is configured to connect various components according to an example embodiment of the present invention. As shown inFIG. 5,electronic gates531,immigration kiosks532,officer terminals541, and augmented-reality devices551 are connected withbackend servers510,511 that are configured to store person or traveler data, such as security related information.
• FIG. 6 illustrates a flow diagram offunctionality600 for operating an augmented-reality device according to an example embodiment of the present invention. In some instances,functionality600 of the flow diagram ofFIG. 6 is implemented by software stored in memory or other computer readable or tangible media, and executed by a processor. In other instances, the functionality may be performed by hardware (e.g., through the use of an application specific integrated circuit (“ASIC”), a programmable gate array (“PGA”), a field programmable gate array (“FPGA”), etc.), or any combination of hardware and software.
• At the outset,functionality600 may retrieve security information and/or image data for a plurality of persons at security checkpoints, at610. Next, at620,functionality600 may generate one or more displays of the security information and/or the image data. Lastly, at630,functionality630 concurrently displays the security information for each of the plurality of persons on an augmented-reality display, wherein the security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the displaying configured to maintain visibility of the respective security checkpoints.
• By employing the embodiments, an augmented-reality device is used to enable a security officer to simultaneously monitor multiple physical manned or automated border control devices or entry control devices, such as immigration kiosks, electronic gates, or manned posts while maintaining full visibility of the area being monitored. Although example configurations have been described, the invention is not so limited. For example, an alternative configuration is to use a camera-equipped mobile device such as a smartphone or tablet in lieu of a head-mounted augmented-reality device. This will still enable situational awareness as the officer will still be able to see most of the area being monitored through the device camera. In another example, an alternative configuration may use a single fiduciary marker on each system under monitoring. In another example, an alternative configuration may eliminate the use of fiduciary markers and instead recognize the shape of the system under monitoring, which would allow the officer to move around the immigration checkpoint rather than remaining in the area where all fiduciary markers are visible. In yet another example, an alternative configuration may provide all processing power on the augmented-reality device. In yet another example, an alternative configuration may provide all processing power on a connected machine, with the augmented-reality device acting as a simple camera using fast wireless network speeds.
• One having ordinary skill in the art will readily understand that the invention as discussed above may be practiced with steps in a different order, and/or with elements in configurations that are different than those which are disclosed. Therefore, although the invention has been described based upon these preferred embodiments, it would be apparent to those of skill in the art that certain modifications, variations, and alternative constructions would be apparent, while remaining within the spirit and scope of the invention.

Claims (18)

We claim:

1. An augmented-reality device, the augmented-reality device comprising:

a processor; and

a non-transitory memory storing one or more programs for execution by the processor, the one or more programs including instructions for:

retrieving security information for a plurality of persons at respective security checkpoints; and

concurrently displaying the security information for each of the plurality of persons on an augmented-reality display, wherein the security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the displaying configured to maintain visibility of the respective security checkpoints.

2. The augmented-reality device according toclaim 1, wherein one or more fiduciary markers are disposed at or adjacent to respective security checkpoints, the fiduciary markers configured to determine location information for an augmented-reality display.

3. The augmented-reality device according toclaim 2, wherein one or more hyper-local location sensors are used to determine the location information.

4. The augmented-reality device according toclaim 1, wherein the augmented-reality device includes augmented-reality glasses.

5. The augmented-reality device according toclaim 1, wherein the augmented-reality device includes a smartphone or tablet.

6. The augmented-reality device according toclaim 1, further comprising instructions for retrieving real-time image data, and superimposing the security information of the real-time image data.

7. A non-transitory computer readable storage medium storing one or more programs configured to be executed by a processor, the one or more programs comprising instructions for:

retrieving security information for a plurality of persons at respective security checkpoints; and

concurrently displaying the security information for each of the plurality of persons on an augmented-reality device comprising an augmented-reality display, wherein the security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the displaying configured to maintain visibility of the respective security checkpoints.

8. The non-transitory computer readable storage medium according toclaim 7, wherein one or more fiduciary markers are disposed at or adjacent to respective security checkpoints, the fiduciary markers configured to determine location information for an augmented-reality display.

9. The non-transitory computer readable storage medium according toclaim 8, wherein one or more hyper-local location sensors are used to determine the location information.

10. The non-transitory computer readable storage medium according toclaim 7, wherein the augmented-reality device includes augmented-reality glasses.

11. The non-transitory computer readable storage medium according toclaim 7, wherein the augmented-reality device includes a smartphone or tablet.

12. The non-transitory computer readable storage medium according toclaim 7, further comprising instructions for retrieving real-time image data, and superimposing the security information of the real-time image data.

13. A method for operating an augmented-reality device, the method comprising:

retrieving security information for a plurality of persons at respective security checkpoints; and

concurrently displaying the security information for each of the plurality of persons on an augmented-reality display, wherein the security information is superimposed on or adjacent to each of the plurality of persons located at respective security checkpoints, the displaying configured to maintain visibility of the respective security checkpoints.

14. The method according toclaim 13, wherein one or more fiduciary markers are disposed at or adjacent to respective security checkpoints, the fiduciary markers configured to determine location information for an augmented-reality display.

15. The method according toclaim 14, wherein one or more hyper-local location sensors are used to determine the location information.

16. The method according toclaim 13, wherein the augmented-reality device includes augmented-reality glasses.

17. The method according toclaim 13, wherein the augmented-reality device includes smartphone or tablet.

18. The method according toclaim 13, further comprising instructions for retrieving real-time image data, and superimposing the security information of the real-time image data.

Images