US20030199325A1 - Apparatus and a method for more realistic shooting video games on computers or similar devices using visible or invisible light and an input computing device - Google Patents

Abstract

An apparatus, system, and a method for providing more realistic shooting input for shooting video games on computers or similar devices is provided. The apparatus may be comprised of a mock shooting device, such as a gun, having one or two lighting devices containing a plurality of light sources. The lighting devices may use visible or invisible lights. The apparatus may also include a screen device for displaying visual target objects of a video shooting game, at which a game player can shoot at with the mock shooting device, a video camera that captures video images of the mock shooting devices, and an input computing device that computes the hit position of the mock shooting device on the screen device based on the captured video images. The determined hit position can then be fed to from the input computing device to the video shooting game software run by the game computing device which can determine if a visual target object is actually hit or not, and reacts accordingly. The mock shooting device may further be comprised of a wireless sender or a wired communication line to send command signals from the mock shooting device to the input computing device. The system and method can be extended to a plurality of game players using mock shooting devices with differently colored or uniquely spatially arranged lighting devices so that a plurality of hit positions may be determined.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
• This present application is a continuation in part of and claims the priority of U.S. patent application Ser. No. 10/326,212 titled “AN APPARATUS AND A METHOD FOR MORE REALISTIC SHOOTING VIDEO GAMES ON COMPUTERS OR SIMILAR DEVICES USING VISIBLE OR INVISIBLE LIGHT”, filed on Dec. 20, 2002, which is a continuation in part of and claims the priority of U.S. patent application Ser. No. 10,128,623, titled “AN APPARATUS AND A METHOD FOR MORE REALISTIC SHOOTING VIDEO GAMES ON COMPUTERS OR SIMILAR DEVICES” filed on Apr. 23, 2002, inventor and applicant Xiaoling Wang.[0001]
FIELD OF THE INVENTION
• This invention relates to the field of systems and methods for video games, which entail the use of mock shooting devices, such as mock guns. These video games are typically comprised of computer software which is run on computers or similar devices.[0002]
BACKGROUND OF THE INVENTION
• Video games, which entail the use of mock shooting devices, are popular and entertaining. These video games are typically comprised of computer software which is run on computing devices, such as home personal computers. However, most computer video games, which entail the use of mock shooting devices typically, use computer peripherals, such as a keyboard, a mouse or a joystick to aim and shoot at visual targets on a computer or video screen. Other similar devices, such as the PLAYSTATION (trademarked) from SONY (trademarked) and the XBOX (trademarked) from MICROSOFT (trademarked), use a game pad or other game control device to aim and shoot at visual targets on a computer video screen. These types of peripheral devices make the shooting games somewhat less realistic.[0003]
• There have been some attempts to make video games which entail the use of mock shooting devices, more realistic. All known prior art in the field of shooting video games, as described in the U.S. Pat. No. 5,366,229 to Suzuki and U.S. Pat. No. 6,146,278 to Kobayashi, incorporated herein by reference, rely on three major components: a mock gun that can emit a light beam to a target on a screen to be shot at, a video camera that photographs the screen for detecting an intersecting point of the light beam on the screen, and a position determination device that determines the actual position of the light beam on the screen. The position of the light beam on the screen can then be fed back to shooting video game control computer software to determine if a visual target on a screen is “hit” or not. Some visual and audio feedback signals indicating hit or miss can be generated. Although these systems are more realistic than the shooting video games with keyboards or joysticks, they are not very suitable for use with the shooting video games on computers or similar devices.[0004]
• The main reason is the fact that a normal video camera used to photograph a computer monitor screen may not be able to provide steady video images of the computer monitor screen due to the difference in frequencies of the monitor and the video camera. The monitor refresh frequency is typically selectable between sixty—one hundred and twenty Hz while the video camera capturing frequency is typically less than thirty Hz. The video camera capturing frequency is also processing speed and image size dependent. Fast computers may be able to capture thirty video frames per second (thirty Hz) with an image size of 640 by 480 pixels. Slow computers may only be able to capture ten frames per second (ten Hz) with the same image size and thirty frames per second for a smaller size of for example 320 by 240 pixels. Only if both frequencies are identical or the monitor refresh frequency divided by the camera capturing frequency is an integer in a more general term, steady video images of the monitor screen may be captured. Since a computer user may use any refresh frequency from a wide range of monitor refresh frequencies and most video cameras have a typical capturing frequency of between ten and thirty Hz, it is very common that video cameras do not provide steady video images from a computer monitor due the frequency mismatch.[0005]
• For capturing a steady video from a computer monitor, there are only two options. The first option involves very high-end video cameras with special auto-sync functions. Although they can in general provide steady video images from most computer monitors, they are very expensive. The second option is to preset the computer monitor refresh frequency to say sixty Hz or seventy-five Hz to keep the needed synchronization with the typical video camera with say thirty Hz or twenty-five Hz, respectively. However, this is only a valid option if the same manufacturer ships the video camera and the computer monitor used by the shooting video game and computer users do not modify the refresh frequency of their computer monitor. Because video cameras, computers, monitors and video games in general are not from a single vendor or manufacturer and computer users very often do modify the refresh frequency of their monitors, this second option is very often not practical.[0006]
• In addition to the frequency mismatch problem mentioned above, the camera in the prior art as described in the U.S. Pat. No. 5,366,229, incorporated by reference herein, must be placed somewhere near a game player and facing the same orientation as the game player for capturing the display screen. Although this may not present a serious problem in a professionally designed game playing place, it could be very challenging to place the video camera at home in such a way that it may not easily be occluded at anytime during the game and may not easily be bumped into. This is not always practical. In order to solve the difficult camera placement problem, the camera as described in the U.S. Pat. No. 6,146,278, incorporated herein by reference, are integrated with the mock shooting device so that the camera is always facing the target screen without the danger of occlusion. However, this arrangement makes the mock shooting device somewhat expensive and the integrated video camera totally single-purposed. Furthermore, the mock shooting device with the camera must be connected to the computing device directly via a cable, which may also cause inconvenience when playing.[0007]
• The above mentioned drawbacks, namely, the frequency mismatch between the display screen and the low-cost video camera, the difficult placement of the video camera facing the screen, relatively high cost for a mock shooting device with an integrated camera, as well as a needed connection cable between the mock shooting device and the computing device, can seriously limit the applicability of the prior art techniques for game players who want to play realistic video shooting games with their computers at home.[0008]
SUMMARY OF THE INVENTION
• The present invention in one embodiment comprises a game computing device; an input computing device; a screen device; and a first mock shooting device comprised of one or more light sources which are a part of and fixed to the first mock shooting device. The input computing device is typically electrically connected to the game computing device. The game computing device is typically electrically connected to the screen device. The input computing device uses the light from the one or more light sources to determine whether the first mock shooting device is aimed towards a first location on the screen device. The one or more light sources may flash light when the first mock shooting device is triggered. A video camera may be used to capture video images of the one or more light sources. The video camera may be electrically connected to the input computing device and may provide data about the one or more light sources to the input computing device.[0009]
• In at least one embodiment of the present invention the apparatus is comprised of lighting devices using invisible light, such as infrared light, which is only invisible to human eyes, but well visible to common video sensors, such as a low-cost web cam. The use of the lighting devices with invisible light can effectively eliminate possible attention distractions of a game player due to the flashing lights of the lighting devices with visible light.[0010]
• In at least one embodiment of the present invention the apparatus is comprised of at least three light sources fixed to the first mock shooting device and the light sources are not located in a single line segment. Light sources may include point light sources or area light sources. A second mock shooting device comprised of one or more light sources may also be provided. Each of the light sources of the first mock shooting device may emit light of a first color and each of the light sources of the second mock shooting device may emit light of a second color, wherein the first color and the second color are different. The light sources of the first lighting device may have a first spatial arrangement and the light sources of the second lighting device may have a second spatial arrangement, wherein the first spatial arrangement and the second spatial arrangement are different. Since invisible light, such as infrared light, does not have a particular color, a typical video camera usually cannot distinguish for example colors from different near infrared wavelengths. When lighting devices use invisible light, different spatial arrangements for different mock shooting devices can be necessary for distinguishing multiple mock shooting devices.[0011]
• The present invention also includes a method of using light from one or more light sources fixed to a first mock shooting device to determine whether the first mock shooting device is aimed towards a first location on a screen device. The method may include capturing an image of the light through the use of a video camera, wherein the light can be visible or invisible to humans but visible in at least in some embodiments of the present invention to the video camera.[0012]
• The present invention in one or more embodiments discloses a new system that may use a low-cost video camera, such as a typical web cam, for capturing video images of a mock shooting device instead of a computer monitor or a television screen. From the captured video images, the pose including position and orientation of the mock shooting device can be determined by the input computing device. Please note that pose of an object is defined as to include both position and orientation of the object in space, as used commonly in the field of computer vision. We will use the terminology throughout the present invention. From the pose of the mock shooting device, the hit position on the screen or the location on the screen towards which the mock shooting device is aimed, can be computed. This system can solve the frequency mismatch problem and the difficult video camera placement problem in the prior art. It also provides a more cost effective and practical solution for game players using their computers or similar devices at home.[0013]
• The present invention is designed to provide a system and a method that can make video games, which employ a mock shooting device, much more realistic on computers and/or similar devices.[0014]
• A system, apparatus, and a method according to the present invention uses a mock shooting device, such as a mock gun, a mock machine gun, or a mock rocket launcher, with two lighting devices, a position lighting device and a command lighting device, each containing one or more light sources. A game player uses the mock shooting device with the position lighting device being turned on to aim at one of one or more target objects displayed on a screen by a video shooting game. A typical low-cost video camera captures video images containing images of the light emitted from the light sources from the position lighting device fixed to and/or apart of the mock shooting device. The hit position at which the mock shooting device aims (points to) can quickly be determined from the captured video images by the input computing device. The determined hit position is then passed to the video shooting game software running on the game computing device that displays a target cursor at the hit position on a screen device. When the game player moves the mock shooting device, the determined hit position represented by the target cursor on the screen, such as a cross hair cursor, moves accordingly. This gives the game player immediate feedback about where his/her mock shooting device is pointing to and allows the player to plan how the mock shooting device should be moved to get on top of the target object of interest. When the mock shooting device is triggered, the command lighting device of the mock shooting device may flash light to signal a shooting command. The triggered hit position can then be fed to the video shooting game software that determines if a visual target is “hit” or not, and can react accordingly.[0015]
• A system, apparatus, and a method according to the present invention uses a mock shooting device with an integrated control pad. The integrated control pad is comprised of a plurality of control buttons commonly available to the control pads for Xbox (trademarked) or Playstations (trademarked). The integrated control pad is used to send control commands (signals), such as different weapon selection commands or shooter's positioning, among many other commands. The commands of the control pad can be sent from the mock shooting device to the computing device in the following three ways:[0016]
• (1) The control commands can be encoded and sent via a lighting device showing different lighting patterns or colors or a combination of both. The lighting patterns or colors can then be captured by the video camera. The video image containing the light patterns can then be analyzed and decoded as commands by the input computing device.[0017]
• (2) A wireless connection can be used to send the control signals from the mock shooting device to the computing device. The wireless connection can be radio-wave based, such as the connections of a cordless telephone between its base station and its handset or a remote garage door opener, or infrared light based, such as a common remote control for home electronics. This embodiment typically would require a receiving device to be placed and connected to the computing device.[0018]
• (3) A wired connection can also be used to send the control signals from the mock shooting device to the computing device. The wired connection can be the same as the wired connections between a game console and its control pad. Although a wired connection is simpler and cheaper than a wireless connection, it does constrain the freedom of movement of a game player.[0019]
• The system, apparatus, and method in accordance with embodiments of the present invention offer the following advantages:[0020]
• (1) The video camera needed for the system can be a general-purpose, low cost video camera that can be used for many other applications, such as videoconferencing.[0021]
• (2) A game player may be able to use his/her existing web cam for playing the shooting game.[0022]
• (3) The mock shooting device according to some embodiments of the present invention does not need a cable to connect to the computing device. This lack of need for a connection cable imposes less movement constraints and provides a greater possible shooting distance range for the mock shooting device.[0023]
• (4) The mock shooting device needed for the system can also be a multi-purpose low cost device. The mock shooting device, such as a mock machine gun, can be used and played just like a toy machine gun with or without the shooting game.[0024]
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a perspective view schematically illustrating the overall structure of the preferred embodiment of the present invention;[0025]
• FIGS. 2A and 2B illustrate point and area light sources shown in video images;[0026]
• FIG. 3 is a block diagram or flow chart schematically illustrating a method which can be executed by a hit position determination device and a command determination device;[0027]
• FIG. 4 is a perspective view schematically illustrating the overall structure of another embodiment of the present invention;[0028]
• FIG. 5 is a perspective view schematically illustrating the overall structure of another embodiment of the present invention with a wireless connection for sending commands; and[0029]
• FIG. 6 is a perspective view schematically illustrating the overall structure of another embodiment of the present invention with a wired connection for sending commands.[0030]
DETAILED DESCRIPTION OF THE INVENTION
• The present invention in one or more embodiments provides a solution that can make shooting video games much more realistic on computers or similar devices, such as the PLAYSTATION (trademarked) from SONY (trademarked), that contain at least one processor, a memory device and/or a storage device, a monitor or a display screen, such as a television set, a low cost video camera, and some input devices, such as a game pad, and/or joysticks.[0031]
• A system, apparatus, and method according to the present invention uses a mock shooting device, such as a mock gun, a mock machine gun, or a mock rocket launcher, with a position lighting device and a command lighting device. A game player uses the mock shooting device with the position lighting device being turned on to aim at one of one or more target objects displayed on a screen by a video shooting game. When the mock shooting device is triggered, the command lighting device flashes light to send a shooting command signal to the computing device. Both lighting devices include one or more light sources and are mounted on or built in the mock shooting device. The mock shooting device can be triggered continuously with a predefined time interval when its triggering device is pulled back and not released or the mock shooting device can be triggered just one time with a quick pull back and release. The mock shooting device may also provide audio or visual feedback signals indicating that the device has been triggered. For example, the mock shooting device may play a very short and typical gun shooting sound clip when it is triggered. When it is continuously triggered, the very short and typical gun shooting sound clip will be repeated with a predefined time interval as long as the trigger is pulled back and not released.[0032]
• A system, apparatus, and method according to the present invention uses a commonly available low-cost video camera, such as a web cam, mounted on top of a screen device, such as a computer monitor or a TV set, to capture the video images containing the light from the lighting devices. The hit position on the screen at which the mock shooting device aims and shoots, can be determined from the captured video images containing the mock shooting device with the lighting devices by the input computing device. The hit position can then be fed to the game computing device running a shooting video game software which can determine if a target is actually hit or not. It should be noted that hereinafter the word “hit”, used throughout this application, is meant to be a virtual hit on the screen by a virtual bullet fired by the mock shooting device, instead of an actual hit in a physical sense.[0033]
• A perspective view of a system, apparatus, and method according to one preferred embodiment of the present invention is shown in FIG. 1. FIG. 1 shows an[0034]apparatus100 comprised of amock shooting device110, ascreen device130, avideo camera150, aninput computing device160, and agame computing device170. Theinput computing device160 may be a small dedicated computing device. Thegame computing device170 may be a personal computer or a game console machine, or other similar devices. Thescreen device130 is electrically connected to thecomputing device170 bycommunications line170a. Thevideo camera150 is electrically connected to theinput computing device160 bycommunications line150a. Theinput computing device160 is electrically connected to thegame computing device170 bycommunications line160a. The communications lines150a,160a, and170amay be comprised of wireless connections, hardwired connections, optical connections, software connections, or any other known communication connections. The communications line160ais in general machine dependent. When Xbox (trademarked) from Microsoft (trademarked) is used as the game computing device,160amust be Xbox (trademarked) compatible. In that case, communications line160amust have a connector identical to the one used by all Xbox (trademarked) controllers. When PS2 (trademarked) by Sony (trademarked) is used as the game computing device, communications line160amust be PS2 (trademarked) compatible. It must have a connector identical to the one used by all PS2 (trademarked) controllers. When a typical personal computer “PC” is the game computing device, communications line160ashould be a USB (Universal Serial Bus) or Firewire (trademarked) compatible.
• The[0035]mock shooting device110 includes aposition lighting device115 and acommand lighting device116. Theposition lighting device115 may be comprised of threelights115a,115b, and115c. Thescreen device130 can display target visual objects to be aimed and shot at. Thevideo camera150 may be used to capture video images from themock shooting device110 and thevideo camera150 can be mounted onto thescreen device130. Theinput computing device160 may be comprised of ahit determination device180, which may be comprised of computer software which is part of and is running on theinput computing device160. Thehit determination device180 may determine the hit position, such ashit position131, on thescreen device130 at which themock shooting device110 was aiming and shooting.
• The shooting path (trajectory)[0036]110ais the virtual shooting path of a virtual bullet from themock shooting device110 to thescreen device130. The light fromlights115a,115b,115c, and116aor some other light is usually non-directional so that they can be observed from a large range of directions. For this reason, each oflights115a-cand116amay be a typical small light bulb or a small LED. Thelights115a-cand116ado not need to be expensive directional lights, such as lasers. Thescreen device130 includes ascreen130aon which visual target objects, such astarget object132, are displayed. Thegame computing device170 is responsible for running theshooting game190, which may be comprised of computer software, that displays visual target objects to be shot at on thescreen130aand reacts accordingly depending on whether a visual target object has been hit or not. With some exceptions, thevideo shooting game190 may be similar to those prior art video shooting games which are typically comprised of computer software and which run on computers or game console machines. One of the differences of the present invention is how user shooting information is inputted into thegame computing device170. The system and method according to the present invention uses a realisticmock shooting device110, avideo camera150, and an input computing device, such as160, for inputting user shooting information while conventional prior art games use a keyboard, mouse, game pad or joysticks which are connected to thegame computing device170.
• In operation, referring to FIG. 1, a game player starts a[0037]video shooting game190 stored in thegame computing device170. Thevideo shooting game190 may be initially supplied to thegame computing device170 via compact disc, floppy disc, downloaded from the Internet, or in any other known manner. Theshooting game190 displays scenes with one or more visual target objects, such ascircular target object132, on thescreen130aviacommunications line170a. Typical examples of thecommunications line170aare common video display cable and the Universal Serial Bus (USB) cable version 1.1 and 2.0 for computer monitors, and composite video, S-video, or RGB (red, green, blue) video cables for television sets. The game player uses themock shooting device110 to aim and shoot at the displayed target objects provided by thevideo shooting game190 on thescreen130a. After the game player starts the game, thelight sources115a-cof theposition lighting device115 will be tuned on using theswitch118. Thelight sources115a-care each rigidly mounted on or integrated within themock shooting device110. Thevideo camera150 placed on top of thescreen device130 captures video images from thelight sources115a-cof theposition lighting device115 and sends the video images throughcommunications line150ato theinput computing device160. Typical and common examples of thecommunications line150aare the Universal Serial Bus (USB) cable version 1.1 and 2.0, or cables made according to the IEEE 1394 standard, such as the FIREWIRE (Trademarked) and the ILINK (Trademarked—copyrighted). The hitposition determination device180 running on theinput computing device160 then processes the captured video images. The hitposition determination device180 computes the position and the orientation of thelighting device115 based on the positions of the plurality oflight sources115a-cof thelighting device115 in the video images. The position and the orientation of themock shooting device110 can then be determined since thelighting device115 has a known and fixed spatial relationship with respect to themock shooting device110. (This assumption is always valid since we require that theposition lighting device115 is either an integral part of or rigidly mounted on the mock shooting device110). Based on the computed position and the orientation of themock shooting device110 relative to thescreen130a, the hit position of the virtual bullet from themock shooting device110 on thescreen130acan finally be calculated. The hit position is then passed to thevideo shooting game190 running ongame computing device170 via thecommunications line160a. By using the hit position information, a target cursor is displayed on thescreen130aso that the game player can see where exactly is his/her shooting device pointing to and determine how it should be moved to hit a target object of interest. When the target cursor is right on top of the target object, the game player can trigger themock shooting device110 to shoot at the target. When themock shooting device110 is triggered, thelight source116aof thecommand lighting device116 flashes light to signal a shooting command. The shooting command is captured by thecamera150, and thecamera150 feeds the captured image to theinput computing device160, and then theinput computing device160 determines the command signal from the image containing the light sources from the command lighting device and sends it further to thegame computing device170 running thevideo shooting game190. Thevideo shooting game190 running on thecomputing device170, determines whether an actual visual target object, such atarget object132, has been hit or not by the virtual bullet frommock shooting device110 and if so theshooting game190 executes the appropriate computer software instructions for a “hit” scenario.
• The position and the orientation of the[0038]mock shooting device110 in space can be determined indirectly via a pose estimation of the rigidly mounted or integratedposition lighting device115. This indirect method reduces the computational complexity and improves the robustness of the method significantly. The advantages can be summarized as follows:
• (1) No difficult object and background separation problem. The pose estimation of a general three-dimensional object, such as the[0039]mock shooting device110, is not always simple, when the object is not easily separable from the background or the environment in which the object exists. The object and background separation problem in general is regarded as a difficult computer vision problem that is not always easily solvable. However, if thelight sources115a-cof theposition lighting device115 have been turned on, the light sources will be imaged as bright blobs in video images. When themock shooting device110 is triggered, thelight source116aof thecommand lighting device116 that flashes light will also be imaged as an additional bright blob in video images captured by thevideo camera150. Bright blobs are in general very easily detectable and hence quickly separable from a background without additional bright light sources. This assumed condition is usually not difficult to be satisfied.
• (2) Low localization complexity of feature points. For object pose estimation, object feature points, such as edge, junction and corner points, must be localized. In general, these image feature points take longer to compute than the detection of simple bright blobs generated by point or area light sources.[0040]
• (3) Furthermore, bright blobs can be detected much more reliably than common image feature points. This is especially true if the image contrast is low and the noise level is high, when the image is taken under a typical low illumination condition.[0041]
• As discussed above, the[0042]position lighting device115 plays a significant role for solving the pose estimation of themock shooting device110. There is a question of how many points do we need to estimate the pose of themock shooting device110 via theposition lighting device115. As stated in the reference by M. L. Liu et. al., which is incorporated by reference herein, three non-collinear corresponding points (i.e. three image points that are not arranged along a single line in space) are sufficient for the pose estimation of an object. However, in order to make the pose estimation more reliable, four or more points may be helpful. For example, a method with four points is proposed in the reference by M. L. Liu et. al. The proposed method works with four non-collinear (i.e. all points are not arranged along a single line in space) points that can either be co-planar (i.e. all points are arranged along a single plane in space) or non-coplanar (i.e. all points are not arranged along a single plane in space). The proposed method may also be extended to handle more points. Because the pose estimation problem with image points is a well-known and solved problem, details will not be described in this invention and can be found in the cited reference. It is important to point out that the cited reference only serves the purpose of a common reference. It does not indicate in any way that the method is the preferred one, but only that it can be used with the system and the method according to one or more embodiments of the present invention. Therefore, we can conclude that a minimum of three non-collinear point light sources, such as115a,115b, and115cshown in FIG. 1, should be used for theposition lighting device115 for pose estimation with reasonable accuracy. For better accuracy, four or more non-collinear point light sources may be used. If high accuracy is not required, less than three points may also be used for providing a rough estimation of the hit position.
• While the[0043]position lighting device115 is designed for the pose estimation, thecommand lighting device116 is for sending command signals. Thecommand lighting device116 as shown in FIG. 1 contains only onelight source116abecause only one command signal, the shooting command in this case, needs to be sent. When more command signals are needed, more light sources may be needed for the command lighting device. The use of more than one light source for a command lighting device, similar todevice116 will be discussed later.
• There are two common types of light sources, which may be used for solving our pose estimation. A point light source is a light source with a very small and isolated, most likely rounded lighting area that represents only a few bright pixels or a very small bright spot in a video image. Typical examples of point light sources in a video image are shown and marked as point light sources[0044]315a-315cinvideo image316 in FIG. 2A. The position of a point light source in a video image can easily be localized through determining the position of the centroid of a small and isolated bright blob. For a point light source, the shape of a point light source, such as pointlight source315a, is normally not used or evaluated for pose estimation due to its compact size. As mentioned previously, we typically need at least three point light sources for estimating the pose of themock shooting device110. In contrast, for an area light source, such as a light source in the shape of a triangle or a rectangle, such as triangularlight source215 invideo image216 in FIG. 2A and rectangularlight source415 invideo image416 shown in FIG. 2B, respectively, the light source's shape may be used for computing the position and the orientation of the light source. In general, one area light source with, say three or four, corners, can be seen as equivalent to three or four point light sources, respectively. As shown in FIG. 2A, for example, the three corner points,215a-c, of a triangular-shaped arealight source215 can easily be extracted and these three extracted corner points can be viewed as similar to the three point light sources315a-c, arranged in a triangular shape. Similarly, a rectangular arealight source415, shown in FIG. 2B, has four corner points,415a-d, that can be seen as or equivalent to four coplanar point light sources515a-d.
• Therefore, one triangular area light source may be sufficient to satisfy the minimum condition of three point light sources for the pose estimation, as mentioned previously. Depending on the design of the[0045]mock shooting device110, thelighting device115 may be comprised of point light sources, area light sources, or a combination of both. In general, more light sources lead to more accurate and robust pose estimation. However, on the other hand, more light sources mean longer computational time and higher production cost.
• FIG. 3 shows a[0046]flow chart500 illustrating a method which can be executed by a hit position determination device running oninput computing device160, such asdevice180 shown in FIG. 1, for determining the hit position of a virtual bullet shot from themock shooting device110. At step510 a video image is captured. The video image may be captured byvideo camera150, which then transmits data via thecommunications line150ato theinput computing device160. The captured video image may be subjected to a bright blob localization process by hitposition determination device180 atstep530. Theinput computing device160, which runs the hitdetermination device180 computer software, may scan through the whole captured video image pixel by pixel and may compare a pixel intensity value with a given or computed threshold value which may be stored in memory of theinput computing device160. Pixels with intensity value greater than the threshold value may be identified as “bright” pixels by theinput computing device160. If theinput computing device160 cannot find any bright pixels in the image, theinput computing device160 determines that no lighting device of themock shooting device110 was turned on when the captured video image was captured and no further processing is needed. Otherwise, theinput computing device160 determines if the detected bright pixels form bright blobs with bright neighboring pixels. Thisstep530 essentially removes noisy pixels and localizes the bright blobs. The identified bright blobs are then compared with a given expected size range of the bright blobs as well as the given expected total number of bright blobs for verifying the correctness of the blob localization. For example, if a system uses three point light sources in its position lighting device and one point light source in its command lighting device, and the blob size of each imaged point light source is between five and ten pixels in diameter, theinput computing device160 will check if the total number of bright blobs is four or three (corresponding to with or without thecommand lighting device116 being turned on) and if the diameter of each bright blob is indeed between five and ten pixels. Only if both checks are successful, theinput computing device160 can be certain that the localized bright blobs are indeed coming from the four or three expected point light sources. Otherwise, theinput computing device160 may decide to go back to look for more bright blobs in the image with a lowered threshold value or post an error message. The localized bright blobs are then subjected to a position determination process atstep540 by theinput computing device160 for blob center and blob corners.
• If only point light sources are used in the lighting device, the[0047]input computing device160 atstep540 will perform position determination for each blob center. The center position of a blob can easily be computed by averaging the pixel coordinates of each pixel within the blob. If one or more area light sources are used, theinput computing device160 atstep540 will perform corner detection for every given bright blob. For example, if one rectangular-shaped area light source is used in the lighting device, theinput computing device160 will localize four expected corners. Since corner detection methods are very common and basic in the computer vision field and described in almost all textbooks about computer vision and image processing, we skip the details for simplicity and clarity of the description. When a mixture of point and area light sources are used, both blob center and corner detections are needed.
• The localized center and/or corner points are then passed to a blob separation process at[0048]step545. Atstep545, theinput computing device160 separates the blobs corresponding to theposition lighting device115 and thecommand lighting device116 by comparing of the blob positions with the known geometry of both lighting devices.
• After the blob separation, the blobs corresponding to the[0049]position lighting device115, have been separated out atstep547, and will be fed to a pose estimation process atstep550. Atstep550, theinput computing device160 takes center and/or corner points as input, and estimates the position and the orientation of thelighting device115. This method works with either point or area light sources. The type of light sources makes only differences instep540 as discussed in the previous section: locate blob center points for point light sources and locate blob corner points for area light sources. A good working method for pose estimation with four feature points is well described in the reference by M. L. Liu et al., which is incorporated by reference herein. Since there are many published pose estimation methods that could be used with the present invention without modification, and the description of the pose estimation process itself is complicated, the present application again skips the details. After the pose (position and orientation) of themock shooting device110 is determined by theinput computing device160 atstep550, theshooting path110aof the virtual bullet from themock shooting device110 as shown in FIG. 1 can easily be obtained.
• Once the[0050]shooting path110aof the virtual bullet from themock shooting device110 is computed by the pose estimation process atstep550, the hit position of a virtual bullet from themock shooting device110 can then be computed by theinput computing device160 by a hit position estimation process atstep560 shown in FIG. 3. The hit position estimation process atstep560 treats thedisplay screen130aas a plane with its known position and orientation and theshooting path110aor line of themock shooting device110 as a line in space with its known position and orientation, and computes the intersection point of the plane (i.e.display screen130a) and the line (i.e. shooting path orline110a). The intersection point is the hit position, such asposition131, of the virtual bullet on thedisplay screen130a. The hit position estimation process atstep560, executed byinput computing device160, finally outputs the screen hit position atstep590 to theshooting game190 which is computer software running on thegame computing device170.
• In parallel to the pose estimation for the blobs from the[0051]position lighting device115, the blobs from thecommand lighting device116, determined atstep548, are fed to the command determination process atstep562. When only one point light source is used in thecommand lighting device116 as in the current example, theinput computing device160 atstep562 simply determines if there is a bright blob from thecommand lighting device116 in the image. If this one bright blob has been found, this process simply outputs the determined shooting command, such as “fired” atstep592.
• Finally, the[0052]video shooting game190 uses the outputs supplied atsteps590 and592 as input to determine if an actual visual target object displayed on thedisplay screen130ahas been hit or not and thevideo shooting game190 reacts depending on how it is programmed.
• The[0053]apparatus100 shown in FIG. 1 may be extended to include a plurality of mock shooting devices, each of which may be identical to themock shooting device110 equipped withlighting devices115 and116 using different colors for multiple game players. If thevideo camera150 is a color camera, light sources in different colors can easily be distinguished. For example, for a dual user apparatus, two mock shooting devices, each like110, one mock shooting device having only red light sources, such as one or more red light sources of a red lighting device and one mock shooting device having only green light sources such as one or more green light sources of a green lighting device, may be operated by two game players. The pose of the two mock shooting devices may be determined separately by locating the red pixels for one of the mock shooting devices and the green pixels for the other in the same video images as long as the red pixels and the green pixels are not overlapping in space. When an overlap of one or more of the lighting sources of the lighting devices occurs, some red or green light sources may be occluded and hence no longer detectable. This may lead to inaccurate pose estimation and in the worst case to wrong pose estimation results. Therefore, if more than one player are playing, it is important to keep a certain minimum distance between all mock shooting devices for accurate pose estimation of the mock shooting devices.
• When invisible light is used by the lighting devices, no color separation is possible. In this case, each of the mock shooting devices should have its own characteristics for easy differentiation. For example, a mock shooting device may use point light sources arranged in a triangular shape, while others may contain point light sources arranged in a rectangular shape or in a more general polygonal shape. Furthermore, if one mock shooting device contains only point light sources, the others may be comprised of area light sources or a combination of point and area light sources. In general, the characteristics of each lighting device, such as its shape and spatial distribution, should be as different as possible for easy separation.[0054]
• There are two main types of video shooting games available on the market today. The first type displays only targets on a screen to be shot at. The main task for a game player playing with this type of shooting game is to shoot and destroy the displayed targets. The present invention in various embodiments can serve this type of video shooting game very well by making them more exciting and realistic. The second type of video shooting games displays not only targets to be destroyed but also a shooter on a screen who can actively seek and destroy the targets. Furthermore, a game player may also change his/her weapon dynamically depending on needs. The game player may for example select to use a simple handgun in some cases, a machine gun when more enemies are present, a rifle with a telescope to play like a sniper, or even an anti-tank missile launcher for destroying enemy tanks. For this type of more advanced video shooting games, it is certainly desirable if a mock shooting device can not only reveal its pose and send a simple shooting command, but also send more advanced commands mentioned previously. Fortunately, only small modifications of the[0055]mock shooting device110 are needed to make this possible, as shown in FIG. 4. FIG. 4shows apparatus600 comprised of amock shooting device610, ascreen device630, avideo camera650, aninput computing device660, and agame computing device670. Theinput computing device660 may be a small dedicated computing device. Thegame computing device670 may be a personal computer, a game console machine, or a similar device. Thescreen device630 is electrically connected to thegame computing device670 by acommunications line670a. Theinput computing device660 is electrically connected to thegame computing device670 by acommunications line660a. Thevideo camera650 is electrically connected to theinput computing device660 bycommunications line650a. The communications lines650a,660aand670amay be comprised of wireless connections, hardwired connections, optical connections, software connections, or any other known communication connections.Devices610,630,650,660, and670 ofapparatus600 shown in FIG. 4 are similar to thedevices110,130,150,160, and170 ofapparatus100 shown in FIG. 1. In comparison with theapparatus100 shown in FIG. 1, theapparatus600 has mainly two differences between the mockingshooting device610 and themock shooting device110. Theposition lighting device615 is similar to theposition lighting device115 shown in FIG. 1. However, unlike thecommand lighting device116, thecommand lighting device616 is comprised of a plurality of light sources instead of one, such aslight sources616a-d. Thecommand lighting device616 may be rigidly mounted to thedevice610. While theposition lighting device615 is activated by aseparate switch618 of themock shooting device610, similar tomock shooting device110, thecommand lighting device616 is controlled by thetrigger612 and a plurality of control buttons of an integrated control pad, such as buttons619a-g. The control buttons619a-gmay be placed anywhere on themock shooting device610 as long as the buttons619a-gcan easily be reached and operated by the thumbs or fingers of a game player. The placement of the control buttons619a-gshown in FIG. 4 serves only as one example for the simple illustration purpose. More ergonomic designs may be employed to make the control buttons more accessible and more efficient. When theposition lighting device615 is turned “ON” by theswitch618, thelight sources615a-ccan easily be imaged as bright blobs by thevideo camera650. Theinput computing device660 performs pose estimation processing steps similar toinput computing device160, to determining the pose of theposition lighting device615. Since theposition lighting device615 is typically rigidly mounted on or an integrated part of themock shooting device610, the pose of themock shooting device610 can be determined. Independently, thetrigger612 of themock shooting device610 controls thecommand lighting device616. When thetrigger612 is pulled back, thecommand lighting device616, includinglight sources616a-d, flashes one or more lights to send the shooting command signal, independent of the state of theposition lighting device615. When each of the control pad buttons619a-gis operated by a game player, thecommand lighting device616 will flash a different and a unique combination of lights of thelight sources616a-d. Since four lights in the current example can theoretically create sixteen unique combinations (four lights, each with two states “ON” and “OFF”, are equivalent to an encode with four bits that can encode sixteen combinations), up to sixteen command signals can be sent. The shooting command can be one of the command signals to be encoded. Thetrigger612 can be seen as a special control button that is always associated with the shooting command. Other control buttons are more flexible and can be associated with any commands or command signals of interest. In general, the more command signals that need to be sent, the more lights that are needed for thecommand lighting device616. For great robustness of the command signal detection with for example error correction capabilities, more lights may be used for sending command signals with a certain degree of redundancy.
• A perspective view of a system, apparatus, and method according to another embodiment of the present invention is shown in FIG. 5. FIG. 5 shows an[0056]apparatus700 comprised of amock shooting device710, ascreen device730, avideo camera750, aninput computing device760, and a game computing device770. Theinput computing device760 may be a small dedicated computing device. The game computing device770 may be a personal computer, a game console machine, or a similar device. Thescreen device730 is electrically connected to the computing device770 bycommunications line770a. Theinput computing device760 is electrically connected to the game computing device770 by a communications line760a. Thevideo camera750 is electrically connected to theinput computing device760 bycommunications line750a. The communications lines750a,760a, and770amay be comprised of wireless connections, hardwired connections, optical connections, software connections, or any other known communication connections.Devices710,730,750,760, and770 of theapparatus700 shown in FIG. 5 are similar to thedevices610,630,650,660, and670 of theapparatus600 shown in FIG. 4. In comparison with theapparatus600 shown in FIG. 4, theapparatus700 has the following two differences:
• (1) The[0057]command lighting device616 shown in FIG. 4 has been replaced with awireless sender720 as shown in FIG. 5. All control command signals are now encoded in the embodiment of FIG. 5, and sent via thewireless sender720 instead of thecommand lighting device616 in FIG. 4.
• (2) A[0058]wireless receiver740 is now added to the apparatus for receiving the control command signals sent by thewireless sender720. Thewireless receiver740 communicates with theinput computing device760 via thecommunications line740a. The communications line740acan be eliminated when thewireless receiver740 is a built-in component of theinput computing device760.
• The wireless connection with the[0059]sender720 and thereceiver740 can be radio wave based or infrared light based. Cordless phones and remote garage door openers are good examples of simple and low-cost wireless systems based on radio waves. Common remote controls used to control home electronics, such as TVs and VCRs, are mostly based on infrared light. In general, radio wave and infrared light are proven and low cost technologies. Both can be used in the embodiments according to the present invention.
• The[0060]apparatus700 may be simplified by integrating thewireless receiver740 into thevideo camera device750. By doing so, theapparatus700 has one less component and oneless connection line740awith the computing device770. This can make the whole system cheaper and compacter with less messy cables. The main disadvantage of this integrated solution is the fact that ordinary web cams can no longer be used with such a system.
• The[0061]apparatus700 may also be simplified by integrating theinput computing device760 into thevideo camera device750. After the integration, thevideo camera device750 possesses its in-camera on-board computing capability of the input computing device. The on-board computing unit can quickly process the images captured by the video sensor without data transfer. By doing so, theapparatus700 has one less component and one less connection line760awith the computing device770. This can make the whole system cheaper and compacter with less messy cables.
• In addition to the above mentioned two separate integration solutions, the[0062]apparatus700 may further be simplified by integrating both thewireless receiver740 and theinput computing device760 into thevideo camera device750. By doing so, theapparatus700 has two less components and two less connection lines,740aand760a. This can make the whole system again cheaper and compacter with less messy cables. The main disadvantage of this integrated solution is again the fact that ordinary web cams can no longer be used with such a system.
• A perspective view of a system, apparatus, and method according to another embodiment of the present invention is shown in FIG. 6. FIG. 6[0063]shows apparatus800 comprised of amock shooting device810, ascreen device830, avideo camera850, aninput computing device860, and agame computing device870. Theinput computing device860 may be a small dedicated computing device. Thegame computing device870 may be a personal computer, a game console machine or a similar device. Thescreen device830 is electrically connected to thecomputing device870 bycommunications line870a. Theinput computing device860 is electrically connected to thegame computing device870 by acommunications line860a. Thevideo camera850 is electrically connected to theinput computing device860 bycommunications line850a. The communications lines850a,860aand870amay be comprised of wireless connections, hardwired connections, optical connections, software connections, or any other known communication connections.Devices810,830,850,860 and870 ofapparatus800 shown in FIG. 6 are similar to thedevices710,730,750,760 and770 ofapparatus700 shown in FIG. 5. In comparison with theapparatus700 shown in FIG. 5, theapparatus800 has the following main differences: The wireless connection including thesender720 and thereceiver740 are now replaced by awired communications line810b. All control command signals from themock shooting device810 are now sent to theinput computing device860 in the embodiment of FIG. 6, and received via thecommunications line810b. The main advantage of this approach is its low cost and simple implementation. The main drawback is the constraint of movement freedom of a game player caused by such a wired cable between themock shooting device810 and thecomputing device870.
• The[0064]apparatus700 shown in FIG. 5 and theapparatus800 shown in FIG. 8 may also be extended to include a plurality of mock shooting devices, each of which may be identical to themock shooting device110 equipped withlighting devices115 and each of which uses a different characteristic, such as a different color or a different spatial arrangement, or a combination of them, for multiple players. In addition, each of the plurality of mock shooting devices may also have a unique identifier. The unique identifier may be just a unique identification number or other unique properties. Each of the plurality of mock shooting devices may use its unique identifier to send command signals so that the computing device can easily distinguish command signals from each of the plurality of mock shooting devices.
• Although the invention has been described by reference to particular illustrative embodiments thereof, many changes and modifications of the invention may become apparent to those skilled in the art without departing from the spirit and scope of the invention. It is therefore intended to include within this patent all such changes and modifications as may reasonably and properly be included within the scope of the present invention's contribution to the art.[0065]

Claims (31)

I claim:

1. An apparatus comprising

a game computing device;

an input computing device;

a screen device;

a first mock shooting device having a position lighting device comprised of one or more light sources which are fixed to the mock shooting device; and

wherein the game computing device is electrically connected to the screen device; and

wherein the input computing device is electrically connected to the game computing device; and

wherein the input computing device uses the light from the one or more light sources of the position lighting device to determine whether the first mock shooting device is aimed towards a first location on the screen device; and

wherein the input computing device sends the aiming location information of the first mock shooting device on the screen device to the game computing device.

2. The apparatus ofclaim 1 wherein

wherein the first mock shooting device is further comprised of a command lighting device which is comprised of one or more light sources; and

wherein the one or more light sources flash light when the first mock shooting device is triggered.

3. The apparatus ofclaim 2 wherein

the first mock shooting device is further comprised of one or more control buttons; and wherein the control buttons may be used to send control command signals; and

the command lighting device with the one or more light sources flashes a unique light pattern for sending a command signal when one of the control buttons on the first mock shooting device is operated.

4. The apparatus ofclaim 1 wherein

the first mock shooting device is further comprised of a wireless sender; and

wherein the wireless sender sends command signals when the control buttons are operated or the first mock shooting device is triggered.

5. The apparatus ofclaim 4 further comprising

a wireless receiver; and

wherein the wireless receiver is electrically connected to the computing device and passes the command signals received from the wireless sender to the input computing device.

6. The apparatus ofclaim 1 further comprising

a video camera that captures video images of the one or more light sources;

wherein the video camera is electrically connected to the input computing device and provides data about the one or more light sources to the input computing device.

7. The apparatus ofclaim 4 further comprising

a video camera with an integrated wireless receiver that captures video images of the one or more light sources of the position lighting device and receives command signals from the wireless sender of the first mock shooting device;

wherein the video camera is electrically connected to the input computing device and provides data about the one or more light sources of the position lighting device and the received command signals to the input computing device.

8. The apparatus ofclaim 1 wherein

the first mock shooting device is further comprised of a wired communications line;

and wherein the wired communications line communicates command signals from the first mock shooting device to the input computing device.

9. The apparatus ofclaim 1 wherein

the light from the one or more light sources of the position lighting device is invisible to human eyes but visible to the video camera.

10. The apparatus ofclaim 1 wherein

wherein at least one of the light sources of the position lighting device is a point light source.

11. The apparatus ofclaim 1 wherein

wherein at least one of the light sources of the position lighting device is an area light source.

12. The apparatus ofclaim 11 wherein

the area light source is a polygonal light source.

13. The apparatus ofclaim 1 wherein

the plurality of light sources of the position lighting device are comprised of at least one point light source and at least one area light source.

14. The apparatus ofclaim 1 wherein

the plurality of light sources of the position lighting device are comprised of at least one light source which emits light of a first color and at least one light source which emits light of a second color; and

wherein the first and second colors are different.

15. The apparatus ofclaim 1 further comprising

a second mock shooting device comprised of a position lighting devices;

wherein the position lighting device of the second marking device is comprised of one or more light sources;

wherein each of the one or more light sources of the position lighting device of the first mock shooting device has a first characteristic;

wherein each of the one or more light sources of the position lighting device of the second marking device has a second characteristic; and

wherein the first characteristic and the second characteristic are different.

16. The apparatus ofclaim 15 wherein

the first characteristic is comprised of a first color of light which is emitted from the one or more light sources of the position lighting device of the first mock shooting device;

the second characteristic is comprised of a second color of light which is emitted from the one or more light sources the position lighting device of the second mock shooting device; and

wherein the first color is different from the second color.

17. The apparatus ofclaim 15 wherein

the first characteristic is comprised of a first spatial arrangement of the one or more light sources of the position lighting device of the first mock shooting device;

the second characteristic is comprised of a second spatial arrangement of the one or more light sources of the position lighting device of the second mock shooting device;

and wherein the first spatial arrangement and the second spatial arrangement are different.

18. The apparatus ofclaim 15 wherein

the first characteristic is comprised of a first combination of spatial arrangement and colors of the one or more light sources of the position lighting device of the first mock shooting device;

the second characteristic is comprised of a second spatial arrangement and colors of the one or more light sources of the position lighting device on the second mock shooting device;

and wherein the first combination and the second combination are different.

19. The apparatus ofclaim 1 wherein

the first mock shooting device is comprised of a first identifier; and further comprising a second mock shooting device comprised of a second identifier;

wherein the first identifier and the second identifier are different; and

wherein the first mock shooting device uses the first identifier to send command signals with a first characteristic;

wherein the second mock shooting device uses the second identifier to send command signals with a second characteristic; and

wherein the first characteristic and the second characteristic are different.

20. An apparatus comprising

a game computing device;

an input computing device;

a screen device;

a mock shooting device is comprised of a lighting device containing one or more light sources which are fixed to the mock shooting device; and

wherein one or more light sources of the lighting device emit visible or invisible light;

wherein the input computing device uses the light from the one or more light sources of the lighting device to determine whether the mock shooting device is aimed towards a location on the screen device;

and wherein the input computing device passes the location information about the mock shooting device to the game computing device

21. The apparatus ofclaim 20 wherein

the mock shooting device is comprised of a further lighting device containing one or more light sources which are fixed to the mock shooting device; and

wherein the further lighting device may emit invisible light;

and wherein the input computing device determines a command signal being sent by the mock shooting device from the light from the one or more light sources of the further lighting device.

22. The apparatus ofclaim 20 further comprising

a video camera that captures video images of the one or more light sources of the lighting device; and

wherein one or more light sources of the lighting device emit light which is invisible to human eyes but visible to the video camera; and

wherein the video camera is electrically connected to an input computing device and the video camera provides data about the one or more light sources of the lighting device to the input computing device.

23. The apparatus ofclaim 20 wherein

the mock shooting device is comprised of a trigger; and

wherein one or more command signals are sent by the mock shooting device when the trigger is operated.

24. The apparatus ofclaim 20 wherein

the mock shooting device is comprised of a plurality of control buttons; and wherein one or more command signals are sent by the mock shooting device when one or more of the plurality of control buttons are operated.

25. An apparatus comprising

An input computing device;

a game computing device;

a screen device;

a mock shooting device which can send a signal via a communications line;

and a lighting device containing one or more light sources which are fixed to or as a built-in part of the mock shooting device; and

wherein one or more light sources of the lighting device emit light; and

wherein the input computing device uses the light from the one or more light sources of the lighting device to determine whether the mock shooting device is aimed towards a location on the screen device;

wherein the communications line is used to send one or more command signals from the mock shooting device to the input computing device;

and wherein the input computing device further sends the command signals and the shooting location information to the game computing device

26. The apparatus ofclaim 25 wherein

the one or more light sources of the lighting device emit invisible light.

27. The apparatus ofclaim 25 further comprising

a wireless sender which is connected to the mock shooting device or a part of the mock shooting device; and

a wireless receiver which is connected to the input computing device; and wherein the wireless sender and receiver use infrared light to send and receive command signals wirelessly.

28. The apparatus ofclaim 25 wherein

the wireless sender sends and the wireless receiver receives electromagnetic waves in the frequency range between one kilohertz and fifty gigahertz to send and receive command signals wirelessly.

29. The apparatus ofclaim 25 wherein

the mock shooting device is comprised of a trigger; and

wherein one or more command signals are sent by the mock shooting device via the communications line when the trigger is operated.

30. The apparatus ofclaim 25 wherein

the mock shooting device is comprised of a plurality of control buttons; and wherein one or more command signals are sent by the mock shooting device via the communications line when one or more control buttons are operated.

31. The apparatus ofclaim 25 further comprising

a video camera that captures video images of the one or more light sources of the lighting device; and

wherein the video camera is electrically connected to an input computing device and the video camera provides data about the one or more light sources of the lighting device to the input computing device.

Images